I have a python script which runs fine but takes far too long to run so I'm hoping some of the more experienced python coders can offer suggestions about where I can be more efficient. The script I've written will hopefully be run as a windows task at a given time interval to constantly update a table in SDE which will be consumed by a web map.
As I said, the table is in SDE but the source data is a table in an Access database. Due to a daily database maintenance period, the Access database cannot be touched unless the presence/absence of two files is true (hence the conditional statement about the files). If it's ok to proceed, here's what happens:
It took roughly 20 minutes to transfer 2200 records. That seems WAY too slow so any suggestions would be greatly appreciated! Here's the code:
As I said, the table is in SDE but the source data is a table in an Access database. Due to a daily database maintenance period, the Access database cannot be touched unless the presence/absence of two files is true (hence the conditional statement about the files). If it's ok to proceed, here's what happens:
- Delete all records from the SDE table
- Connect to the Access database
- Step through the records in an Access Query and transfer that information into the SDE table
It took roughly 20 minutes to transfer 2200 records. That seems WAY too slow so any suggestions would be greatly appreciated! Here's the code:
Code:
import sys
import os
import pyodbc
import arcpy
from datetime import datetime
from arcpy import env
file01 = r"\\Path\to\File01" #This file must exist
file02 = r"\\Path\to\File02" #This file must NOT exist
expression = '1=1' #SQL shorthand which select all records
theTable = "tblGageData"
# The tables within DIADvisor must not be accessed during its daily database maintenance.
# OneRain recommends checking for the existence and non-existence of two specific files.
# If both conditions are true, it is safe to proceed with accessing the data within the
# dvLive Access database
if os.path.exists(file01) and not os.path.exists(file02):
print "Processing start time: " + str(datetime.now())
env.workspace = r"Database Connections\SdeConnection.sde"
try:
# Set some local variables
tempTableView = "gageTableView"
# Execute MakeTableView
arcpy.MakeTableView_management(theTable, tempTableView)
# Execute SelectLayerByAttribute to select all records
arcpy.SelectLayerByAttribute_management(tempTableView, "NEW_SELECTION", expression)
# Execute GetCount and if some records have been selected, then execute
# DeleteRows to delete the selected records.
if int(arcpy.GetCount_management(tempTableView).getOutput(0)) > 0:
arcpy.DeleteRows_management(tempTableView)
# Now connect to the DIADvisor access database and import the most recent data
# This requires the PYODBC module for Python
cstring = r'DRIVER={Microsoft Access Driver (*.mdb)};DBQ=Path\to\Access\Database\Database.mdb;Provider=MSDASQL;'
conn = pyodbc.connect(cstring)
cursor = conn.cursor()
sqlString="SELECT * FROM LAST3DAYS"
counter = 0
# Loop through the results returned via the ODBC connection
for cRow in cursor.execute(sqlString):
curSensorId = cRow.SENSOR_ID
curEpoch = cRow.EPOCH
curData = cRow.DATA
counter += 1
#Insert a new row into the SDE table with the current DIADvisor record's information
curSde = arcpy.InsertCursor(r"Database Connections\SdeConnection.sde\tblGageData")
row = curSde.newRow()
row.SENSOR_ID = curSensorId
row.EPOCH = curEpoch
row.DATA = curData
curSde.insertRow(row)
# Close the ODBC connection and perform some variable cleanup
cursor.close()
conn.close()
del row
del conn
del curSde
del cRow
del cursor
print "Number of record(s) in the DIADvisor database: " + str(counter)
print "Processing end time: " + str(datetime.now())
except Exception as e:
# If an error occurred, print line number and error message
import traceback, sys
cursor.close()
conn.close()
tb = sys.exc_info()[2]
print "Line %i" % tb.tb_lineno
print str(e)
else:
sys.exit()