GIS Logo GSP 118 (318): GIS Programming

The Ultimate ArcGIS Tools

1. Architecture

You now have all the skills needed to create virtually any tool for ArcGIS that can be created. You can circumvent the problems introduced by ArcGIS crashing and Esri modifying Python 26 by having a tool call a Python script that launches another Python script. This script can then call other programs, like R, and then display results in tKinter without the lock up problems experienced with ArcGIS. This adds some complexity but this is made up for in the flexibility you'll gain.

One of the most powerful tools for statistics is "R". R contains a huge number of statistical functions and is free! R is not directly supported within ArcGIS but R is a programming language in it's own right and you can create scripts for R to execute. The problem is that R and ArcGIS do not play well together and this can crash ArcGIS. The solution is to create a Python script that is sublaunched by another Python script.

The example below will show you how to create a tool in ArcGIS that sublaunches another Python script that then calls R. Don't worry too much about the R code but you have seen all the Python methods used here during this class.

2. Python Script 1: Python Calling R

Below is the Python script that will be executed by the tool script below. The steps to have this process work are a bit more complicated than we have seen in the past. To test this script:

  1. Save the Python script as "C:/ProjectsPython/RegressionTool/RRegression.py".
  2. Create a file in Excel with an X and a Y column and add about 10 numbers to each column.
  3. Save this file as "C:\Temp\Inputs.csv".
  4. Make sure that the path to "R" is correct for your computer.
  5. Make sure that the "Python Image Library" or "PIL" is installed. If not, search for it on Google and run the appropriate installer.
  6. Run the file to see that it displays a linear regression chart in a window. 
#####################################################################################
# Script to sublaunch R from Python and create a linear regression model from
# a file with x and y values in it.
# To run the script, create a "CSV" file with an "X" and a "Y" column with values
# to be used for the linear regression.
#
# Authors: Jim Graham and Andy Larkin
# Date: 4/22/2014
#####################################################################################

# Import the standard libraries
import os.path
import subprocess
import time # bring in the time library so we can "wait" between drawing
import sys

# Import tkinter for a window to display the graph and the file dialog
from Tkinter import *
import tkFileDialog

#Import the Python Image Library to read image file formats like PNG
import Image, ImageTk

# Paths
WorkingFolder="C:/Temp/"
RPath="\"C:/Program Files/R/R-3.0.1/bin/R.exe\""
#RPath="\"C:/Program Files/R/R-2.14.1/bin/R.exe\""

# This is the R Script that will be saved to a file and then executed with a subprocess
RScript="""
########################################################### 
# Setup global variables 
FolderPath = 'C:/Temp/' 

InputFilePath = paste(FolderPath,'Inputs.csv',sep='') 
PlotFilePath = paste(FolderPath,'OutputPlot.gif',sep='') 
ResultsFilePath = paste(FolderPath,'OutputResults.txt',sep='') 

########################################################### 
# Read the data into R 
TheData = read.csv(InputFilePath) 

# Create a linear model for Y, Given X 
LinearModel=lm(TheData$Y~TheData$X) 
 
########################################################### 
# Direct plot output to a png file 
png(PlotFilePath) 

# Draw the points in the plot 
plot(TheData$X,TheData$Y, main=PlotTitle, xlab=XAxisLabel, ylab=YAxisLabel) 

# Add the model output to the plot 
abline(LinearModel) 

# Finalize output to the png file 
dev.off() 
 
########################################################### 
# Direct output of the model results to a text file 
sink(ResultsFilePath, append=FALSE, split=FALSE) 

# Print the model reuslts 
LinearModel 

# Return output to the console  
sink()	  """

try:
	# Setup some dummy variables for when we are debugging this script
	PlotTitle="Title"
	XAxisLabel="X Axis"
	YAxisLabel="Y Axis"
	
	# Get the title and labels from the command line
	TheArguments=sys.argv
	if (len(TheArguments)>1):
		PlotTitle=TheArguments[1]
		XAxisLabel=TheArguments[2]
		YAxisLabel=TheArguments[3]
	
	# Add the title and labels to the start of the R Script
	RScript="PlotTitle <- '"+PlotTitle+"' \n " + \
		"XAxisLabel <- '"+XAxisLabel+"' \n " + \
		"YAxisLabel <- '"+YAxisLabel+"' \n " + \
	        RScript
	
	# Setup Tkinter
	root = Tk()
	root.title("Blobs")
	root.resizable(0, 0)
	
	# Write out the R script to the working folder
	TheFile=open(WorkingFolder+"RScript.R","w")
	TheFile.write(RScript)
	TheFile.close()
	
	# Launch the subprocess to run the R script to do regression
	subprocess.call(RPath+" --no-save <"+WorkingFolder+"RScript.R >"+WorkingFolder+"RConsoleOutput.txt")
	
	# Setup the canvas for the image
	canvas = Canvas(root, width=500, height=500, bd=0, highlightthickness=0)
	canvas.pack()
	
	# Add the photo to the canvas and wait for the user to click the close box
	image = Image.open(WorkingFolder+"OutputPlot.gif")
	photo = ImageTk.PhotoImage(image)    
	#photo = PhotoImage(file=WorkingFolder+"OutputPlot.gif")
	item = canvas.create_image(10, 10, anchor=NW, image=photo)

	# read the line with the parameters from the output text file
	TheFile=open(WorkingFolder+"OutputResults.txt")
	for i in range(6):
		TheFile.readline()
	TheLine=TheFile.readline()
	TheFile.close()
	
	# parse the line to get the A and B parameters
	TheTokens=TheLine.split()
	B=float(TheTokens[0].strip())
	A=float(TheTokens[1].strip())
	
	# create the equation for the regression to add to the chart
	Equation="y="+format(A)+"x"
	if (B>=0): Equation=Equation+"+"
	Equation=Equation+format(B)
	
	# add a label with the equation
	w = Label(root, text=Equation)
	w.pack()	

	# display the window and wait for the user to close it
	while True:
		root.update_idletasks() # redraw
		root.update() # process events
		time.sleep(.01)

except TclError: # called when the user presses the close button
	pass # to avoid errors when the window is closed

Take a look at the "C:/Temp" folder and you should see the following files:

  • Inputs.csv - This is the data file that R will read to create the regression equation
  • RScript.R - This is the script that was saved the by Python script above and then executed by R
  • OutputResults.txt - Contains the results of the regression including the coeficients for the linear regression equation so we can display them at the bottom of the wnidow.
  • Output.gif - This is the graphic that appeared in the wnidow.
  • RConsoleOutput.txt - This is the output from the R console that we would normally see in R or R-Studio and may contain important error information.

Debugging these scripts can be a bit challenging. If you have problems, one approach is to load the "R" script into R and run it to see if it is having problems. Also, make sure the file paths and folders are setup as needed.

3. Calling the Script from the Command Line

The script above have been setup to read the "Plot Title" and labels for the x and y axis from the command line. The script then adds some lines of code to the "R" script to create the variables for the title and labels. Try running the script from the command line and changing the title and labels.

4. Python Script 2: Python Calling Python

Below is the code for the Python script that will be added to ArcGIS as a tool script. Before creating the tool, lets run it in an IDE.

Before starting, you'll want to:

  1. Download the "Shapefile.py" file and save it into "C:/ProjectionPython/Resusable".
  2. Make sure the paths are correct for the versions of Python you are using.
  3. Download the Redwood Shapefile for California and place it in "C:/Temp". This shapefile contains data from the Forest Inventory Analysis database including the Height of redwood trees. It also contains data from the BioClim/WorldClim website on annual precipitation and other climate variables.

Now, run the script in the Wing IDE.

#####################################################################################
# Script to sublaunch a Python script from an ArcGIS tool.
# The tool will create a linear regression graph in ArcGIS using a shapefile.
# To test the tool:
# - Create a folder at "C:/Temp" for a working folder
# - Save the associated "California_Climate" shapefile into the folder
# - Make sure the paths below point to the appropriate versions of Python 
#   and the associated Python script to call R.
#
# Authors: Jim Graham and Andy Larkin
# Date: 4/22/2014
#####################################################################################

#####################################################################################
# Import standard packages
import subprocess

# Add a path to the reusable code
import sys  
sys.path.append("C:/ProjectsPython/Reusable")
import shapefile

#####################################################################################
# Setup the paths
WorkingFolder="C:/Temp/"
PathToScripts="C:/ProjectsPython/RegressionTool/"
PathToPython="C:/Python27/ArcGIS10.1/python.exe"
PathToPython="C:/Python27/python.exe"

# Variable to switch when we are running as a tool in ArcGIS
RunningAsATool=False

# Setup the default parameters (when running without being a tool)
TheShapefile=WorkingFolder+"CA_Redwood_MaxHeight.shp"
Field1="Height"
Field2="AnnualPrec"

#####################################################################################
# Get the parameters from the ArcGIS tool

if (RunningAsATool):
   import arcpy
   # The the user (and us!) know we are running)
   arcpy.AddMessage("Running Tools")
   
   # Get the parameters from the tool
   TheShapefile=arcpy.GetParameterAsText(0)
   Field1=arcpy.GetParameterAsText(1)
   Field2=arcpy.GetParameterAsText(2)
   
   # Print the parameters back to the tool for debugging
   arcpy.AddMessage("TheShapefile="+TheShapefile)
   arcpy.AddMessage("Field1="+Field1)
   arcpy.AddMessage("Field2="+Field2)
   
#####################################################################################
# Write out two of the attributes from the shapefile into an "Inputs" file for R

# Open the file for the Input data to R and write out the header
TheFile=open(WorkingFolder+"Inputs.csv","w")
TheFile.write("X,Y\n")

# Open the shapefile 
TheShapefile=shapefile.Reader(TheShapefile)

# get the list of fields (attribute column headings) from the file
TheFields=TheShapefile.fields

# find the index to each of the selected fields
FieldIndex=0
for TheField in TheFields:
   if (TheField[0]==Field1): FieldIndex1=FieldIndex
   if (TheField[0]==Field2): FieldIndex2=FieldIndex
   FieldIndex+=1

# get the records (attribute rows) from the shapefiles dbf file
TheRecords=TheShapefile.records()

# Loop through each row in the attributes    
for TheRecord in TheRecords:
    Value1=TheRecord[FieldIndex1]
    Value2=TheRecord[FieldIndex2]
    TheFile.write(format(Value1)+","+format(Value2)+"\n")

# Close the file
TheFile.close()
   
#####################################################################################
# Call the R script to create the chart and display it in a window

# Setup the labels for the title and axis
Title="Linear Regression"
XAxisLabel=Field1
YAxisLabel=Field2

# Create a parameter string for the command
Parameters=" \""+Title+"\" \""+XAxisLabel+"\" \""+YAxisLabel+"\""

# Create the command that will be sent to python to run R
Command="\""+PathToPython+"\" "+PathToScripts+"RRegression.py"+Parameters+ " >"+WorkingFolder+"PythonConsoleOutput.txt"

# Show the command for debugging
if (RunningAsATool): arcpy.AddMessage("Command="+Command)

# Open the process and wait for it to finish
TheProcess=subprocess.Popen(Command)
return_code = TheProcess.wait()
   
#####################################################################################
# Let the user know we are done

# Let the user know that this script finished successfully
if (RunningAsATool): arcpy.AddMessage("Done")

 

5. The ArcGIS Tool

The final step is to create a tool in ArcGIS and have it call the script. Create the tool with the follownig steps:

  1. In ArcCatalog, create a new "Toolbox" in the "RegressionTool" folder.
  2. In the new toolbox, add your script.
  3. Give the script a good name and make sure that it is set to "Store relative path names".
  4. Have the tool point to your "RegressionTool.py" script
  5. Add the following parameters:
    1. "Input Layer" as a "Feature Layer"
    2. "X" as a "Field". Set the "Obtained From" property for this field to the "Input Layer"
    3. "Y" as a "Field". Set the "Obtained From" property for this field to the "Input Layer"

Run the tool but don't be surprised if you have some issues. Go back and add message boxes and "AddMessage()" function calls until you find the problem. You can also go back through the steps above to make sure all 3 scripts are working.

When the tool works, you should see your graph appear in a "tkinter" window over ArcGIS. You'll also see a Python command window and the regular "Script" window within ArcGIS. On closing the "tk" window, the other windows should also close without locking up ArcGIS!

6. In Summary

The code above works and should be able to integrate a large number of different applications into ArcGIS. You'll still want to plan on problems with working directories and versions of Python, ArcGIS, R, and other packages and applications.

Additional Resources