Understanding ArcGIS Data Management
Explore the fundamentals of ArcGIS data management, including distinguishing between geodatabase feature classes, raster classes, and associated files, working with layers and visualizations, saving data and map documents, and utilizing MapDocument in Python scripting for data display manipulation. Discover the intricacies of datasets and data drilling techniques within ArcGIS.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
Data Dr Andy Evans
This lecture Finding data Editing data Useful arcpy bits Important: Note that in these slides we've cut the names down to fit cleanly on the slides; if you want to use kwargs look up the originals in the docs.
Arc files In general in Arc we can distinguish between: Geodatabase Feature Classes / Raster Classes and the associated files (for example, shapefiles are wrapped as Feature Classes). These contain the data and associated shape information. Layers, the visualisation components, with symbology etc. These are generally linked to a specific Feature Class/Raster Class. To deal with data we deal with Feature Classes; Visualisation, Layers.
Components The application (not generally available in Python) What we think of as a map file is the "MapDocument" this includes the current map data and associated toolbars. ArcDesktop can either show a single Map (data view) or a set of Maps and associated layout (layout view). Whichever is currently shown is the "active view". Each map is made up of multiple Layers. These may be associated with Layer files, but will certainly be associated with a spatial dataset, for example a Feature Class (vector data) or Raster Class. Spatial datasets can be viewed as either Layers, or as Attribute Tables. Each object in a Feature Class is a Feature or a row in the Attribute Table. Features/Feature Classes/Attribute Tables have columns called Fields. A particular Feature will have a Value in a Field.
Datasets By and large there's little distinction between how you refer to things except when saving as files: c:/geodatabase.gdb c:/geodatabase.gdb/dataset c:/dataset.shp c:/dataset.dbf c:/dataset.lyr # Features or table, depending on use # Geography + (potentially) attributes in associated files # Just table # Layer
Saving Layers and MapDocuments have options to save: .save () .saveACopy (file_name, {version}) For featureClasses, use: arcpy.FeatureClassToShapefile() To open a file, you generally just use the filepath in the tool you want.
MapDocument You can drill down from the MapDocument to get at data. More usual to get hold of the MapDocument for changing data display. mxd = arcpy.mapping.MapDocument("map.mxd") mxd = arcpy.mapping.MapDocument("CURRENT") If you change the display, it may not update, and you may need to call arcpy.RefreshActiveView(), RefreshTOC(), RefreshCatalog().
Drilling down Drilling from the map document directly to the data in total is pointlessly complicated in Python. mxd = arcpy.mapping.MapDocument("CURRENT") ldf = arcpy.mapping.ListDataFrames(mxd)[0]: lyr = arcpy.mapping.ListLayers(mxd, None, df)[0] src = lyr.dataSource arr = arcpy.da.FeatureClassToNumPyArray(src, "*") print(arr)
Finding data Use list methods to access objects directly. Use geoprocessing tools to get and use objects directly. Use selections Use cursors First, a few list functions...
List functions ListFields(dataset, wild_card, field_type) Returns a list of fields found in the input value ListIndexes(dataset, wild_card) Returns a list of attribute indexes found in the input value ListDatasets(wild_card, feature_type) Returns the datasets in the current workspace ListFeatureClasses(wild_card, feature_type, feature_dataset) Returns the feature classes in the current workspace ListFiles(wild_card) Returns the files in the current workspace ListRasters(wild_card, raster_type) Returns a list of rasters found in the current workspace ListTables(wild_card, table_type) Returns a list of tables found in the current workspace ListWorkspaces(wild_card, workspace_type) Returns a list of workspaces found in the current workspace Returns a list of versions the connected user has permission to use ListVersions(sde_workspace) Each can take a wildcard search and some a feature type, e.g. fcs = arcpy.ListFeatureClasses("G*", "polygon") Geoprocessing tools exists for copying, deleteing, altering or building each; mainly in the management toolbox.
Finding data Use list or search methods to access objects directly. Use geoprocessing tools to get and use objects directly. Use selections Use cursors For example arcpy.MakeQueryTable_management() will make a table based on a SQL query (more later on these). A overview of a few elements you might come across when using tools...
FeatureSets/RecordSets/TableViews Occasionally tools demand/will take FeatureSets/RecordSets/TableViews. These are memory-based copies of Feature Classes / Tables that offer a lightweight and efficient alternative to using the whole table. feature_set = arcpy.FeatureSet("c:/base/roads.shp") record_set = arcpy.RecordSet("c:/base/roads.shp") table_view = arcpy.MakeTableView_management () # See docs for params. feature_set = arcpy.FeatureSet() feature_set.load("c:/data/roads.shp") Result objects can also contain Feature and Record sets.
SpatialReference Include projection details, resolution, etc. Not unusual to need these for tools: spatial_ref = arcpy.Describe(featureclass).spatialReference
Extent arcpy.env.extent used by a variety of tools to determine which areas to process. Can be set in a variety of ways for the whole map: arcpy.env.extent = "MAXOF" arcpy.env.extent = arcpy.Extent(-107.0, 38.0, -104.0, 40.0) arcpy.env.extent = "-107.0 38.0 -104.0 40.0" Individual features / selections may also have an extent variable.
Finding data Use list or search methods to access objects directly. Use geoprocessing tools to get and use objects directly. Use selections Use cursors
Selecting When a selection is made, tools should (check) only work with selection. : Returns a layer's selection as a Python set of object IDs. layer.getSelectionSet() setSelectionSet(method,oidList) : Sets a layer's selection using a Python set of object IDs. layer.getSelectedExtent() : Gets the extent of the selected features. NEW: new selection from the oidList. DIFFERENCE : selects the features that are not in the current selection but that are in oidList. INTERSECT: selects the features that are in the current selection and the oidList. SYMDIFFERENCE: selects the features that are in the current selection or the oidList but not both. UNION : selects all the features in both the current selection and those in the oidList. Object IDs can be got using the OID@ token in a cursor, for example (see later). Selections also possible from Tables: http://desktop.arcgis.com/en/arcmap/latest/analyze/arcpy-mapping/tableview-class.htm
Selecting by attributes arcpy.SelectLayerByAttribute_management(layer/view,{select},{where}) NEW_SELECTION ADD_TO_SELECTION REMOVE_FROM_SELECTION SUBSET_SELECTION Replace any existing selection (default). Add to current selection / select If no selection exists, this option has no effect. Only records common to current and new selection are selected. SWITCH_SELECTION Records that were selected are removed from the selection; records not currently selected are added to the selection. CLEAR_SELECTION Clear.
SQL Selections, cursors, and some tools take SQL. Expressions have to be strings, but also contain quotes. Simplest to triple quote outside: arcpy.SelectLayerByAttribute_management(lyr,"#",""""roadclass" = 2""") i.e. WHERE "roadclass" = 2 However, in some cases, the quotes need to be different. Personal geodatabases use []. Given this, do, in doubt: fieldname = "roadclass" whereclause = """{} = 2""".format(arcpy.AddFieldDelimiters(fc, fieldname)) arcpy.SelectLayerByAttribute_management(lyr,"#",whereclause)
Select by geography arcpy.SelectLayerByLocation_management( layer, {overlap}, {features}, {distance}, {select}, {invert})) Select by whether features in the layer overlap / are contained within a set of features. Select by distance from a set of features. INVERT : sets to e.g. find layer features not overlapped by features.
Finding data Use list or search methods to access objects directly. Use geoprocessing tools to get and use objects directly. Use selections Use cursors
Find and edit data Both operations are commonly easiest done through cursors. Cursors can be imagined as sub-sets of tables, or a pointer than leaps between relevant rows in a table. Arc has three: arcpy.da.SearchCursor() Read-only access arcpy.da.InsertCursor() Inserts rows arcpy.da.UpdateCursor() Updates or deletes rows Note the location in the da library. There are older arcpy direct cursors, but these are less efficient.
Setup arcpy.da.SearchCursor(intable,fields,{where},{spat_ref},{explode},{sql}) arcpy.da.InsertCursor(intable,fields) arcpy.da.UpdateCursor(intable,fields,{where},{spat_ref},{explode},{sql}) Field names can be * but this will slow things down a lot. Can be tuples or lists. Where clause is SQL. SQL clause is for additional ordering of results, etc. Explode causes complex shapes to be listed as one constituent point per row. Spatial reference can be used to reproject data during reading and writing. See: http://desktop.arcgis.com/en/arcmap/latest/analyze/python/setting-a-cursor-s-spatial-reference.htm
SearchCursor fc = 'c:/data/mygeodatabase.gdb/schools' fields = ['COUNTY', 'SCHOOL_NAME'] where = """"COUNTY" = "WEST_YORKS"""" sql = 'ORDER BY SCHOOL_NAME' for row in arcpy.da.SearchCursor(fc, fields, where, sql_clause=sql): print(u'{0}, {1}'.format(row[0], row[1])) del cursor # To free up memory. Note that after we miss optional params we have to use kwargs. The "u" forces the text into Unicode, just incase there are non-ascii characters. Note that only two fields are in each row returned - the cursor filters out the others. See also arcpy.MakeQueryTable_management()
This lecture Finding data Editing data Useful arcpy bits
InsertCursor fc = 'c:/data/mygeodatabase.gdb/schools' fields = ['COUNTY', 'SCHOOL_NAME'] c = arcpy.da.InsertCursor(fc, fields): row_tuple = ("West Sussex", "Nutbourne Academy") c. insertRow(row_tuple) del cursor Other fields get filled with default values.
UpdateCursor fc = 'c:/data/mygeodatabase.gdb/schools' fields = ['SCHOOL_NAME'] where = """"SCHOOL_NAME" = "Coal Hill School"""" for row in arcpy.da.SearchCursor(fc, fields, where): row[0] = "Coal Hill Academy" cursor.updateRow(row) del cursor Also cursor.deleteRow() for deleting.
Cursors Search and update also have a manual next() method. If no more rows, a StopIteration exception is returned. cursor.reset resets back to start. If the fields are objects, you can drill into them. For example, a SHAPE field may contain a point object with a .X and .Y value. Update and insert cursors will lock a table for editing. This only ends when: 1) with statement concludes 2) reset() called 3) Cursor runs through completely 4) Cursor is deleted using del(cursor). If you have trouble running these when the data is displayed, remove it from the map.
Getting shapes Use "SHAPE@" as field in cursor (OID@ gets the ObjectID field). SHAPE@XY will return a tuple with X,Y or centroid values. Full list of potential tokens at: http://desktop.arcgis.com/en/arcmap/latest/analyze/python/reading-geometries.htm
Features may be: Polygon Polyline PointGeometry MultiPoint Getting shapes These contain Point objects as an arcpy.Array (except PointGeometry which contains a Point). If multi-part (partCount; getPart() ) this will be an array of arrays. Polygons with holes come as one array, with the external (clockwise) polygon first, then Nones between each internal (anticlockwise) ring. Each has a SpatialReference (includes, e.g. coordinate system), plus a variety of methods to find centroids, areas, etc. Also include method to get the shape as JSON. Will also take in other geometries for results like clip, contains, union, etc. Some of these you can also get directly by using search field tokens, for example SHAPE@AREA amd SHAPE@JSON.
Examples from docs fc = "c:/data/gdb.gdb/roads" cursor = arcpy.da.InsertCursor(fc, ["SHAPE@"]) array = arcpy.Array([ arcpy.Point(5997611.48964, 2069897.7022), arcpy.Point(5997577.46097, 2069905.81145) ]) polyline = arcpy.Polyline(array) cursor.insertRow([polyline]) cursor = arcpy.da.InsertCursor(fc, ["SHAPE@XY"]) xy = (5997594.4753, 2069901.75682) cursor.insertRow([xy]) Note that shapes may be adjusted to work as they go in (e.g. polygon orders reversed).
Binary data files With cursors: http://desktop.arcgis.com/en/arcmap/latest/analyze/python/data-access-using- cursors.htm
To add a field, use: Fields arcpy.AddField_management() / AddFields_management() Field types: http://desktop.arcgis.com/en/arcmap/latest/manage-data/geodatabases/arcgis-field-data-types.htm To find a field (and other field info): desc = arcpy.Describe(layer).fieldInfo field = desc.findFieldByName (field_name) To delete a field, use: arcpy.DeleteField_management() To change values based on a calculation, use: arcpy.CalculateField_management() / CalculateFields_management() Calculations can be written in Arcade: a simple language for geoprocessing expressions. https://developers.arcgis.com/arcade/ For other operations, see: http://desktop.arcgis.com/en/arcmap/latest/tools/data-management-toolbox/an-overview-of-the-data-management-toolbox.htm
Useful tools arcpy.Merge_management(): merge spatial data to form a new dataset. arcpy.Append_management(): same, but add data into one dataset. Both take arcpy.FieldMap() / FieldMappings: Make rules for splitting and merging field data. arcpy.JoinField_management(): join two tables as relational database.
Edit sessions Replicate editing as a user. Allow multiple simultaneous editors. Allows rollback of edits. Easy way: with arcpy.da.Editor(workspace) as edit: # Make changes Editing closed one exiting the with.
Full session edit = arcpy.da.Editor(workspace) edit.startEditing(False, True) edit.startOperation() # Do stuff edit.stopOperation() edit.redoOperation () edit.undoOperation () edit.stopEditing(True)
This lecture Finding data Editing data Useful arcpy bits
Useful arcpy functions Geometries stored in Feature Classes can export to JSON/GeoJSON: j = shape.JSON To convert JSON/GeoJSON to shapes with unknown projection: shape = arcpy.AsShape(j) Can also process more than one with: arcpy.FeaturesToJSON_conversion() arcpy.JSONToFeatures_conversion()
Useful arcpy classes Graph / GraphTemplate: Area graphs Bar graphs (Horizontal and vertical) Min/Max Boxplots Bubbleplots Histograms Line Pie Polar Scatter http://desktop.arcgis.com/en/arcmap/latest/analyze/arcpy-classes/graph.htm http://desktop.arcgis.com/en/arcmap/latest/analyze/arcpy-classes/graphtemplate.htm
Statistics arcpy.Statistics_analysis(input_table, output_table, {statistic}, {field}) SUM MEAN MIN MAX RANGE STD COUNT FIRST LAST
Mapping library Anything to do with the look of maps and map export (e.g. pdf; old versions of Arc; image formats), repair lost data paths etc. http://desktop.arcgis.com/en/arcmap/latest/analyze/arcpy-mapping/introduction-to-arcpy- mapping.htm arcpy.mapping.ExportToPDF(mxd, r"C:\outputs\map.pdf")
Adding data to a map mxd = arcpy.mapping.MapDocument("CURRENT") df = mxd.activeDataFrame newlayer = arcpy.mapping.Layer("crime_sorted.shp") arcpy.mapping.AddLayer(df, newlayer,"TOP") #BOTTOM or AUTO_ARRANGE Remember to then save the map file: mxd.saveACopy(r"m:\GEOG5790M\map.mxd")
Symbology Information on how to render a layer. Read information: sym = lyr.symbology To apply a symbology in Python you can only take one manually applied to another layer and copy it, then adapt it: with arcpy.mapping as am: newlayer = am.Layer("crime_sorted.shp") oldlayer = am.Layer("albertsquare/buildings.lyr") am.UpdateLayer(df, newlayer, oldlayer, True) newlayer.symbology.valueField = "Join_Count" newlayer.symbology.addAllValues() am.AddLayer(am.MapDocument("CURRENT").activeDataFrame, newlayer,"TOP")
Mapping library Repairing data paths is a popular use: http://desktop.arcgis.com/en/arcmap/latest/analyze/arcpy- mapping/updatingandfixingdatasources.htm Functions exist for, for example, finding and replacing all workspace addresses in a set of maps.
Spatial Analysis Mainly for defining parameters and operations to go with raster spatial analysis geoprocessing tools (rather than containing the tools themselves). For example, arcpy.sa.ApplyEnvironment() applies current env conditions to a Raster.
Map Algebra Raster operations. Cast to Raster and cell-by-cell operations occur. (Remember you may need to checkout the sa library) outRaster = (Raster("rasterdata1") + Raster("rasterdata2")) outRaster = arcpy.sa.Slope("rasterdata1" * 2)
What you can't do in arcpy Arcpy is limited to the functions and classes it has, which is a subset of the ArcObjects framework. For example, it is hard to drill down through objects to data, because cursors are the preferred interaction route (saver than direct data access). However, the whole framework is there and registered with Windows through the Component Object Model. This means, in theory, we should be able to get at it programmatically.
The COM Remember that an interface is a set of promises about the kind of functions/methods a class will contain. They allow code to use other code without knowing anything other than the method names. In Microsoft languages, reassignment of interface labels is key. You get an object and look at it through the perspective of one of the interfaces it matches. You do this by casting it to the interface. It is used to hold together a great deal of the Component Object Model (COM) at the heart of Microsoft software.
Example COM-style code IMxDocument mxDocument = (IMxDocument)application.getDocument(); IMap mxDoc = (Imap)mxDocument.getFocusMap(); Note the manifest typing, but the types are interfaces rather than real classes. The objects are of classes that implement multiple interfaces.
Accessing the COM However, you can gain access to this through comtypes, a library for interacting with the COM framework, and Mark Cederholm's Snippets module which works with this. This should be a last resort; chances are, if you've had to resort to this, you've missed something and/or a different language would be better.
