Module 3: LiDAR Visibility

 Module 3: LiDAR Visibility

This weeks module was learning how to do visibility analysis within ArcGIS Pro. There are two types of visibility analysis which are viewshed analysis and line-of-sight (LOS) analysis. The viewshed analysis is trying to answer the question, "Which areas in a landscape are visible from an observer point?" Where given one or more locations, the output is a map of the visible areas. On the other hand LOS tries to answer the question, "Which segments are visible along a specific site line?" Where given one or pairs of locations, the output is a map of visible segments along a set of line. It is important to note that both of the analyses use elevation models, such as digital elevation models (DEM) or a triangulated irregular network (TIN).

During this weeks module we took four courses (Introduction to 3D Visualization, Performing Line of Sight Analysis, Performing Viewshed Analysis in ArcGIS Pro, Sharing 3D Content Using Scene Layer Packages). Each one of the courses were taken on the ESRI website where they explained what you needed to do and then did an exercise with the data they gave you in the beginning. 

The first exercise was Introduction to 3D Visualization course shows how the data can be seen in 3D mapping. There are two 3D maps which are the Local Scene and the Global Scene. The local scene is used more for looking at data within a city whereas the global scene is more of looking at things internationally, such as flight patterns. The next part of this course was understanding how to present the elevation within the 3D maps. There are three types: On the Ground, Relative to the Ground, and Absolute Height. As said on the ESRI website "Elevation types are a property of the layer and are dependent on the type of elevation surface that you choose in your scene." Also, you can also manipulate height variables using techniques such as cartographic offset and vertical exaggeration. Cartographic offset vertically adjusts the height value (or z-value) of the entire layer, raising or lowering all features in the layer by a given height. Vertical exaggeration is used to emphasize subtle changes in a surface. Another thing that was learned in this exercise was extrusion of 2D features to 3D features. Extrusion enables you to take your points, lines, and polygon features and stretch them vertically to create real-world 3D objects. There are fours ways to extrude data: Adding to the feature's base height, Adding to the feature's minimum height, Adding to the feature's maximum height, and Extruding to an absolute value. Lastly, in this exercise we learned how to add 3D symbology to our maps including trees, water, and buildings. Also, we learned how to make those features look more realistic such as moving water.

The next exercise was Performing Line of Sight Analysis course determines whether two points in space are intervisible. A line of sight calculates intervisibility between the first vertex (the observer) and the last vertex (the target) along a straight line between the two. A line of sight considers any obstructions provided by a surface or multipatch feature class. Visibility between these points is determined along the sight line. There are three general processes that are used to perform visibility analysis. First you need to determine observers and targets. Second construct sight lines. Third determine line of sight. It is important to note the color of a sight line indicates the locations where the surface is visible and where it is hidden.

The third exercise was Performing Viewshed Analysis in ArcGIS Pro course we learned how to modify the input features to model the visibility from a known vantage point. The tool creates an output that models the areas that are visible from given vantage points. The Viewshed is where you can symbolize the visible area from an observer point in a raster. The geoprocessing tool considers the height of the individual, along with any obstructions surrounding the point. The areas that are considered visible from the observer point will be indicated in the output raster. The Viewshed tool is controlled through fields that are added to the input data to control the observation point elevation values, vertical offsets, horizontal and vertical scanning angles, and scanning distances. This tool was able to help an analyst see where lights cross the landscape depending on the height and the radius of the light beam. This is important because you can change the height and see where more or less areas cross one another.

The fourth and final exercise was Sharing 3D Content Using Scene Layer Packages course we learned how to change the data and share it with people who have ESRI accounts. ArcGIS Pro allows you to investigate and visualize your data in an intuitive and interactive 3D environment from any angle or perspective. You can use a scene for inspection and exploration workflows, for visualization, for communicating analytical output, or for storytelling about real-world projects and scenarios. Once again, it is important to know if you want a local or global scene. Also, we learned that there are three steps or workflow to authoring your data: 1. Load your data, 2. Display 2D data as 3D layers, and 3. Convert 2D data as 3D data. Before presenting your data there are some tips and tricks you need to remember: 1. Have all your source data and the scene in the same coordinate system, 2. Structure your content and decide what the user must see, 3. Define an area of interest (AOI) for your scene, and 4. 3D symbology is required for feature layers to be published and shared. After making your data there is a benefit of sharing your data because a wide variety of people can see it and it can be shared easily. It is important to note when using ArcGIS Pro, you can save your scene layer and its data as a scene layer package (an SLPK file). To determine the most appropriate option for sharing scene content with your intended audience, you must answer the following questions: 1. With whom do you want to share the content? (You can choose to share content with the public or a specific group), 2. How do you want to share the content?, and 3. What type of content will be shared? (Every content item that is published to a portal can be shared). To upload a scene layer package and publish a hosted scene layer, you will perform the following workflow steps: 1. Sign in to your organization, 2. Open My Content, 3. Add the scene layer package on your computer as an item, 4. Type a title and tags that describe the scene layer package, 5. Check the box next to Publish This File As A Hosted Layer, and 6. Add the item.

All the courses were very useful because first you learn about what you are doing in the exercise and why you are doing it. Also, you get step-by-step instructions of what you need to do with the data that was given to you. Lastly, there was a quiz at the end to help you understand everything you learned.

Module 2: LiDAR (Wetland Delineation)

 Module 2: LiDAR (Wetland Delineation)

This weeks module was about using LiDAR data to create different models including Digital Elevation Model (DEM), Digital Surface Model (DSM), Canopy Density, Vegetation Height, and a LiDAR map. 

The first step was downloading data from a state website. This is important because if you know how to download the data you can use many different websites to get your LiDAR data. This data is shown in .laz file that will need to be converted to las data. To convert your data, you need to open up the geoprocessing tool “Convert LAS” and put in your data. This will create an output LAS dataset that is a .lasd data file. In order to open the data under the insert tab you click the down arrow of the New Map and click on New Local Scene. This will open the 3D dataset of LiDAR.

Before doing the second step, so that you can calculate the data, we need to make sure that we have the 3D analyst and Spatial Analyst extension by clicking Project and then Licensing. If the words are black that means you have access to the extension. Next, we need to calculate the forest height from the LiDAR point cloud. First, we need to get the geoprocessing tool “Point File Information” where we run the .lasd data that we previously created. Make sure that the coordinate system is the same as the .lasd data you got from the website. (It is important to note that many surveyors across the US use State Plane when they collect data). After that we need to create a DEM dataset. First you need to click you data under the contents pane and then click on LAS Dataset Layer. Then click on LAS Points arrow and change the points to Ground. Once you do that you search for the geoprocessing tool “LAS Dataset to Raster”. Next you need to select the .lasd file as the input and change the value field to Elevation. Under interpolation type you need to select Binning with cell assignment as Maximum, and Void Field Method as Natural Neighbor. The Output Data Type is Integer and Sampling Type is Cell Size. Lastly, you need to make the Sampling Value as 6 and Z Factor as 1. This produced the DEM dataset that was back and white. To change the color right click the data and click symbology to what you want it to look like on the map. To create a DSM dataset, you need to change the LAS Dataset Layer to Non-Ground. When using the geoprocessing tool “LAS Dataset to Raster” keep everything the same and it will once again show the data as black and white. After that to get the tree height you want to get the geoprocessing tool “Minus” and set the Input Raster 1 as the DSM file and the Input Raster 2 as the DEM file. This was able to show the height raster data, and you can look at the attribute table to see where the negative and positive values are located. Looking at the data shows the negative values are where the roads and shadows of the trees are located. Where the positive values are where the trees are located and the different heights.

The map below shows both the LiDAR data and the DEM data that was derived from the LiDAR data. This shows that the visible points are in different colors to represent the height of the topography in the area. It looks like the lowest height is a dark blue and starting in the northern portion of the data to a red in the southern portion of the data. Also, you are able to see where the roads are located with the help of the base map being topographic. Lastly, you can see the slight rises and dips within the point cloud data.

The third step was to calculate biomass density. This is done by a lot of different geoprocessing tools. The first geoprocessing tools is “LAS to MultiPoint” where you put the .lasd data from the beginning. To get the metadata that represents the bare earth you need to set the average point spacing from the attribute table from the Point File Information dataset that created as the value and set the class code as 2. To create the metadata for the vegetation it is the same value but the class code is 1. After that you want to convert the MultiPoint files to Raster with the “Point to Raster” geoprocessing tool. You will put the data you named for the bare earth as the input. Set the Value field to OID and set the Cell Assignment to Count. Lastly you use the average point spacing times 3 and round to the nearest whole number. For the vegetation you will do the same thing. Next you want to create a binary file where 1 is assigned to all values that are not null by using the “Is Null” geoprocessing tool for both raster files you previously created. The next geoprocessing tool is “Con” so that if a value of 0 is encountered it will accept it as true value and if it is 1 it pulls from the original raster. You will set the Input from the previous data tool and set Input true raster or constant value to 0. Lastly, se the Input false raster or constant value to the raster dataset. Do the same thing for the vegetation dataset. The “Plus” geoprocessing tool is next that combines the vegetation and bare earth count datasets to derive the overall density returns. To get the Plus result from integer to float you need to use the geoprocessing tool “Float” which will provide the true representation of density from the next tool. Lastly, to calculate the density we need to use the “Divide” geoprocessing tool where you set the Input raster to the count dataset for the vegetation and set input raster 2 as the Float result. Once again it shows up as black and white so you want to change the symbology to what you want.

The image below shows canopy density from the LiDAR data. The information being conveyed in the density map is the areas that have trees and the areas that are not located. This would be helpful to foresters because it shows how good the vegetation is in the area. The higher the number the better the vegetation will be because it is showing a higher reflective value. Also, it could show where roads and houses are located.

The fourth and final step was to create a chart. This is done by clicking on the Height (results from the Minus tool) and go to the Data tab and click Create Chart arrow and click Histogram. For the Histogram to show up you need to click on Band_1 that will show the value and count of the information from the shapefile.

The final image below shows the height of the vegetation and a histogram chart from the LiDAR data. The graph tells you that the highest count of tree height is 63.25 feet tall and it is almost a normal bell curve of the different heights on the landscape. Also, this says the area is growing roughly at the same time and if they needed to cut down trees for money they would be able to get more across the landscape. Lastly, the values tell you that there are some areas that are higher topographically and not just the trees themselves.

Module 1: Crime Analysis

 Module 1: Crime Analysis

This weeks module focused on how to use ArcGIS Pro to make crime analysis maps. There are three different types of maps that are used to represent crime, which are grid-based thematic, kernel density, and Local Moran's I. Grid-based thematic mapping overlays a regular grid of polygons on top of the crime events to produce a count of crime events per grid cell. Kernel density mapping is when points lying near the center of a search area are weighted more heavily than those lying near the edge. Local Moran's I is asking the question of if there are similar features nearby.

Grid-based Thematic

When making the Grid-based Thematic Hotspot Mapping you once again need to use the Spatial Join tool that shows the information from the 2017 homicides within the grid cells. The Select By Attributes tool you need to select the correct column so you can get the information you want which was any grid that had a homicide count greater than zero. The way I exported this feature was under the Data tab where I selected “Layer From Selection” where it creates the feature off what I selected. Next, we needed to get the top 20% of the features that were selected previously, and this is done by opening the attribute table sorting the Join_Count field and dividing the total number of records by 5 and making sure to rounding down. Since there is no selection tool for this you need to do it manually by clicking the first box and then hold the shift button to where you want to end to get all the numbers in between. After all of that we need to dissolve the features together. This is done by making a new tab that says Dissolve and using the field calculator to give the entire column the same value. Then find the Dissolve tool to create the multipart features. You can see it in the image below.

Kernel Density

First, we want to search for the Kernel Density Tool. As stated earlier make sure there are no spaces in your folders and your features. This was done similar to the maps we did for Washington DC kernel map. We need to make sure the output cell size and the search radius are in same which is in feet. After creating the map, we needed to only make two breaks based off the mean and the maximum value. This is done by right clicking the feature and hitting the symbology tab. Once you are in the tab you need to click the more button and select “Show Statistics” which will show all the information we need to make your break values. Next, we need to make the raster to a polygon but first need to use the Reclassify tool to reclassify the feature. After that we need to use the Raster to Polygon tool to same the feature to a feature class within the project geodatabase. Lastly, we needed to do the Select by Attributes and find the value of 2 which is three times the mean of the original kernel density. The image below that shows the kernel density from the 2017 hotspots.

Local Moran''s I

The last map we produced was the Local Moran’s I Hotspot which is supposed to use crime counts or rates aggregated with meaningful boundaries. Once again we used the Spatial Join tool between census tracts and 2017 homicides. Then we needed to create a new field for crime rate and create a field calculator where homicides per 1000 housing units. Next, we used the Cluster and Outlier Analysis (Anselin Local Moran’s I) tool and leave the parameters as their default settings. After that we needed to get the high-high clusters because it shows the areas that had a high among of crime per 1000 households. Lastly, we dissolve those features so it looks good on the map which you can see below.

Based on all the information above it appears that the kernel density map would be the best because it shows a smoother and more continuous map instead of breaking it into blocks or zones. Some lessons I learned when doing this module was when setting up your environment setting make sure the extent and the mask are the same. If it is not, then the feature class will show up empty. Also, in the Kernel Density (Spatial Analyst) tool you need to make sure there are no spaces in either the folder the information is saved in and the feature class.

Module 6: Geometries

 Module 6: Geometries

This weeks lab was about how to use geometries within Python and ArcGIS Pro. We wrote a script that creates a TXT file and write to it the coordinates and object IDs for vertices in a shapefile. This lab had us use two for loop statements. It was more difficult to understand how to make two for loop statements. Working with geometries is very time consuming but we learned about how to use geometry tokens. Geometry tokens provide shortcuts to specific geometry properties of individual features in a feature class. Another important factor we learned was when reading geometry types (polygons and polylines) requires extra code because an array of point objects is returned for each feature. As a result, an extra iteration is required to interact with the array first before you can get to the points that make up the array. This is where the for loop statements came in and made it easier to get all the information we may need. So that we did not have to do another for loop statement we did the getPart() method. The getPart() method returns a single point object for one single-part feature. If it is a polyline or polygon, it returns an array of point objects of the different features you are trying to get. In short, the getPart() method allows you to access and look through the different parts within a single geometry object. The hardest part I had was getting the information to print in the textfile and I remembered using the Try statements so that I could do the except and finally statements so that I could get the information to print. Being able to use those statements were important because it could show me where the issue was happening before things printed into the text file. Lastly, my other issue was figuring out the .write() statement so the information did actually write into the textfile. That is when I created the line I wanted to write and then put it the parentheses. The image below shows a portion of the information into the textfile.

This image below is the flowchart of how I created the script for the information I wanted for the rivers shapefile. Once again it was difficult to show the for loops and how they were being represented.

This module was probably the most difficult one because of the two for loop statements and getting the information to write into the textfile. Once I remembered the try statement I was able to make sure I got all the information correctly into the textfile.

Module 5: Explore and Manipulate Data

 Module 5: Explore and Manipulate Data

In this weeks module the focus was on how we can explore and manipulate data that are within ArcGIS Pro. The script creates a new file geodatabase, copy feature class data to the new file geodatabase, create a Search Cursor for the cities feature class, and create a new dictionary.

First we needed to create a new file geodatabase that goes into the results folder. It is important to note that before any of this can begin you need to make sure you import arcpy and import env. Also, to make sure you can create the geodatabase more than once you need to have the overwrite output to True. If you are getting an error message if trying it again it is because ArcGIS Pro is open and it needs to be closed before running it in Python IDLE. If you run it in the Notebook portion in ArcGIS Pro you do not seem to have that problem. The image below shows the outcome of the script that was written.

The next portion of the script was creating a feature class list and copying that into the new geodatabase that was previously created. When making sure the data gets copied you need to use the Describe() function. Also, to get all the data copied you need to make sure to the script as the following arcpy.CopyFeatures_management(fc, outputEnv + "/ejd31.gdb/" + fcdesc.basename). This piece of code makes sure everything gets copied and that the basename of the feature shows up when it gets copied. In the image below shows that each feature got copied and how long it took to get the information to copy into the new file geodatabase.

After that I needed to create a Search Cursor that only printed the Name, Feature, and Pop_2000 only for the county seat within the cities shapefile. I knew how to write the code for one row in the attribute table but was not sure for three. I made the fields code as fields = [“NAME”, “FEATURE”, “POP_2000”] and the sqlQuery = “FEATURE = ‘County Seat’”.  I used the arcpy.da.SearchCursor because I did not want to change the table but to just find the specific rows and print them. Also, you need to make sure to use a for loop statement to make sure only the specific rows get printed and not the other ones in the attribute table. The image below shows all the features that were only County Seat with the name and population of those features.

     

The last portion of the script was creating a new dictionary based off the information from the cities shapefile attribute table. The dictionary needs to focus on the name and population of the county seat features. Another issue I was having was trying to get the for loop statement to create a dictionary for an empty dictionary. I knew how to make the dictionary but was not sure how to make the statement work. I did try different versions and none were working for me. I think asked the discussions page and Serenity helped me with the script needed to have another line of code saying with arcpy.da.SearchCursor(fc, fields, sqlQuery) as cursor: The information in the parentheses is based off the previous script above. The difference is that I needed to add the specific row numbers for the keys and values that I wanted represented in the dictionary. Lastly, I chose the dictionary variable[keys] = variables which was county_seats[name] = population. The image below shows the new dictionary with just the information of the Name and Population of the county seats in the cities shapefile

An important note is that to get the copying features to print separately the code is print(f”Copying: {fc} \n”). This was able to help me get them to print one at a time instead of all of them at once over and over again. Knwoing how to get the information to print properly will help know what is being copied in the first place. The script creates a new file geodatabase, copy feature class data to the new file geodatabase, create a Search Cursor for the cities feature class, and create a new dictionary. The image is of the flowchart that I used for the script that was written. 

Module 4: Geoprocessing

 Module 4: Geoprocessing

In this weeks module the focus was on geoprocessing tools, creating our own ModelBuilder and toolboxes. The first portion of the assignment was using the ModelBuilder to clip sols within the basin shapefile, select a specific attribute within the soils shapefile, and erase the "Not prime farmland" within the soils shapefile. Within the ModelBuilder the you can drag in your data and geoprocessing tools. These tools are similar as ones that I have used within the maps. The ModelBuilder is important because you can create what you want done all at once instead of each step separately. In the image below it shows how I added the basin and soils shape file. Next added in the clip analysis tool where I said I wanted the soils only within the basin shapefile. Which creates the shapefile which I called soil_Clip. Then to erase the "Not prime farmland" portion I needed to add the Select analysis tool which I created a script. Lastly, this created Soils_Selected.shp.

This image below shows what the ModelBuilder produced which was all the areas that were not apart of the "Not prime farmland" shapefiles.

The next portion of this module was writing script that adds XY coordinates, creates a 1000 meter buffer, and dissolve the buffers into a single feature for the hospitals shapefile. When designing the script that adds the XY coordinates, buffers, and dissolve the hospital shapefile I needed to make sure there is arcpy and env workspace. The workspace is the location of where the data is coming from to add the information to the shapefile. Next, I made sure to overwrite the output features so the code could be done repeatedly. The first portion of the code was the XY coordinates on the shapefile by creating the shapefile as the in_features. To make sure the shapefile gets the XY coordinates the code needs to be “arcpy.management.AddXY(in_features)”. For the buffer portion I needed to write “arcpy.Buffer_analysis” Within this code I need to make sure to put in the buffer zone and where I want the shapefile to go which was the results tab in Module. Lastly, for the dissolving portion it is the same code as the buffer analysis, but I needed to make sure to add two open string portions and one that says ‘ALL’. This will make sure all the areas that overlap dissolve and create a whole new shapefile. If I did not want to create a new shapefile, I could name it the same shapefile as above. Another thing that needed to be added was the GetMessages() function so you can see the start time and if the script was successful in running. I put this in after the the other print option. The code was written as "print(arcpy.GetMessages())". Within ArcGIS Pro you do not necessarily need to add arcpy but I still added “import arcpy” in case the script runs in IDLE. Also, to make sure things are overwritten you need to make sure the box in clicked in the options tab under geoprocessing and write the code “env.overwriteOutput = True”. Lastly, to make a space within your script you can use “\n” within the print portion of the script. The image below is the result of the script that I wrote to add the XY coordinates, 1000 meter buffer, and dissolve the buffer for the hospitals shapefile.

This weeks module was very useful because it is able to learn how to use ModelBuilder to make shapefiles all at once instead of one at a time. Also, I learned how to do Batch processing that can help me do a lot of data all at once. Lastly, I was able to learn how to create a script that can add XY coordinates, add a buffer, and dissolve that buffer instead of doing it through the geoprocessing tools individually.

Module 3: Debugging and Error Handling

 Module 3: Debugging and Error Handling

In this weeks module the focus was on how to debug code and understand what the error messages are saying to the person writing or using scripts. There are different errors that can be seen when writing script. The first common error message is syntax error as known as the "grammar" of the code. This includes the spelling, punctuation, and indentation. On the other hand there are run-time errors that come up after the code gets run. Some examples are NameError, TypeError, and LicenseError. When doing this week we did three different scripts that we either fixed or did a try-except statment.

In the first script we had to look for two errors that were within the code. When trying to figure out the errors you can either click the "Run Module" or "Check Module" within IDLE. For me the easiest one was to save the script and clicking "Run Module". Then it comes out saying what the error is and what line it is located on. The first error was a NameError that within Line 5 the name 'fc' is not defined. When looking at the rest of the code it showed that the name 'fc' needed to be capitalized. The second error was another NameError in Line 8 where the name 'field' was not defined. Once again looking at the rest of the script I was able to notice I needed to add an 's' to the end to match other portions of the script. The picture below is the output of the script that was written.

The second script there were several errors or exceptions. Once again it was easy to save the script and click "Run Module" to find the first error. The first error was a NameError in Line 6 that indicated 'Filepath' is not defined. When looking in the rest of the code you can see it needed to be 'filepath' in the parenthesis. Next was an OSError which indicates that the file path in Line 6 is not connected to anything on the computer. I fixed the OSError to filepath ="S:/GISProgramming/Module3/Data/ Mod3_Data/TravisCountyAustinTX.aprx". The next error was a NameError in Line 8 saying 'arcp' is not defined and I noticed it was the dataset 'arcpy' that needed to fixed. Afterwards there was another OSError found in Line 7 and noticed one of the slashes were in the wrong way and that I needed to add more to the dataset file path. Then the next error was AttributeError that says 'spatialRefernce" does not exist. All I had to do was spell it correctly as "spatialReference." The next error was a TypeError that says 'type' object is not iterable in Line 13 and all I needed to do is add a 's' to the end of map. After fixing all the errors and exceptions it finally produced the outcome of the spatial reference and the four shapefiles within the ArcGIS Pro map which is seen in the picture below.

The third and final script that was written had errors but we needed to do a try-except statement. This statement allows a script to run successfully which helps with seeing the exceptions within the code and “trap” it so the code can continue run. When including the try-except statement you want to run the code to see what the error is that is stopping the code from running. First you want to put “try” statement before the code that has the error(s). Next you want to then put the “except” statement below the error and handles the error. Then you will run the code again to see if there are any other errors in the code. If there is another error, you can include in the current “except” statement or make another. Lastly, when you have all the errors it should be able to run the code and explain to someone why there were some errors as well as where they were located. I have included the output of both the parts and a flow chart of how to do Part A of the try-except statement.

In this module it was very helpful to learn about the different errors so that in the future my script runs without issues. Also, if I do run into issues I can fix it easily by being able to read what the errors and fixing them. Lastly, learning how to do the try-except statement lets the code run even if there are some errors that you might not necessarily know how to fix at the moment.