Creating Effective Landsat Timelapses

When asked to describe a satellite time-lapse, it is extremely likely that one would extrapolate some sort of time-series creation that shows gradual change with the use of natural colour images. It is however in this method of displaying satellite imagery, that a great deal of information is lost. To elaborate, individuals viewing such time-series creations cannot interpret quantifiable descriptors that may aid in conveying a message. Take for example the time-lapse provided below.

For what it is, a simple visual attempt at identifying change, the time-lapse above does an effective job at conveying change. Having said this however it does not provide the viewer with quantifiable observations, like the area developed per year, or even the ratio of land to water, as an example. Such statistics would be extremely useful in not only further illustrating a point, but also further encapsulating a viewer in the creation made. With this being said there are methods in which information can be derived from satellite imagery to provide viewers with a greater level of detail.

Within the realm of remote sensing, or the field in which data are collected without having a physical presence, there are a number of methods that allow individuals to interpret quantifiable data from the images one may capture. The most significant method being, a change detection. As a brief summary of what a change detection is, it is a temporal analysis between two, or more, periods of time, where either multi-spectral, or hyper-spectral images, and subsequent band combinations, are used to identify quantifiable areas of change between temporal foci. A change detection is of significant importance as it imparts the viewer with a definitive measure of change, far from the interpreted observation of change found within a regular timelapse. Taking a step back, one could quite easily identify, and even argue that in order to create a more effective timelapse, one could combine aspects of a change detection, creating an enhanced, and ever effective geo-visualization. Such a creation would bridge the gap between the lay, and the academic.

But such a thought begs the question; how could this be accomplished? Better yet, how could a timelapse, comprised of hundreds of composite spectral images, that displays a quantifiable characteristic, be created in an timely, and effective manor?

Put simply, programming is the answer to such questions. 

By creating a number of programs, an effective timelapse can be created that can be an effective evolution, from what was previously used. In an attempt to elaborate such a plan, such an attempt follows three steps:

  1. Create a program, a web-driver, that downloads an entire dataset, in this case all Landsat 4 – 5, imagery for a specific row, and path, in the absence of an API, or database. For an effective geo-visualization to be created, a great deal of data are needed.
  2. Instantiate a program, that opens each file, in this case Landsat image, and pulls three bands to be used to create a composite image that will better show a specific type of geographic change..
  3. Develop a program, that classifies imagery based on a predefined signature class, a file containing information on predetermined training classifications, that will then find, and append, the total area of a classification to a CSV file.

Put simply, the series of programs that will be created will create the intended geo-visualization, as well as do so in an efficient manor.

Attached are two links to a Dropbox folder, both links are direct connections to both programs needed to, download, and compose the images in focus.

Image Downloader:

https://www.dropbox.com/s/63ig83vd30ydk0a/Composite_Composer.py?dl=0

Image Composer:

https://www.dropbox.com/s/63ig83vd30ydk0a/Composite_Composer.py?dl=0

*Note, that in-depth comments have been provided for each stage, within each program. A general idea, as well as flow of each program should be quite easy to interpret.

It should be noted however, in the process of creating such a geo-visualization, a significant error arose. To elaborate, in the process of creating the final program, the GIS software, in this case ArcGIS, through the manipulation of ArcPY, failed on multiple attempts to read a signature file, and create a classification for each image. After much research it was concluded that an internal, however still unresolved issue was the main cause. As such, the final, and most important program could not be created.

The images created, were then edited in Adobe Lightroom, a image manipulation software, to remove any significant errors, and stitched together using Adobe Premiere Pro. Additional peices of information, such as titles, summaries, as well as dates were added within Adobe After Effects. The final geo-visualization can be seen below.

In summary, the Landsat timelapse created is extremely unique, it can safely be argued that there are no other geo-visualization methods like it currently present. In regards to what was attempted to be solved, it however did not accomplish all the tasks set out. From the perspective of the viewer, it is an easier method of data interpretation, however, it still lacks the quantifiable data aspect.

With all of this being said however, it is hoped that in the future, when previously mentioned bugs in software are fixed, another, more successful attempt at this geo-visualization can be made.

Time-series Animation of Power Centre Growth in the Greater Toronto Area for the Last 25 Years

By: Jennifer Nhieu
Geovisualization Class Project @RyersonGeo, SA8905, Fall 2018

Introduction:

In 1996, there were 29 power centres with 239 retail tenants accounting for just under five million square feet of retail space (Webber and Hernandez, 2018). 22 years later, in 2018, there are 125 power centres with 2,847 retail tenants accounting for 30 million more square feet of retail space (Webber and Hernandez, 2018). In addition, power centres expand in an incremental manner, either through the purchase and integration of adjoining parcels or the conversion of existing parking space into new stores (Webber and Hernandez, 2018). This development process often leads to retail centres becoming “major clusters of commercial activity that significantly exceed the original approved square footage total” (Webber and Hernandez, 2018, pg. 3).

Data and Technology:

To visualize this widespread growth of power centres from 1996 to 2017, a time-series animation map was created on Kepler.gl. (beta version) using power centre growth data provided by the Centre for the Study of Commercial Activity (CSCA) at Ryerson University, who undertakes an annual field survey-based inventory of retail activity in the Greater Toronto Area. Kepler.gl was created by Uber’s visualization team and released to the public on the summer of 2018. It is an “open source, high-performance web-based application for visual exploration of large-scale geolocation data sets. Kepler.gl can render millions of points representing thousands of trips and perform spatial aggregations on the fly” and is partnered with Mapbox, a “location data platform for mobile and web applications that provides open-source location features”. (Uber Technologies Inc., 2018 and Mapbox, 2018).

Methodology:

The data provided by the CSCA includes information on the shopping centre’s name, a unique identification code for each power center, the longitude and latitude coordinates for each power centre, its square footage information from the year it was built to 2017 and etc. The data had to be restructured on Microsoft Excel to include a pseudo date and time column, which would include the years 1992 to 2017, as well as 1-hour time intervals to allow Kelper.gl to create an animation based on date and time. The table below is an example of the data structure required to create a time animation on Kepler.gl.

Table 1: Data structure example*

Table 1: Data structure example*
*The data in this table has been modified due to confidentiality reasons.

Below is the time series animation set up for visualizing power centre growth on Kepler.gl. This process can also be replicated to produce another time-series animations:

  1. Visit https://kepler.gl/#/
  2. Click Get Started.
  3. Drag and drop .csv file onto application.
  4. Hold and drag to navigate to the Toronto GTA.
  5. On the contents bar, click + Add Layer on the Layers
  6. Under Basic Layer Type, click Select A Type, then select Point.
  7. Under Columns, Lat* = COORDY and Lng* = COORDX.
  8. Under Color, click Color Based On, then select SQ FT.
  9. Under Color, click the color scheme bar, select a preferred light to dark colour scheme.
  10. Under Color, Color Scale, select quantize.
  11. Under Color, Opacity, set to 4.
  12. Under Radius, Radius Based on, Select a field, select
  13. Under Radius, Radius Range, set the range from 1 to 60.
  14. On the contents bar, click + Add Layer on the Filters
  15. Click Select a field, then select
  16. In the slider, drag the rightmost square notch to highlight only 2 bars with the left square notch.
  17. Press the play button the start the animation.

Notes:

  • The speed of the animation can be adjusted.
  • The legend can be shown by clicking the bottom circular button located to the top right corner of the screen.
  • Hover your mouse over a point to see the metadata of the selected power centre.
Figure 1: Power Centre Growth in the Toronto GTA (1992 – 2017)

Limitations:

During the implementation process, it became apparent that Kepler.gl is more focused on graphics and visuals than it does on cartographic standards. The program does not allow the user to manually adjust class ranges on the legend, nor does it accurately display continuous data. The proportional symbols used to represent power centre growth displays flashing or blinking symbols rather than a gradual growth in the symbols. There was an attempt to correct this problem by duplicating the values in the date and time column, and then adding additional pseudo date and time values between each year. However, when tested, the animation exhibited the same flashing and blinking behaviour, therefore it became apparent that is problem exists in the programming of Kepler.gl and not in the data itself. Furthermore, by duplicating these values, the file would exceed the maximum file size on chrome (250mb), and limit performance on Safari, the two web browsers it runs on.

Conclusion:

Regardless of the limitations, as the current Kepler.gl is still in its early beta version, it still has a lot of potential to incorporate user feedback from industry professionals and run additional testing before the final release.

References:

Webber, S. and Hernandez, T. (2018). Retail Development and Planning Policy.   Centre for the Study of Commercial Activity, Ryerson University. Toronto, CA.Uber Technologies Inc. (2018). Kepler.gl. Retrieved from http://kepler.gl/#/
Mapbox. (2018). About Mapbox. Retrieved from https://www.mapbox.com/about/

Invasive Species in Ontario: An Animated-Interactive Map Using CARTO

By Samantha Perry
Geovis Project Assignment @RyersonGeo, SA8905, Fall 2018

My goal was to create an animated time-series map using CARTO to visualize the spread of invasive species across Ontario. In Ontario there are dozens of invasive species posing a threat to the health of our lakes, rivers, and forests. These intruding species can spread quickly due to the absence of natural predators, often damaging native species and ecosystems, and resulting in negative effects on the economy and human health. Mapping the spread of these invasive species is beneficial for showing the extent of the affected areas which can potentially be used for research and remediation purposes, as well as awareness for the ongoing issue. For this project, five of the most problematic or wide-spread invasive species were included in an animated-interactive map to show their spatial and temporal distribution.

The final animated-interactive map can be found at: https://perrys14.carto.com/builder/7785166c-d0cf-41ac-8441-602f224b1ae8/embed

Data

  1. The first dataset used was collected from the Ontario Ministry of Natural Resources and Forestry and contained information on invasive species observed in the province from 1982 to 2012. The data was provided as a shapefile, with polygons representing the affected areas.
  2. The second dataset was downloaded from the Early Detection & Distribution Mapping System (EDDMapS) Ontario website. The dataset included information about invasive species identified between 2010 and 2018. I obtained this dataset to supplement the Ontario Ministry dataset in order to provide a more up-to-date distribution of the species.

Software
CARTO is a location-intelligence based website that offers easy to use mapping and analysis software, allowing you to create visually appealing maps and discover key insights from location data. Using CARTO, I was able to create an animated-interactive map displaying the invasive species data. CARTO’s Time-Series Widget can be used to display large numbers of points over time. This feature requires a map layer containing point geometries with a timestamp (date), which is included in the data collected for the invasive species.

CARTO also offers an interactive feature to their maps, allowing users control some aspects of how they want to view the data. The Time-Series Widget includes animation controls such as play, stop, and pause to view a selected range of time. In addition, a Layer Selector can be added to the map so the user is able to select which layer(s) they wish to view.

Limitations
In order to create the map, I created a free student account with CARTO. Limitations associated with a free student account include a limit on the amount of data that can be stored, as well as a maximum of 8 layers per map. This limits the amount of invasive species that can be mapped.

Additionally, only one Time-Series Widget can be included per map, meaning that I could not include a time-series animation for each species individually, as I originally intended to. Instead, I had to create one time-series animation layer that included all five of the species. Because this layer included thousands of points, the map looks dark and cluttered when zoomed out to the full extent of the province (Figure 1). However, when zoomed in to specific areas of the province, the points do not overlap as much and the overall animation looks cleaner.

Another limitation to consider is that not all the species’ ranges start at the same time. As can be seen in Figure 1 below, the time slider on the map shows that there is a large increase in species observations around 2004. While it is possible that this could simply be due to an increase in observations around that time, it is likely because some of the species’ ranges begin at that time.

Figure 1. Layer showing all five invasive species’ ranges.

Tutorial

Step 1: Downloading and reviewing the data
The Ontario Ministry of Natural Resources and Forestry data was downloaded as a polygon shapefile using Scholars GeoPortal, while the EDDMapS Ontario dataset was downloaded as a CSV file from their website.

Step 2: Selection of species to map
Since the datasets included dozens of different invasive species in the datasets, it was necessary to select a smaller number of species to map. Determining which species to include involved some brief research on the topic, identifying which species are most prevalent and problematic in the province. The five species selected were the Eurasian Water-Milfoil, Purple Loosestrife, Round Goby, Spiny Water Flea, and Zebra Mussel.

Step 3: Preparing the data for upload to CARTO
Since the time-series animation in CARTO is only available for point data, I had to convert the Ontario Ministry polygon data to points. To do this I used ArcMap’s “Feature to Point” tool which created a new point layer from the polygon centroids. I then used the “Add XY Coordinates” tool to get the latitude and longitude of each point. Finally, I used the “Table to Excel” conversion tool to export the layer’s attribute table as an excel file. This provided me with a table with all invasive species point data collected by the Ontario Ministry that could be uploaded to CARTO.

Next, I created a table that included the information for the five selected species from both sources. I selected only the necessary columns to include in the new table, including; Species Name, Observation Date, Year, Latitude, Longitude, and Observation Source. This combined table was then saved as an excel file to be uploaded to CARTO.

Finally, I created 5 additional tables for each of the species separately. These were later used to create map layers that show each species’ individual distribution.

Step 4: Uploading the datasets to CARTO
After creating a free student account with CARTO, I uploaded the six datasets as excel files. Once uploaded, I had to change the “Observation Date” column from a “string” to “date” data type for each dataset. A “date” data type is required for the time-series animation to run.

Step 5: Geocoding datasets
Each dataset added to the map as a layer had to be geocoded. Using the latitude and longitude columns previously added to the Excel file, I geocoded each of the five species’ layers.

Step 6: Create time-series widget to display temporal distribution of all species
After creating a blank map, I added the Excel file that included all the invasive species data as a layer. I then added a Time-Series Widget to allow for the temporal animation. I then selected Observation Date as the column to be displayed, meaning that the point data will be organized by observation date. I chose to organize the buckets, or groupings, for the corresponding time-slider by year.

Since “cumulative” was not an option for the Time-Series layer, I had to use CARTCSS to edit the code for the aggregation style. Changing the style from “linear” to “cumulative” allowed the points to remain on the screen for the duration of the animation, letting the user see the entire species’ range in the province. The updated CSS code can be seen in the screenshots below.

Step 7: Creating five additional layers for each species’ range
Since I could only add one Time-Series Widget per map, and the layer with the animation looks cluttered at some extents, I decided to create five additional layers that show each of the species’ individual observation data and range.

Step 8: Customizing layer styles
After adding all of the layers, a colour scheme was selected where each of the species’ was represented by a different colour to clearly differentiate between them. Colours that are generally associated with the species were selected. For example, the colour purple was selected to represent Purple Loosestrife, which is a purple flowering plant. The “multiply” style option was selected, meaning that areas with more or overlapping occurrences of invasive species are a darker shade of the selected colour.

A layer selector was included in the legend so that users can turn layers on or off. This allows them to clearly see one species’ distribution at a time.

Step 9: Publish map
Once all of the layers were configured correctly, the map was published so it could be seen by the public.

Visualizing Alaska’s Forest Damage in Twenty Years

Author: Anitha Muraleedharan
Geovis Project Assignment@RyersonGeo,
SA 8905, Fall 2018 (Rinner)

Forest Damage in Alaska

Alaska is a dynamic region and has a long history of changeable climate. Alaska has lost a lot of its forests due to insect infestation, fire, flood, landslides, and windthrow. Aerial surveys are conducted to monitor forest health for the State of Alaska and to identify insect and some disease pest trends. This time series map animation will visualize the forest damage in Kenai Peninsula, Tanana Region and Fort Yukon Region of Alaska during the years 1989 to 2010. This blog will cover the entire processes involved in creating this visualization.

Data

The spatial data of the forest damage survey conducted during the period from 1989 to 2010 by the Alaska Department of Natural Resources are readily available for download from AK State Geo-Spatial Data Clearinghouse (http://www.asgdc.state.ak.us/?#2952). The shapefiles are available individually for each year from 1989 to 2010 except for years from 2000 to 2007. These data were used for preparing this Time Series map animation.

Preparing Data for Animation in QGIS

QGIS 3.2.3 64bit was used to prepare the data for animated map visualization of Alaska’s forest damage. QGIS is a free and open-source cross-platform desktop geographic information system (GIS) application that supports viewing, editing, visualization and analysis of geospatial data. Since the data were available only as individual files, the first step in preparing the data was to merge this data together into one shapefile. For this task, I used the Merge Vector Layers Tool in QGIS which merged all the individual shapefiles into a single shapefile.

Steps to Merge multiple vector layers into one

  • Step1: Add all the vector layers, intended to be merged, into QGIS
  • Step2: Go to Vector →Data Management Tools → Merge Vector Layers in the menu
  • Step3: Click input layers button and select all the layers needed to be merged
  • Step4: Click Merged Layer button to give a name for the merged output layer
  • Step4: Click Run in Background button to create the merged layer and add it to QGIS

Fig. 1 Merge Vector Layer Tool in QGIS

The next task was to format the timestamp column to fit the QGIS Time Manager plugin tool that will be used to create the animated map visualization. The timestamp column for this data was “SURVEY_YR” which was in four-digit format. The QGIS Time Manager Plugin requires that the timestamp data be in YYYY-MM-DD format. For this, a new field was created with name “Damage_Yr” and type string and used the Field Calculator tool in Processing Toolbox of QGIS.

Fig. 2 Field Calculator Tool in QGIS

In the Field Calculator tool, the expression “tostring(SURVEY_YR) + ‘-01-01’ ” was used to concatenate data in the field “SURVEY_YR” and the “-01-01”  together to make the timestamp in YYYY-MM-DD format and copy the data to the new field “Damage_Yr”.

Fig. 3 Attribute table showing the Damge_Yr in YYYY-MM-DD format after update.

Visualizing the Time Series

The Time Manager plugin was downloaded and installed in QGIS. The forest damage data was then added as a layer in QGIS. The Google Terrain map was added as the base map for this time series animation. The following steps were performed to add the Google Terrain map to QGIS.

  • Step1: Add a new connection to XYZ Tiles in QGIS and give it a name, say “Google Terrain”
  • Step2: Use https://mt1.google.com/vt/lyrs=t&x={x}&y={y}&z={z} as the URL.
  • Step3. Click Ok and then double-click the created layer to add the “Google Terrain” as the layer.

After the data was added, it was time to apply symbology to the polygon data showing the forest damage in QGIS. The layer was styled using the attribute “Damage_Yr” and categorized with sequential symbology. Once the data was styled, the Time Manager plugin needed to be configured to visualize the time series animation.

In the Time Manager Settings window, the Forest damage layer which needs to be animated was added using the “Add layer” button. The Damage_Yr column was chosen for the Start and End time and “Accumulate features” option was selected to show the features accumulated on the map as the year changes during the animation. 500 milliseconds duration was set in the animation options to show each year for that many seconds in the animation before showing the next year. To display each year as a label in the map during the animation, time format was set as “%Y” and the font, font size, and text color were also set.

Fig. 4 Time manager settings window

Fig. 5 Time display options.

The time frame in the Time manager dock was set as years since the forest damage in each year will be animated and displayed. The time frame size for the animation was set as 1 since we have data for each year from 1989 to 2010. The animation can be played by clicking the play button and QGIS will show the forest damage of Alaska in each year from 1989 to 2010 on the map window for 500 milliseconds each.

Fig. 6 Time Manager dock showing settings for the animation in QGIS

Converting the Time Series into Video

The Time Manager allows exporting the animation to a video. However, there is no option to add a legend onto the rendered maps in the animation in QGIS. Therefore, the maps were exported as .PNG image files. The map frames were exported first with the full extent of the map and subsequently, two more times with map zoomed to areas Tanana and Fort Yukon respectively for showing different areas in one animation. The legend along with title and data source labels were then added for each exported map frame using photoshop.

Finally, VirtualDub software was used to convert the .PNG files to video for each series of maps. VirtualDub is a free and open-source video capture and video processing utility for Microsoft Windows written by Avery Lee.  The generated .PNG files were then renamed in ascending order sequence in the format “frameXXX.png” where XXX is the frame number. For example, frame000, frame001 and so on. This is required for VirtualDub to detect the files as a sequence of images and then combine it to a video. The steps followed to create the animated video is as given below.

  • Step1: Open VirtualDub software
  • Step2: Go to File → Open video file option in the menu and navigate to the images folder
  • Step3: Click the first image in the map image series and VirtualDub will automatically add all the other images that are in sequence
  • Step5: Go to Video → Frame rate and set fps as 0.5 to show each frame for 500 milliseconds in the video
  • Step6: Preview the video and save it using File → Save as AVI option in the menu

Fig. 7 Combining the png files in VirtualDub software

Results


Watch the visualization on YouTube