The Toronto Financial Institution Market: Bridging the gap between Cartography and Analytics using Tableau

Nav Salooja

“Geovis Project Assignment @RyersonGeo, SA8905, Fall 2019”

<script type='text/javascript' src='https://prod-useast-a.online.tableau.com/javascripts/api/viz_v1.js'></script><div class='tableauPlaceholder' style='width: 1920px; height: 915px;'><object class='tableauViz' width='1920' height='915' style='display:none;'><param name='host_url' value='https%3A%2F%2Fprod-useast-a.online.tableau.com%2F' /> <param name='embed_code_version' value='3' /> <param name='site_root' value='/t/torontofimarketgeovisprojectsa8905fall2019' /><param name='name' value='TheTorontoFIMarketDashboard/TorontoFIMarket' /><param name='tabs' value='yes' /><param name='toolbar' value='yes' /><param name='showAppBanner' value='false' /></object></div>

Introduction & Background

Banking in the 21st century has evolved significantly especially in the hyper competitive Canadian Market. Big banks nationally have a limited population and wealth share to capture given Canada’s small population and have been active in innovating their retail footprint. In this case study, TD Bank is the point of interest given its large branch network footprint in the Toronto CMA. Within the City of Toronto the bank has 144 branches and is used as the study area for the dashboard created.  The dashboard analyzes the market potential, branch network distribution, banking product recommendations and client insights to help derive analytics through a centralized and interactive data visualization tool.

Technology

The technology selected for the geovisualization component is Tableau given its friendly user interface, mapping capabilities, data manipulation and an overall excellent visualization experience. However, Alteryx was widely used for the build out of the datasets that run in Tableau. As the data was extracted from various different sources, spatial element and combining datasets was all done in Alteryx. The data extracted for Expenditure, Income and Dwelling Composition was merged and indexed in Alteryx. The TD Branches was web scrapped live from the Branch Locator and the trading areas (1.5KM Buffers) are also created in Alteryx. The software is also used for all the statistical functions such as the indexed data points in the workbook were all created in Alteryx. The geovisualization component is all created within the Tableau workbooks as multiple sheets are leverged to create an interactive dashboard for full end user development and results.

Figure 1 represents the Alteryx Workflow used to build the Market, Branch and Trade Area datasets
Figure 2 provides the build out of the final data sets to fully manipulate the data to be Tableau prepared

Data Overview

There are several data sets used to build the multiple sheets in the tableau workbook which range from Environics Expenditure Data, Census Data and webscrapped TD branch locations. In addition to these data sets, a client and trade area geography file was also created. The clients dataset was generated by leveraging a random name and Toronto address generator and those clients were then profiled to their corresponding market. The data collected ranges from a wide variety of sources and geographic extents to provide a fully functional view of the banking industry. This begins by extracting and analyzing the TD Branches and their respective trade areas. The trading areas are created based on a limited buffer representing the immediate market opportunity for the respective branches. Average Income and Dwelling composition variables are then used at the Dissemination Area (DA) geography from the 2016 Census. Although income is represented as an actual dollar value, all market demographics are analyzed and indexed against Toronto CMA averages. As such these datasets combined with Market, Client and TD level data provide the full conceptual framework for this dashboard.

Tables & Visualization Overview

Given the structure of the datasets, six total tables are utilized to combine and work with the data to provide the appropriate visualization. The first two tables are the branch level datasets which begin with the geographic location of the branches in the City of Toronto. This is a point file taken from the TD store locator with fundamental information about the branch name and location attributes. There is a second table created which analyzes the performance of these branches in respect to their client acquisition over a pre-determined timeframe.

Figure three is a visualization of the first table used and the distribution of the Branch Network within the market

The third table used consists of client level information selected from ‘frequent’ clients (clients transacting at branches 20+ times in a year. Their information builds on the respective geography and identifies who and where the client resides along with critical information that is usable for the bank to run some level of statistical analytics. The client table shows the exact location of those frequent clients, their names, unique identifiers, their preferred branch, current location, average incomes, property/dwelling value and mortgage payments the bank collects. This table is then combined to understand the client demographic and wealth opportunity from these frequent clients at the respective branches.

Figure four is the visualization of the client level data and its respective dashboard component

Table four and five are extremely comprehensive as they visualize the geography of the market (City of Toronto at a DA level). This provides a trade area market level full breakdown of the demographics and trading areas as DAs are attributed to their closest branch and allows users to trigger on for where the bank has market coverage and where the gaps reside. However, outside of the allocation of the branches, the geography has a robust set of demographics such as growth (population, income), Dwelling composition and structure, average expenditure and the product recommendations the bank can target driven through the average expenditure datasets. Although the file has a significant amount of data and can be seen as overwhelming, selected data is fully visualized. This also has the full breakdown of how many frequent clients reside in the respective markets and what kind of products are being recomened on the basis of the market demographics analyzed through dwelling composition, growth metrics and expenditure.

Figure five is the visualization of the market level data and its respective dashboard component

The final table provides visualization and breakdown of the five primary product lines of business the bank offers which are combined with the market level data and cross validated against the average expenditure dataset. This is done to identify what products can be recommended throughout the market based on current and anticipate expenditure and growth metrics. For example, markets with high population, income and dwelling growth with limited spend would be targeted with mortgage products given the anticipated growth and the limited spend indicating a demographic saving to buy their home in a growth market. These assumptions are made across the market based on the actual indexed values and as such every market (DA) is given a product recommendation.

Figure six is the visualization of the product recommendation and analysis data and its respective dashboard component

Dashboard

Based on the full breakdown of the data extracted, the build out and the tables leveraged as seen above, the dashboard is fully interactive and driven by one prime parameters which controls all elements of the dashboard. Additional visualizations such as the products visualization, the client distribution treemap and the branch trends bar graph are combined here. The products visualization provides a full breakdown of the products that can be recommended based on their value and categorization to the bank. The value is driven based on the revenue the product can bring as investment products drive higher returns than liabilities. This is then broken down into three graphs consisting of the amount of times the product is recommended, the market coverage the recommendation provides between Stocks, Mortgages, Broker Fees, Insurance and Personal Banking products. The client distribution tree map provides an overview by branch as to how many frequent clients reside in the branch’s respective trading area. This provides a holistic approach to anticipating branch traffic trends and capacity constraints as branches with a high degree of frequent clients would require larger square footage and staffing models to adequately service the dependent markets. The final component is the representation of the client trends in a five year run rate to identify the growth the bank experienced in the market and at a branch level through new client acquisition. This provides a full run down of the number of new clients acquired and how the performance varies year over year to identify areas of high and low growth.

This combined with the primary three mapping visualizations, creates a fully robust and interactive dashboard for the user. Parameters are heavily used and are built on a select by branch basis to dynamically change all 6 live elements to represent what the user input requires. This is one of the most significant capabilities of Tableau, the flexibility of using a parameter to analyze the entire market, one branch at a time or to analyze markets without a branch is extremely powerful in deriving insights and analytics. The overall dashboard then zooms in/out as required when a specific branch is selected highlighting its location, its respective frequent clients, the trade area breakdown, what kind of products to recommend, the branch client acquisition trends and the actual number of frequent clients in the market. This can also be expanded to analyze multiple branches or larger markets overall if the functionality is required. Overall, the capacity of the dashboard consists of the following elements:

1. Market DA Level Map
2. Branch Level Map
3. Client Level Map
4. Client Distribution (Tree-Map)
5. Branch Trending Graph
6. Product Recommendation Coverage, Value and Effectiveness

This combined with the capacity to manipulate/store a live feed of data and the current parameters used for this level of analysis bring a new capacity to visualizing large datasets and providing a robust interactive playground to derive insights and analytics.

The link for this full Tableau Workbook is hosted here (please note an online account is required):https://prod-useast-a.online.tableau.com/t/torontofimarketgeovisprojectsa8905fall2019/views/TheTorontoFIMarketDashboard/TorontoFIMarket?:showAppBanner=false&:display_count=n&:showVizHome=n&:origin=viz_share_link

Geovisualization of Crime in the City of Toronto Using Time-Series Animation Heat Map in ARCGIS PRO

Hetty Fu

Geovis Project Assignment @RyersonGeo, SA8905, Fall 2019

Background/Introduction

The City of Toronto Police Services have been keeping track of and stores historical crime information by location and time across the City of Toronto since 2014. This data is now downloadable in Excel and spatial shapefiles by the public and can be used to help forecast future crime locations and time. I have decided to use a set of data from the Police Services Data Portal to create a time series map to show crime density throughout the years 2014 to 2018. The data I have decided to work with are auto-theft, break and enter, robbery, theft and assault. The main idea of the video map I want to display is to show multiple heat density maps across month long intervals between 2014 to 2018 in the City of Toronto and focus on downtown Toronto as most crimes happen within the heart of Toronto.

The end result is an animation time-series map that shows density heat map snapshots during the 4-year period, 3-month interval at a time. Examples of my post are shown at the end of this blog post under Heat Map Videos.

Dataset

All datasets were downloaded through the Toronto Police Services Data Portal which is accessible to the public.

The data that was used to create my maps are:

  1. Assault
  2. Auto Theft
  3. Robbery
  4. Break and Enter
  5. Theft

Process Required to Generate Time-Series Animation Heat Maps

Step 1:  Create an additional field to store the date interval in ArcGis Pro.

Add the shapefile downloaded from the Toronto Police Services Portal intoArcGIS Pro.

First create a new field under View Table and then click on Add.             

To get only the date, we use the Calculate Field in the Geoprocessing tools with the formula

date2=!occurrence![:10]  

where Occurrence is the existing text field that contains the 10 digit date: YYYY-MM-DD. This removes the time of day which is unnecessary for our analysis.

Step 2: Create a layer using the new date field created.

Go into properties in the edited layer. Under the time tab, place in the new date field created from Step 1 and enter in the time extent of the dataset. In this case, it will be from 2014-01-01 to 2018-12-31 as the data is between 2014 to 2018.

Step 3: Create Symbology as Heat Map

Go into the Symbology properties for the edited layer and select heat map under the drop down menu. Select 80 as its radius which will show the size of the density concentration in a heat map. Choose a color scheme and set the method as Dynamic. The method used will show how each color in the scheme relates to a density value. In a Dynamic setting versus and constant, the density is recalculated each time the map scale or map extent changes to reflect only those features that are currently in view. The Dynamic method is useful to view the distribution of data in a particular area, but is not valid for comparing different areas across a map (ArcGIS Pro Help Online).

Step 4: Convert Map to 3D global scene.

Go to View tab on the top and select convert to global scene. This will allow the user to create a 3D map feature when showing their animated heat map.

Step 5: Creating the 3D look.

Once a 3D scene is set, press and hold the middle mouse button and drag it down or up to create a 3D effect.

Step 6: Setting the time-series map.

Under the Time tab, set the start time and end time to create the 3 month interval snapshot. Ensure that “Use Time Span” is checked and the Start and End date is set between 2014 and 2018. See the image below for settings.

Step 7: Create a time Slider Steps for Animation Purposes

Under Animation tab, select the appropriate “Append Time” (the transition time between each frame). Usually 1 second is good enough, anything higher will be too slow. Make sure to check off maintain speed and append front before Importing the time Slider Steps. See below image.

Step 8: Editing additional cosmetics onto the animation.

Once the animation is created, you may add any additional layers to the frames such as Titles, Time Bar and Paragraphs.

There is a drop down section in the Animation tab that will allow you to add these cosmetic layers onto the frame.

Animation Timeline by frames will look like this below.

Step 9: Exporting to Video

There are many types of exports the user can choose to create. Such as Youtube, Vimeo, Twitter, Instagram, HD1080 and Gif. See below image for the settings to export the create animation video. You can also choose the number of frames per second, as this is a time-series snapshot no more than 30 frames per second is needed. Choose a place where you would like to export the video and lastly, click on Export.

Conclusion/Recommendation/Limitation

As this was one of my first-time using ArcGIS Pro software, I find it very intuitive to learn as all the functions were easy to find and ready to use. I got lucky in finding a dataset that I didn’t have to format too much as the main fields I required were already there and the only thing required was editing the date format. The number of data in the dataset was sufficient for me to create a time series map that shows enough data across the city of Toronto spanning 3 months at a time. If there was less data, I would have to increase my time span. The 3D scene on ArcGIS Pro is very slow and created a lot of problems for me when trying to load my video onto set time frames. As a result of the high-quality 3D setting, I decided to use, it took couple of hours to render my video through the export tool. As the ArcGIS Pro software wasn’t made to create videos, I felt that there was lack of user video modification tools.

Heat Map Videos Export

  1. Theft in Downtown Toronto between 2014-2018. A Time-Series Heat Map Animation using a 3 month Interval.
  2. Robbery in Downtown Toronto between 2014-2018. A Time-Series Heat Map Animation using a 3 month Interval.
  3. Break and Enter in Downtown Toronto between 2014-2018. A Time-Series Heat Map Animation using a 3 month Interval.
  4. Auto Theft across the City of Toronto between 2014-2018. A Time-Series Heat Map Animation using a 3 month Interval.
  5. Assault across the City of Toronto between 2014-2018. A Time-Series Heat Map Animation using a 3 month Interval.

Visualizing Toronto Fire Service Response

By: Remmy Zelaya

Geovis Project Assignment @RyersonGeo, SA8905, Fall 2019

CARTO is an online tool to create online maps, dashboards, and perform spatial analysis. Basic membership is free and no coding experience is required to get your maps online. I creating my project on visualizing Toronto Fire Service data entirely in CARTO. The embedded map is below or you can click here to see it in a new tab.

I’ll briefly explain how I created my map and how you can too. 

Before we get to CARTO, we’ll need our data. The City of Toronto’s Open Data portal contains lots of free data on city services and life. From the portal I downloaded shapefiles of TFS stations and run areas (catchment areas for fire stations), and a CSV file of fire incidents.

Next create a CARTO account if you don’t already have one. Once logged in, the CARTO home page will have links to “Getting Started”, “New Map”, and “New dataset.” The Getting Started page is an excellent tutorial on CARTO for first time users. 

Before we start making a map, we will need to upload our data. Click “new dataset” and follow the prompts. Note, CARTO requires shapefiles to be archived in a ZIP file. 

Once that is done, click on “new map” and add your uploaded datasets. CARTO will add your datasets as layers to the map, zoom to layer extent, and automatically create a point layer out of the CSV file. 

The map is on the right side of the screen and a control panel with a list of the uploaded layers is on the right. From here we can do a couple of things;

  • Re-title our map by double clicking on the default title
  • Rearrange our layers by dragging and dropping. Layer order determines drawing order. Rearrange the layers so that the stations and incidents points are on top of the run area polygon.
  • Change the base map. I’ve used Positon Lite for a simple and clean look. Note, CARTO has options to import base maps and styles from other site, or to create your own.
  • Click on the layer card to bring up that layer’s options menu.

Let’s click on the fire stations layer. As with the map we can rename the layer by double clicking on the name. The layer menu has five panes, Data, Analysis, Style, Pop-Up, Legend. The Style pane will be selected by default. The first section of the Style pane is aggregation, which is useful for visualizing dense point layers. We’ll keep the default aggregation of By Point. Section 2 Style controls the appearance of the layer. I’ve changed my point colour to black and increased the size to 12. I need the stations to stand out from the incident points. 

Now with the incidents layer, I decided to use the Animation aggregation option. If the point layer has a column representing time, we can use this to create an animation of the points appearing on the map over time. This option creates a time slider widget at the bottom of the map with a histogram representing the amount of fires over time.

With the run areas, I decided to create a choropleth map where run areas with higher amount of incidents would appear darker on the map. To do this, I first need to determine how many incidents points fall into each run area. Go to the run area menu, click on Analysis, then “+Add New Analysis.” CARTO will navigate to a new page with a grid of its spatial analysis options. Click on “Intersect and Aggregate” which finds “overlapping geometries from a second layer and aggregate its values in the current layer.”

CARTO will navigate back to the Analysis pane of the run area menu and display options for the analysis. Run area should already be selected under Base Layer. Choose incidents as the target layer, and under Measure By select count. CARTO will display a message stating new columns have been added to the data, count_vals and count_vals_density. 

There will be an option to style the analysis. Click on it. Choose “by value” for Polygon Colour, and choose the new count_vals_density for Column, then select an appropriate colour scheme.

CARTO’s widget feature creates small boxes on the right of the map with useful charts and stats on our data. You click on the Widgets pane to start add new widgets from a grid (as with Analysis) or can add new widgets based on a specific layer from that layer’s Data pane. CARTO has four types of widgets;

  • Category creates a horizontal bar chart measuring how many features fit into a category. This widget also allows users to filter data on the map by category. 
  • Histogram creates a histogram measuring a selected variable
  • Formula displays a statistic on the data based on a selected formula
  • Time Series animates a layers according to its time information.

As with layers, clicking on a widget brings up its option menu. From here you can change the source data layer, the widget type, and configure data values. For my Fires by Run Area widget, I used the incidents layer as the source, aggregated by id_station (fire station ID numbers) using the count operation. This widget counts how many incidents each station responded to and displays a bar chart of the top 5 stations. Clicking on a station in the bar chart will filter the incidents by the associated station. After this, I added four formula based widgets.

We’re nearly done. Click on the “publish” button on the bottom left to publish the map to the web. CARTO will provide a link for other users to see the map and an HTML embed code to add it to a web page. I used the embed code to added the embedded map to the beginning of the post.

Thanks for reading. I hope you’ll use CARTO to create some nice maps of your own. You may be interested in checking out the CARTO blog to see other projects built on the platform or the Help section for my information on building your own maps and applications.

Toronto Theft: A Neighbourhood Investigation

Geovis Project Assignment @RyersonGeo, SA8905, Fall 2019

By: Julia DiMartella-Orsi

Introduction:

ESRI’s creation of the Story Map changed the way we could visualize data. Not only did it allow for a broader audience to interact and create their own maps due to its easy to use design, it also contained many new amazing functions, templates, and themes. Users can personalize their story by adding in their own images, text, videos, and map layers by creating their own free ArcGIS Online account. Popular templates include Map Series, Tour, Journal, and Cascade.

Get started making your own Story Map here: http://storymaps-classicqa.arcgis.com/en/app-list/

Creating Your Story Map:

Once you have selected the template you want to use the choice is up to you. By clicking the “+” symbol you can choose to include text, media sources such as a videos, a new title page, or immersive content such as a web map.

ESRI also designed Story Maps to link to outside content and various social media sites such as Flickr and Unsplash. ‘Link to Content’ is also extremely useful as it allows users to add photos and videos found on the internet directly to their story map by copying and pasting their link.

To add interactive web maps into your story map users can link map layers from their ArcGIS Online account. Layers can be created in ArcGIS Online, but also in ArcMap where layers are exported as a zip file and imported onto your ArcGIS Online base map. Map layers can also be found online using the ‘add layer from the web’ or ‘search for layers’ options.  The layers that appear are based on the type of ArcGIS Online account you have created. Enterprise accounts contain additional layers provided by your organization, however ESRI also has free downloadable layers available for users without an organization.

Users also have the option to make their story maps public by clicking the globe icon, or private for their own personal use by clicking the lock icon. To save your story map select the floppy disk icon. Your saved map will appear under ‘My Content’ in your ArcGIS Online account.

My Story and Creating Web Maps:

Over the last few years, theft in Toronto has been increasing at a rapid rate. According to the Toronto Police Service, Toronto experienced a total of 5430 thefts between 2014-2018. However, these are only those that have been reported and documented by police. In order to analyze the distribution of theft across the city, the Toronto Police created a point dataset that summarized when and where each theft took place. Additional datasets were also created for prominent types of theft such as bicycle and auto theft.

To compare the number and types of theft in each Toronto neighbourhood I decided to create a story map using the Cascade template. This created a scrolling narrative that would allow viewers to observe the data in a clear, unique way. The reason why I chose to use a story map was due to the number of layers I wanted to compare, as well as use the ‘swipe tool’ to easily compare each neighbourhood. Therefore, I created a series of choropleth maps based on the 2014-2018 theft/crime data from the Toronto Police Open Data Portal.

The following steps were used to create each web map used in my Story Map:

Step 1: Download the point data and add the layer into ArcMap.

Step 2: Use the ‘spatial join’ analysis tool and select your neighbourhood boundary file as the target layer and the theft point data as the join feature. Make sure to select ‘join one to one’. This will produce a new layer with a ‘count’ field that counts the number of thefts in each neighbourhood – each neighbourhood is given a count.

Step 3: In order to produce accurate results, you must normalize your data. To do so add a new field into your attribute table (same layer with the count field) titled ‘Area’, and right click to select ‘calculate geometry’. Change the property to ‘area’ and choose the units you wish to use. Click ‘ok’ and the results will populate your new field.

Step 5: Export the layer and save it as a compressed zip folder. Import the data into ArcGIS Online by clicking the “Add” tab.

Step 6: Once you import your layer you are given a variety of styles to choose from. Select the one you like best (ex: choropleth) as well as the field you wish to map – in this case select ‘count’. To normalize ‘count’ select the ‘divided by’ dropdown and choose your ‘Area’ field. Change the colour of your map to your preference by clicking ‘symbols’.

Step 7: Save your layer to and select the tags that relate to your topic. The layer will now appear in ‘My Content’ where it can be added to your Story Map.

Step 8: To compare each layer add both layers you wish to compare to your story map by using the “+” symbol. Once you have done so, choose the transition type (ex: horizontal swipe) you want to use by clicking on the arrow below. The transition will take place as the user scrolls through your story map.

My Story Map titled “Toronto Theft: A Neighbourhood Investigation” can be viewed here:

https://arcg.is/uiemr

Putting BlogTO on the map (literally) – Tutorial

Kyle Larsen
SA8905 – Cartography and Geovisualization
Fall 2019

Instagram is a wealth of information, for better or worse, if you’ve posted to Instagram before and your profile is public, maybe even if it’s not, then your information is out there just waiting for someone, someone maybe like me, to scrape your information and put it onto a map. You have officially been warned.

But I’m not here to preach privacy or procure your preciously posted personal pics. I’m here to scrape pictures from Instagram, take their coordinates, and put them onto a map into a grid layout over Toronto. My target for this example is a quite public entity that thrives off exposure, the notorious BlogTo. Maybe only notorious if you live in Toronto, BlogTo is a Toronto-based blog about the goings on in the 6ix as well as Toronto life and culture, they also have an Instagram that is almost too perfect for this project – but more on that later. Before anything is underway a huge thank-you-very-much to John Naujoks and his Instagram scraping project that created some of the framework for this project (go read his project here, and you can find all of my code here)

When scraping social media sometimes you can use an API to directly access the back end of a website, Twitter has an easily accessible API that is easy to use. Instagram’s API sits securely behind the brick wall that is Facebook, aka it’s hard to get access to. While it would be easier to scrape Twitter, we aren’t here because this is easy, maybe it seems a little rebellious, but Instagram doesn’t want us scraping their data… so we’re going to scrape their data.

This will have to be done entirely through the front end, aka the same way that a normal person would access Instagram, but we’re going to do it with python and some fancy HTML stuff. To start you should have python downloaded (3.8 was used for this but any iteration of python 3 should give you access to the appropriate libraries) as well as some form of GIS software for some of the mapping and geo-processing. Alteryx would be a bonus but is not necessary.

We’re going to use a few python libraries for this:

  • urllib – for accessing and working with URLs and HTML
  • selenium – for scraping the web (make sure you have a browser driver installed, such as chromedriver)
  • pandas – for writing to some files

If you’ve never done scraping before, it is essentially writing code that opens a browser, does some stuff, takes some notes, and returns whatever notes you’ve asked it to take. But unlike a person, you can’t tell python to go recognize specific text or features, which is where the python libraries and HTML stuff comes in. The below code (thanks John) takes a specific Instagram user and return as many post URLs as you want and adds them to a list, for your scraping pleasure. If you enable the browser head you can actually watch as python scrolls through the Instagram page, silently kicking ass and taking URLs. It’s important to use the time.sleep(x) function because otherwise Instagram might know what’s up and they can block your IP.

But what do I do with a list of URLs? Well this is where you get into the scrappy parts of this project, the closest to criminal you can get without actually downloading a car. The essentials for this project are the image and the location, but this where we need to get really crafty. Instagram is actually trying to hide the location information from you, at least if you’re scraping it. Nowhere in a post are coordinates saved. Look at the below image, you may know where the Distillery District is, but python can’t just give you X and Y because it’s “south of Front and at that street where I once lost my wallet.”

If you click on the location name you might get a little more information but… alas, Instagram keeps the coordinates locked in as a .png, yielding us no information.

BUT! If you can scrape one website, why not another? If you can use Google Maps to get directions to “that sushi restaurant that isn’t the sketchy one near Bill’s place” then you might as well use it to get coordinates, and Google actually makes it pretty easy – those suckers.
(https://www.google.com/maps/place/Distillery+District,+Toronto,+ON/@43.6503055,-79.35958,16.75z/data=!4m5!3m4!1s0x89d4cb3dc701c609:0xc3e729dcdb566a16!8m2!3d43.6503055!4d-79.35958 )
I spy with my little eye, some X and Y coordinates, the first set after the ‘@’ would usually be the lat/long of your IP address, which I’ve obviously hidden because privacy is important, that’s the takeaway from this project right? The second lat/long that you can gleam at the end of the URL is the location of the place that you just googled. Now all that’s left is to put all of this information together and create the script below. Earlier I said that it’s difficult to tell python what to look for, and what you need is the xpath, which you can copy from the html (right-click an element and then right-click that html and then you can get the xpath for that specific element. For this project we’re going to need the xpath for both the image and the location. The steps are essentially as follows:

  • go to Instagram post
  • download the image
  • copy the location name
  • google the location
  • scrape the URL for the coordinates

There are some setbacks to this, not all posts are going to have a location, and not all pictures are pictures – some are videos. In order for a picture to qualify for full scraping it has to have a location and not be a video, and the bonus criteria – it must be in Toronto. Way back I said that BlogTO is great for this project, that’s because they love to geotag their posts (even if it is mostly “Toronto, Ontario”) and they love to post about Toronto, go figure. With these scripts you’ve built up a library of commands for scraping whatever Instagram account your heart desires (as long as it isn’t private – but if you want to write script to log in to your own account then I guess you could scrape a private account that has accepted your follow request, you monster, how dare you)

With the pics downloaded and the latitudes longed it is now time to construct the map. Unfortunately this is the most manual process, but there’s always the arcpy library if you want to try and automate this process. I’ll outline my steps for creating the map, but feel free to go about it your own way.

  1. Create a grid of 2km squares over Toronto (I used the grid tool in Alteryx)
  2. Intersect all your pic-points with the grid and take the most recently posted pic as the most dominant for that grid square
  3. Mark each square with the image that is dominant in that square (I named my downloaded images as their URLs)
  4. Clip all dominant images to 1×1 size (I used google photos)
  5. Take a deep breath, maybe a sip of water
  6. Manually drag each dominant image into its square and pray that your processor can handle it, save your work frequently.

This last part was definitely the most in need of a more automated process, but after your hard work you may end up with a result that looks like the map below, enjoy!

Automobile Collisions Involving TTC Vehicles

Eric Lum
SA8905 Geovis Project, Fall 2019

Toronto is the largest metropolis in Canada, attracting people from far and wide. As such, there are many forms of transportation that pass through the city including cars, bicycles, public transit, regional trains and many more. The Toronto Transit Commission (TTC) is one of the main methods that people rely on, as millions ride their services each and every day. All of these forms of transportation must share the roads, and from time to time collisions occur. This project aims to animate collisions between TTC surface vehicles such as a bus or streetcar, with another form of transportation (not including pedestrians). This visualization will be on the web-mapping service Carto, where a time series map will be produced on the various TTC related collisions.

The collision data for this project was obtained from the Toronto Police open data portal. The “TTC Municipal Vehicle” dataset that was used is a subset of the “Killed and Seriously Injured” dataset, as these are the specific types of collisions that were collected. The data is available for the years 2008-2018, but only the past five years from 2014-2018 were used for the sample size of the project. Information on the collisions provided in the dataset include the latitude, longitude, intersection, vehicle collision type, time, date, year and neighbourhood it occurred in.

The first step of getting the time series web map to work is to create a map and import the data into Carto. The collisions data was downloaded from the Toronto Police as a .csv file, which can easily be uploaded to Carto. Other supporting data used for this map includes the City of Toronto boundary file retrieved from the City of Toronto open data portal and the TTC routes which were retrieved from Scholars Geoportal. In order for these shapefiles to be imported into Carto, they must either be uploaded as .ZIP files or converted to another supported format such as JSON file. Once all the data was ready, it was uploaded through the “Connect Dataset” label shown below.

The next step was to geocode the collision locations with the latitude and longitude provided in the collisions .csv file. This was done through Carto’s geocode feature shown below. To do this, the layer with the data was selected and the geocode option was chosen under the “Analysis” tab. The fields for latitude and longitude were then input.

Once geocoded, the “Aggregation” method for the data needed to be chosen. As this is a visualization project over a span of years, the time series option was chosen. The “Style” also needed to be set, referring to how the points would be displayed. The dataset contained information on the different vehicle types that were involved in the collisions, so the “point colour” was made different for each vehicle. These functions are both shown below.

The same “Style” method for visualization was also applied to the TTC Routes layer, as each type of transportation should be shown with a unique colour. The last part in animating the data is to choose the field for which the timer series is to be based on. The date field in the collisions .csv file was used in the “Widgets” function on Carto. This allows for all the data to be shown on a histogram for the entire time span.

To finalize the map, a legend and basemap were selected. Once happy with my map, I made it public by enabling the “Publish” option at the bottom of the interface. This generated a shareable link for anyone to view.

A snapshot of the final time series map is shown below.

Thank you for viewing my blog post!

To access the full web map on Carto, the link is provided here:

https://ericlum24.carto.com/builder/00c16070-d0b8-4efd-97db-42ad584b9e14/embed

Visualizing Station Delays on the TTC

By: Alexander Shatrov

Geovis Project Assignment @RyersonGeo, SA8905, Fall 2018.

Intro:

The topic of this geovisualization project is the TTC. More specifically, the Toronto subway system and its many, many, MANY delays. As someone who frequently has to suffer through them, I decided to turn this misfortune into something productive and informative, as well as something that would give a person not from Toronto an accurate image of what using the TTC on a daily basis is like. A time-series map showing every single delay the TTC went through over a specified time period.  The software chosen for this task was Carto, due to its reputation as being good at creating time-series maps.

Obtaining the data:

First, an excel file of TTC subway delays was obtained from Toronto Open Data, where it is organised by month, with this project specifically using August 2018 data. Unfortunately, this data did not include XY coordinates or specific addresses, which made geocoding it difficult. Next, a shapefile of subway lines and stations was obtained from a website called the “Unofficial TTC Geospatial Data”. Unfortunately, this data was incomplete as it had last been updated in 2012 and therefore did not include the recent 2017 expansion to the Yonge-University-Spadina line. A partial shapefile of it was obtained from DMTI, but it was not complete. To get around this, the csv file of the stations shapefile was opened up, the new stations added, the latitude-longitude coordinates for all of the stations manually entered in, and the csv file then geocoded in ArcGIS using its “Display XY Data” function to make sure the points were correctly geocoded. Once the XY data was confirmed to be working, the delay excel file was saved as a csv file, and had the station data joined with it. Now, it had a list of both the delays and XY coordinates to go with those delays. Unfortunately, not all of the delays were usable, as about a quarter of them had not been logged with a specific station name but rather the overall line on which the delay happened. These delays were discarded as there was no way to know where exactly on the line they happened. Once this was done, a time-stamp column was created using the day and timeinday columns in the csv file.

Finally, the CSV file was uploaded to Carto, where its locations were geocoded using Carto’s geocode tool, seen below.

It should be noted that the csv file was uploaded instead of the already geocoded shapefile because exporting the shapefile would cause an issue with the timestamp, specifically it would delete the hours and minutes from the time stamp, leaving only the month and day. No solution to this was found so the csv file was used instead. The subway lines were then added as well, although the part of the recent extension that was still missing had to be manually drawn. Technically speaking the delays were already arranged in chronological order, but creating a time series map just based on the order made it difficult to determine what day of the month or time of day the delay occurred at. This is where the timestamp column came in. While Carto at first did not recognize the created timestamp, due to it being saved as a string, another column was created and the string timestamp data used to create the actual timestamp.

Creating the map:

Now, the data was fully ready to be turned into a time-series map. Carto has greatly simplified the process of map creation since their early days. Simply clicking on the layer that needs to be mapped provides a collection of tabs such as data and analysis. In order to create the map, the style tab was clicked on, and the animation aggregation method was selected.

The color of the points was chosen based on value, with the value being set to the code column, which indicates what the reason for each delay was. The actual column used was the timestamp column, and options like duration (how long the animation runs for, in this case the maximum time limit of 60 seconds) and trails (how long each event remains on the map, in this case set to just 2 to keep the animation fast-paced). In order to properly separate the animation into specific days, the time-series widget was added in the widget tab, located next to to the layer tab.

In the widget, the timestamp column was selected as the data source, the correct time zone was set, and the day bucket was chosen. Everything else was left as default.

The buckets option is there to select what time unit will be used for your time series. In theory, it is supposed to range from minutes to decades, but at the time of this project being completed, for some reason the smallest time unit available is day. This was part of the reason why the timestamp column is useful, as without it the limitations of the bucket in the time-series widget would have resulted in the map being nothing more then a giant pulse of every delay that happened that day once a day. With the time-stamp column, the animation feature in the style tab was able to create a chronological animation of all of the delays which, when paired with the widget was able to say what day a delay occurred, although the lack of an hour bucket meant that figuring out which part of the day a delay occurred requires a degree of guesswork based on where the indicator is, as seen below

Finally, a legend needed to be created so that a viewer can see what each color is supposed to mean. Since the different colors of the points are based on the incident code, this was put into a custom legend, which was created in the legend tab found in the same toolbar as style. Unfortunately this proved impossible as the TTC has close to 200 different codes for various situations, so the legend only included the top 10 most common types and an “other” category encompassing all others.

And that is all it took to create an interesting and informative time-series map. As you can see, there was no coding involved. A few years ago, doing this map would have likely required a degree of coding, but Carto has been making an effort to make its software easy to learn and easy to use. The result of the actions described here can be seen below.

https://alexandershatrov.carto.com/builder/8574ffc2-9751-49ad-bd98-e2ab5c8396bb/embed

Mapping Toronto Green Space in Android

By Jacob Lovie | GeoVis Project Assignment                @RyersonGeo | SA8905 | Fall 2018

Introduction

With today’s technology becoming more and more mobile, and the ability to access everything you need on your mobile device, it is more important than even to ensure that GIS is evolving to meet these trends. My GeoVisualization project focused on designing an android application to allow users to explore Toronto’s green space and green initiatives, making layers such as parks and bike stations accessible in the palm of your hand. However, it is not just having access to this that is important. What’s important when working with these technologies is that a user can explore the map and retrieve the information seamlessly and efficiently.

Data and Hosting Feature Services

All the data for the project was retrieved from the city of Toronto’s open data portal. From there, all the data was uploaded to ArcGIS Online and set up as hosted feature services. A base map was also designed using ArcGIS for Developers and hosted. The application was able to target these hosted feature layer and use them in the map, making the size of the app small. The symbology and setup of the hosted feature layers was also done in ArcGIS online, so the app didn’t have to make any changes or set symbology when it wasn’t necessary.

Methods

The developer environment that I worked in to design my app was Android Studio, the baseline for designing any android apps. The programming language used in Android Studio is Java. Within Android Studio, the functionality of ArcGIS Runtime Software Developer Kit (SDK) for Android can be brought in, bringing in all the libraries and functions associated with ArcGIS Runtime SDK for Android. With this I was able to use ArcGIS functionality in android, designing maps, accessing hosted feature services, and perform geoprocessing.

Understanding how ArcGIS SDK for Android worked with in Android Studio was an important key in designing my app. When creating a map, I first had to create a Map object. An object is a variable that is of a certain datatype. If you were talking about having a text object as a variable that could be called, it would be of datatype string, and the word itself would be an object that is callable and referential. The Map object is what is displayed in an activity window (more on this later), which is what the user visualizes when using the app. The map can be set to view a certain area, which was Toronto in my app. A user can pan around the map like they would on any interactive map without having additional coding in Android Studio (it is natural to the Map datatype). The Map also has associated Layer objects that have their own set of parameters.

While designing my app, any time I would want something done in my design, such as creating a map object or adding a layer to a map object, I created a function that wold performs an action. This reduces repetition in the code when I attempted to do something complex multiple times. I designed 3 functions. The first was to create a Map, the second was to add a Layer that could be activated and deactivated in the Map through a switch that would be displayed in the main Activity Window. The final function added a layer that could be queried and would extract information from that layer.

When designing an android app, there are many fine details that are not necessarily considered when using an app on your phone. Simple things like having a window or text appear, opening a second window, or displaying information were things I very much appreciated after designing the app. Within my app, I wanted to display a second activity window to display information on neighbourhoods in Toronto when the user touched them. Within Android Studio this required creating a second activity window, and transferring the information obtained in the map to the second activity. This was done through my displayInformation function. I was then able to create a second activity and display this information using a custom list display to show the attribute data of a selected neighbourhood.

     >>>>>>>>>>>     

Setting up the display in Android Studio is relatively simple. There is an interface that allows you to anchor different objects to parts of the screen. This allows the app to run smoothly across all devices, not based on the size and ratio of the device. The switches in my Main Activity window were anchored to the top left, and to each other. My Map is in the background, but appears as white in this activity window.

The Application

Once all the coding and testing was completed, running the app was simple. I was able to bundle my code and send it to my personal phone, a Galaxy S9. The functions called the hosted service layers and displayed them in the map (Wifi or internet connection was required). I was also able to click on neighbourhoods and it would open my second activity that displayed the attribute information of that neighbourhood. If you want a more in-depth look at my code, it is available at https://github.com/jclovie/GeoVis-Ryerson/.

Creating a Toronto City Ward Model Using Laser Cut Acrylic

by Selasi Dorkenoo

SA8905 Cartography and Geovisualization Fall 2018

To better understand characteristics of the new municipal electoral wards in the City of Toronto, the new 25-ward boundary shapefile provided by the City of Toronto was converted to vector format and laser cut into five translucent sheets of acrylic. Each piece is engraved with the ward ID. Laser cutting allows the puzzle to not only fit together with precision, but also visualized the demographic census data using redundant symbology: opacity (lightness) and height.

Ward boundaries were retrieved from Toronto Open Data Catalogue and imported into ArcGIS Desktop. The model was designed to be cut into 16 in x 8.5 in sheets of 3mm acrylic, including legend items and a scale bar. Features in black (below) represent pieces that were laser cut and features in red represent laser engraving on a piece. Using layout view, the design was exported as a vector (.ai) file and sent to Hot Pop Factory for their laser cutting services.

Once the acrylic was cut, a magnet was super-glued to each piece below the engraved ward IDs. The magnets used were about 6mm in diameter and 2mm in thickness. Magnets were also attached to the scale bar and legend items. Using a magnetic white board as a base for the model, the pieces were stacked and the model itself was complete.


Demographic data at the ward level was retrieved from Toronto Open Data Catalogue as well. Once joined to the ward boundary file, a set of choropleth maps including population density, visible minorities, unemployment rate and average personal income were created. A maximum of five bins can be used to classify the data in each map since only five sheets of acrylic were laser cut for the model.

A catalogue of these maps was printed and packaged with the ward model. Users can browse through the catalogue and select which variable they wish to map. Using dry erase markers they can write the necessary cartographic elements on the mapped area (i.e. legend labels and title).

Movies and Television shows filmed in Toronto but based elsewhere…

by Alexander Pardy
Geovis Class Project @RyersonGeo, SA8905, Fall 2017

Data and Data Cleaning:

To obtain my data I used https://moviemaps.org/ and selected Toronto  the website displays a map that shows locations in the Greater Toronto Area  where movies and television shows were filmed. The point locations are overlaid on top of Google Maps imagery.

If you use the inspect element tool in internet explorer, you can find a single line of JavaScript code within the map section of the webpage that contains the latitude and longitude of every single point.

The data is in a similar format to python code. The entire line of JavaScript code was inputted into a Python script.  The python script writes the data into a CSV file that can then easily be opened in Microsoft Excel. Once the file was opened in Excel, Google was used to search for the setting of each and every single movie or television show, using the results of various different websites such as fan websites, IMDB, or Wikipedia. Some locations take place in fictional towns and cities, in this case locations were approximated using best judgement to find a similar location to the setting. All the information was than saved into a CSV file. Python was then used to delete out any duplicates in the CSV file and was used to give a count of each unique location value. This gives the total number of movies and television shows filmed at each different geographical location. The file was than saved out of python back into a CSV file. The latitude and longitude coordinates for each location was than obtained from Google and inputted into the CSV file.  An example is shown below.

Geospatial Work:

The CSV file was inputted into QGIS as a delimited text layer with the coordinate system WGS 84. The points were than symbolized using a graduated class method based on a classified count of the number of movies or television shows filmed in Toronto. A world country administrative shape file was obtained from the Database of Global Administrative Areas (GADM). There was a slight issue with this shapefile,  the shapefile had too much data and every little island on the planet was represented in this shapefile. Since we are working at a global scale the shapefile contained too much detail for the scope of this project.

Using WGS 84 the coordinate system positions the middle of the map at the prime meridian and the equator. Since a majority of the films and television shows are based in North America,  a custom world projection was created. This was accomplished in QGIS by going into Settings, Custom CRS, and selecting World Robinson projection. The parameters of this projection was then changed to change the longitude instead of being the prime meridian at 0 degrees, it was changed to -75 degrees to better center North America in the middle of the map. An issue came up after completing this is that a shapefile cannot be wrapped around a projection in QGIS.


After researching how to fix this, it was found that it can be accomplished by deleting out the area where the wrap around occurs. This can be accomplished by deleting the endpoints of where the occurrence happens. This is done by creating a text file that says:

This text box defines the corners of a polygon we wish to create in QGIS.  A layer  can now be created from the delimited text file, using custom delimiters set to semi colon and well-known text. It creates a polygon on our map, which is a very small polygon that looks like a line. Then by going into Vector, Geoprocessing Tools, Difference and selecting the input layer as the countries layer and the difference layer as the polygon that was created. Once done it gives a new country layer with a very thin part of the map deleted out (this is where the wrap around occurred). Now the map wraps around fine and is not stretched out. There is still a slight problem in Antarctica so it was selected and taken out of the map.

Styling:

The shapefile background was made grey with white hairlines to separate the countries. The count and size of the locations was kept the same. The locations were made 60% transparent. Since there was not a lot of  different cities the  symbols were classified to be in 62 classes, therefore each time the number increased, the size of the point would increase.  The map is now complete. A second map was added in the print composer section to show a zoomed in section of North America. Labels and lines were then added into the map using Illustrator.

Story Map:

I felt that after the map was made a visualization should also be created to help covey the map that was created by being able to tell a story of the different settings of films and television shows that were filmed in Toronto.  I created a ESRI story map that can be found Here .

The Story Map shows 45 points on a world map, these are all based on the setting of television shows and movies that were filmed in the City of Toronto. The points on the map are colour coded. Red point locations had 4-63 movie and television shows set around the points. Blue point locations had 2-3 movie and television shows set around the points. Green point locations had 1 movie or television show set around the point. When you click on a point it brings you to a closer view of the city the point is located in. It also brings up a description that tells you the name of the place you are viewing and the number of movies and television shows whose settings takes place in that location. You also have the option to play a selected movie or television show trailer from YouTube in the story map to give you an idea of what was filmed in Toronto but is conveyed by the media industry to be somewhere else.