4. More Data with Leaflet Maps


A quick note about the practicals...

Remember to use Mozilla Firefox to do these practicals. Please do NOT use Microsoft Internet Explorer / Edge, this is not suitable for web development. Coding should be done in Notepad++

You do not have to run every bit of code in this document. Read through it, and have a go where you feel it would help your understanding. If I explicitly want you to do something, I will write an instruction that looks like this:

This is an instruction that tells you something to either think about, or do.

Shortcuts: Part 1 Part 2


In which we load Open Data onto our map live from the web, and visualise it using Turf.js!

Part 1

Before we get stuck into making our map this week, we are going to have a look at a couple more parts of JavaScript: Arrays and Loops.

Arrays in JavaScript

Where a variable holds a single value, arrays are effectively a list of variables, containing as many values as you wish. A common use of an array might be, for example, to store a set of coordinate pairs for some points that you wish to draw onto your map.

Making an Array

There are two main ways to create an array, both of which provide the same result:

var arr = Array(element0, element1, ..., elementN);
var arr = [element0, element1, ..., elementN];

In this case, element0, element1, ..., elementN is a comma-separated list of values for the values that you want to store in the array (it’s elements, e.g. [0, 12, 36, 55]). When these values are specified, the array is initialised with them as the array’s elements. If you want to make an empty array (with no elements - one that has not been initialised), then you simply do the same but with no arguments / elements:

var arr = Array();
var arr = [];

Similarly, you may want to make an array with a particular number of ‘empty’ elements in it that you will populate later on. This is also quite simple:

var arr = Array(10);

If you want to find out how many values are stored in your array, you can see this by looking at the length property of the array object like this:

 arr.length

So, for example:

var fruit = ["apple", "banana", "orange"];
console.log(fruit.length);

would write 3 to the console, because there are three values in the array.

Using an Array

Once you have a number of elements in an array (either empty or with values), you will want to be able to refer to them in order to either set or retrieve a value. This is easy, because each element in an array has an index value that you can use to refer to it. The first element has index 0, the second 1, next 2 and so on…

so:

var fruit = ["apple", "banana", "orange"];
console.log(fruit[0]);		// prints 'apple'
console.log(fruit[1]);		// prints 'banana'
console.log(fruit[2]);		// prints 'orange'
console.log(fruit.length);	// prints '3'

Note how the length of the array is 3, but the index only goes up to 2. This is because the index count starts at 0, which is the cause of many many coding errors! Always remember, the highest index in an array is always length-1.

Multi-dimensional Arrays

It is quite helpful to think about arrays as a list of variables, and the array itself as a special tye of variable (one that can hold multiple values). It follows, then, that each item in an array can also be another array, giving you an array of arrays, or a list of lists. We call this a multi-dimensional array. Here is an example:

//normal arrays
var fruit = ["apple", "banana", "orange"];
var snacks = ["mars", "twix", "crisps"];
var lunch = ["pizza", "curry", "lasagne"]; 

//multi-dimensional arrays
var food = [fruit, snacks, lunch];

As always, note the quote marks for the string values, but not for variables.

You can then access elements in the multidimensional array by giving two sets of index numbers:

console.log(food[0]);		//prints '["apple", "banana", "orange"]'
console.log(food[1][2]);	//prints 'crisps'
console.log(food.length);	//prints 3, because food contains 3 arrays
console.log(food[0].length);	//prints 3, because food[0] contains 3 items

As you might expect, you can build multi-dimensional arrays directly using either the Array() or the [] syntax (I prefer the latter). Here is an example:

var food = [["apple", "banana", "orange"], ["mars", "twix", "crisps"], ["pizza", "curry", "lasagne"]];

Do you see how that works? there is an ‘outer’ set of [], and then three ‘inner’ sets of [], one for each of the inner arrays. You can see this more clearly below, which is exactly the same code as above but separated out and commented:

var food = 					//variable name
	[					//open outer array
		["apple", "banana", "orange"], 	//inner array #1
		["mars", "twix", "crisps"], 	//inner array #2
		["pizza", "curry", "lasagne"]	//inner array #3 (last one so no comma!)
	];					//close outer array

Working with Arrays

We have covered the basics of arrays already, such as creating, populating and referencing them. There are many other things that you can do with arrays using the functions that are built into them, here are some examples:

NOTE: have a look through these, but don’t try to memorise them all, you can always look back here to see how they work as and when you need them!

.concat() joins two arrays and returns a new array:

//make two arrays
var a = [1, 2, 3];
var b = [4, 5, 6];

//concatenate them
console.log(a.concat(b));	// Array [ 1, 2, 3, 4, 5, 6 ]

.join() joins all elements of an array into a string. If you pass a string as an argument, then it will place that string as a delimiter between each element:

//make an array
var myArray = ["Wind", "Rain", "Fire"];

//join it without a delimiter
console.log(myArray.join()); 		// "WindRainFire"

//join it with a delimiter
console.log(myArray.join(", ")); 	// "Wind, Rain, Fire"

.push() adds one or more elements to the end of an array:

//make an array
var myArray = ["1", "2"];

//add another element to it
myArray.push("3"); 	

//print the array
console.log(myArray);	// Array ["1", "2", "3"]

.slice(startIndex, endIndex) extracts a section of an array and returns it in a new array. Note that startIndex is the first element that you want and endIndex is the first element that you don’t want (not the last one that you do want):

//make an array
var myArray = ["a", "b", "c", "d", "e"];

//get the elements in index 1 and 2
subset = myArray.slice(1, 3);

//print the result
console.log(subset);	// Array["b", "c"]

.reverse() transposes the elements of an array: the first array element becomes the last and the last becomes the first:

//make an array
var myArray = ["1", "2", "3"];

//reverse it and print the result
console.log(myArray.reverse()); 	// Array [ "3", "2", "1" ]

The above is by no means a comprehensive list, just a few of the most commonly used ones. If you would like to see a full list, or see further examples, take a look here.

JavaScript: Loops and Iteration

Last week we learned to use conditional statements to decide whether or not a block of code should run. This week, we will learn another way of gaining control over our code: loops. Loops help us to control how many times a block of code will run, allowing you to doing something multiple times but only write the code once.

for Statement

For example, if you wanted to run a statement 5 times, you could avoid using:

console.log("Hi Jonny!");
console.log("Hi Jonny!");
console.log("Hi Jonny!");
console.log("Hi Jonny!");
console.log("Hi Jonny!");

with:

for (var i = 0; i < 5; i++) {
	console.log("Hi Jonny!");
}

This may look a little complex, but really it’s quite simple. The format may be considered as:

for ([initialExpression]; [condition]; [incrementExpression]) {
  statement
}

Here’s the breakdown…

for tells the browser that it is a for statement.
initialExpression
(var i = 0)
The initial expression runs once, the first time that the loop is called. It is normally used to initialise a variable to be used as a loop control variable (effectively a counter to keep track of how many times the loop has run)
condition
(i < 5)
The condition expression runs every time that the loop runs (so 5 times in the above example). It is used to determine whether or not the loop should run again by testing the value of the loop control variable (in this case seeing if it is below 5)
incrementExpression
(i++)
The increment expression runs every time that the loop runs (so 5 times in the above example). It is normally used to increment the loop counter variable (look back to last week to remind yourself exactly what ++ does)
statement
(console.log("Hi Jonny!");)
The statement is the block of code that runs repeatedly until the condition expression determines that the loop should stop

Why is the less than (<) rather than the less than or equal to (<=) operator used in the condition expression of the above example? If you don’t know this - ask!

Looping through an array

To combine the two things that we’ve learned then, we can combine out for loop and array.length to loop through each member of an array:

/loops through the array and prints out each value in turn
for (var i = 0; i < array.length; i++){
	console.log(array[i]);
}

Looping through each member of an array is one of the most common usages of a loop, so make sure that you are familiar with this bit (expecially how the loop control variable i is being used to access each element in the array one by one).

JavaScript: JavaScript Object Notation

We already know a little bit about JavaScript Object Notation (JSON), because we used a special type of it last week - GeoJSON. GeoJSON is a special variant of JSON that we use to store geographical information. Today we are going to focus upon JSON in general. As we learned last week, JSON is a lightweight data-format that is easy for humans to read and write, and is easy for machines to parse and generate. It is also used across almost all programming languages (in spite of the name), and is a great way to load data into your website.

JSON is called JavaScript Object Notation because it is based upon existing JavaScript data structures (even when it is used in other languages). Fortunately you already know about these structures from above: Arrays and Objects. JSON is always organised using one of these two types:

JSON Objects ({}):

JSON Objects (as you saw last week) are an unordered set of name/value pairs. An object begins with { (left brace) and ends with } (right brace). Each name and value are separated by : (colon) and the name/value pairs are separated by ,(comma). Here is an illustration:

image Image from json.org

JSON Arrays:

JSON Arrays (as you saw earlier today) are an ordered collection of values. An array begins with [ (left bracket) and ends with ] (right bracket). Values are separated by , (comma). This is exactly the same as we looked at earlier in this practical. Here is an illustration:

image Image from json.org

As you would expect in JavaScript, values in JSON can contain objects, arrays, numbers, strings, booleans, and null, because they are essentially just variables!

Using JSON (Parsing and Encoding)

Now this is important: When you pass JSON, though it LOOKS like JavaScript, it is really just a string. To be able to use it, you must first parse (convert) it from a string to data. Likewise, if you have some data that you want to send somewhere using JSON, then you need to convert it to a string first. Fortunately, JavaScript now has a JSON object containing functions that we can use to read and encode JSON.

The JSON object only has two methods:

  • JSON.parse(), which converts a JSON string into data.
  • JSON.stringify(), which converts JSON data into a string.

So if you have a JSON string jsonString and you want to parse it into data, you can simply go:

var jsonData = JSON.parse(jsonString);

If you then wanted to change it back, you would go:

var jsonString = JSON.stringify(jsonData);

Once you have used JSON.parse(jsonString), you have a fully fledged object or array containing the data that you wanted. It’s that simple!

Part 2

Before we do anything else:

Set up a blank map web page using the usual template and set the map view to [53.4807593, -2.2426305], 15

JavaScript: AJAX

One of the most important aspects of Web GIS is being able to visualise and analyse up-to-date datasets from a variety of different sources. That is what we will be focusing upon today: we are going to make a crime map of Manchester using the latest Open Data from the the data.police.uk API, which is a service that allows you to download a month’s worth of crimes in JSON format, with latitude/longitude coordinates attached.

To do this, however, we will need to be able to connect to elsewhere on the web from our website. To do this, we will use AJAX: Asynchronous JavaScript and XML. However, before we do, we should look a little closer at what the A and the X in AJAX means:

Asynchronous If things happen synchronously they happen one after the other. Things happening asynchronously, on the other hand, can happen at the same time. A non-programming analogy might be the difference between stopping what you are doing to cook a pizza (synchronous), and continuing what you are doing and ordering a pizza to be delivered. One is slow and cheap, the other is fast and expensive. Fortunately, modern computers are typically quite powerful, and so we can generally ‘afford’ to download a reasonable amount of data to our websites.
XML XML means eXtensible Markup Language, and HTML is a variety of it. When the term AJAX was first coined, XML was the best way to encode data, and so was included in the name. Nowadays, however, JSON is a much more popular approach, and most AJAX in the modern web doesn’t use XML anymore, though the ‘X’ still remains in the name. I guess that this is at least partly because AJAJ sounds silly…

In reality, many people who talk about AJAX don’t really know what they are talking about. It is a (now inappropriate) term that was coined in an online article, and does not describe any technology, simply the act of using the XMLHttpRequest object in JavaScript to retrieve some data (that may or may not be XML…) from elsewhere on the web.

Here’s how you make one:

/**
 * Make a request for JSON over HTTP, pass resulting text to callback when ready
 */
function makeRequest(url, callback) {

	//initialise the XMLHttpRequest object
	var httpRequest = new XMLHttpRequest();

	//set an event listener for when the HTTP state changes
	httpRequest.onreadystatechange = function () {
	
		//a successful HTTP request returns a state of DONE and a status of 200
		if (httpRequest.readyState === XMLHttpRequest.DONE && httpRequest.status === 200) {
				callback(JSON.parse(httpRequest.responseText));
		}
	};

	//prepare and send the request
	httpRequest.open('GET', url);
	httpRequest.send();
}  

I know that this look complicated, but it’s quite simple really, and fortunately you can just use this template each time that you use it.

All you need to understand is that you give it a url (uniform resource location, a.k.a. web address), it goes and gets the JSON string that is available there, and stores it in the variable jsonString. Most of this work is done in the onreadystatechange event listener, which is called each time the HTTP request returns some information. During a single request, the HTTP request will return several packages of data, each of which are accompanied by a status code that tells us both the progress of the request itself, and what happened. This is why we wait for the XMLHttpRequest.DONE state (meaning that the request is complete) and 200 status code (meaning that the data has been successfully retrieved) to come back, because any other combination of codes means that the data are either not ready yet, or could not be retrieved.

Once the JSON data have been retrieved, the above code parses the JSON String into the JSON Object and passes it to the callback function so that you can process the data. The callback function is defined using the second argument, and is simply a function that is called when the data has been retrieved and is ready to be processed. The term callback refers to the use of a telephone, where if you called someone and they weren’t able to talk, they would call back when they were ready (in this case, when the data are ready).

There are two common ways to define the **callback function **: you can either pass the name of a function that you have already defined like this:

/**
 * This is a normal function that will be used as the callback
 */
function myCallback(jsonData){
	
	// print out the contents of the dataset to the console
	console.log(jsonData);
}

// this is the url from which you will get the data
var url = 'http://blah.blah/blah'

// get the data, and set the callback function
makeRequest(url, myCallback);

Make sure that you can understand how each part of this snippet works (particularly the onreadystatechange) before you move on. If you don’t get it, ask!

Mapping Crime Data with Leaflet

Now we have the ability to use arrays, loops and AJAX, we can start to do some more complicated mapping in Leaflet. To demonstrate this, we will use up-to-date, real Open Data in order to examine patterns of crime in Manchester. Here is the url at which we will access the data:

https://data.police.uk/api/crimes-street/all-crime?lat=53.4807593&lng=-2.2426305

Using what you know about AJAX requests, add the necessary code to your initMap() function in order to retrieve data from the above url and set the callback to crimesToMap.

Now create the corresponding crimesToMap(data) function in the global scope, remembering that it needs to receive a single argument containing the resulting data (see above if you need to refresh your memory). Make the function print the data to the console so that you can see what it looks like.

So now we have a web page with a fully-functional Leaflet map and some up to date crime data, it’s time to put them together!

Converting JSON to GeoJSON

Now, we are taking in data in JSON format (as you can see from the console), and we now need to convert it into GeoJSON so that we can add it to our map. There are several ways that we could do this (including just converting it manually), but I am choosing to do it using the excellent turf.js JavaScript library. turf.js is an Open Source library for “Advanced Geospatial Analysis”, and it contains functionality for a wide variety of GIS operations.

Have a quick look at the turf.js website and see what functionality is available (list in the menu down the left)

Whilst “Advanced Geospatial Analysis” is perhaps a bit of a stretch for what Turf does, it is certainly very handy for dealing with geometry, handling operations such as interpolation and aggregation, and (crucially) for creating GeoJSON. Like Leaflet, Turf has been specifically designed to work with GeoJSON, meaning that it works well with GeoJSON and that it works seamlessly with Leaflet. We will be using Turf a lot from now on, so today is a good chance to get used to it!

Add the necessary <script> tags to the <head> of your website to download Turf.js from:

https://cdn.jsdelivr.net/npm/@turf/turf@5/turf.min.js

Like GeoJSON (and most GIS platforms), Turf recognises three main classes of geometric feature:

point An infinitely-precise point in space (used to represent discrete locations, e.g. a crime)
lineString A string of two or more connected points (used to represent linear features, e.g. a road)
polygon A ring of three or more points that enclose an area (used to represent area features, e.g. a woodland)

As you might expect from a JavaScript Library, Turf represents each of these classes as objects, each of which has a constructor (turf.point(), turf.lineString() and turf.polygon() respectively). We are going to use these object constructors to turn our JSON crimes into GeoJSON points on a map. To work out how we can achieve this, we need to understand how the point contstructor works, and also have a look at a crime in JSON format:

[
	{
		"category":"anti-social-behaviour",
		"location_type":"Force",
		"location":{
			"latitude":"53.487670",
			"longitude":"-2.261138",
			"street":{
				"id":726227,
				"name":"On or near Cannon Street"
			}
		},
		"context":"",
		"outcome_status":null,
		"persistent_id":"35be930198751601b9ade72536ccaa47a6f9e1d8e6cd51432a41ccea0c4ccf8f",
		"id":70451018,
		"location_subtype":"",
		"month":"2018-12"
	},
	...
]

As you can see, this is a Array of Objects. To turn this into turf.point objects, we are going to need to extract the .location.latitude and .location.longitude from each crime, and pass them to the turf.point() constructor and add the results to an array. We will then collect them together into a Feature Collection using the turf.featureCollection() constructor, and finally convert them to a Leaflet L.geoJson layer and add them to the map using the L.geoJson() constructor and .addTo() function.

Though this maybe sounds like a lot of stuff - it is really just a collection of tiny steps, each of which is quite simple. This is the best way to approach programming (and many other things in life!) - if something looks too hard / complicated, break it up into smaller, manageable chunks until you can see exactly what needs to be done! Let’s give it a go, one step at a time:

1. create an empty variable in the global scope called geojson, and an empty array in the local scope called points

2. use a for statement to loop through the data array that you passed into this when you called it in makeRequest

3. inside the loop, extract the data that you need from the crime object and pass to the turf.point() constructor like this:

	points.push(turf.point( [ parseFloat(data[i].location.longitude), parseFloat(data[i].location.latitude) ], {} ));

4. after the loop has completed, you should have an array of turf.point objects. You can now convert this into a Feature Collection (allowing you to treat the many points as one layer) like this:

//convert array of points to a feature Collection
var pointsCollection = turf.featureCollection(points);

5. finally, add the points to a L.geoJson layer and load into the map.

//convert the feature collection to a GeoJSON layer and add to the map
geojson = L.geoJson(pointsCollection, {}).addTo(map);

Complete the above 5 steps inside your crimesToMap() function

If all went to plan, you should have something like this:

Map with Crime Points

Note: in this section, I will be using screenshots rather than live maps. This is because having several maps downloading the entire dataset several times across 70 students would be a lot of data transfer, and it might slow things down a little in the lab!

Or if you zoom out a little:

Zoomed Out Map with Crime Points

What do you think? Is this a good map?

Hopefully, your answer to the above questions should be NO!. This illustrates one of the most common problems with Web Maps that dates right back to the earliest days of web map mashups - too much data obscuring any patterns that might otehrwise be observed. Clearly, programmers of our calibre can’t be seen to be producing maps like this, so we need to do something about it. Fortunately, we can turn back to Turf.js to help with this…

Aggregating data with Turf.js

When we have lots of point data like this (as is very common in web applications), a nice solution to make it more easily understandable is to aggregate it into regions of a given size and shape that tesselate together. Typically, these will be either squares, hexagons or triangles, like this:

Tesselations Image Source

If we can aggregate our point data to these shapes, we can then colour then according to the number of points that is within each one (just like how we coloured the countries by population last week, rather than had a point for every single person…). The good news is that Turf.js makes this process easy for us, as it has functions to both create the tessalating shapes and count the number of points within each polygon. Isn’t that convenient…

Let’s give it a go then - we’ll say that we want to aggregate our crime points to a hexagon grid, where each hexagon is 100m apart. Once again, we start by breaking down our problem into a number of small steps:

  1. Build a hex grid
  2. Loop through each polygon in the grid and work out how many points are contained within it
  3. Save a new turf.polygon containing both the hexagon geometry and the number of crimes contained within it as a property
  4. Convert the array of polygons into a Feature Colection
  5. Convert the Feature Collection into an L.GeoJson layer and add to the map

See, not so scary, right? Let’s give it a go…

1. immediately after you have made your pointsCollection, use the turf.hexGrid() constructor to make a hexagon grid at 100m intervals. We will use turf.bbox() to define the extent of the hex grid, which returns the bounding box of the feature(s) that you pass to it as an argument. In this case, therefore, the hex grid will have the same dimensions as the crime points Feature Collection:

// build a 100m hex grid to cover the area of interest
var hexgrid = turf.hexGrid(turf.bbox(pointsCollection), 100, {units: 'meters'});

2. initialise an empty variable called pointsWithin and an empty array called polygons. Now create a for statement to loop through hexgrid.features.

3. inside the loop, use turf.pointsWithinPolygon to work out how many crime points (from your pointsCollection) are within each hexagon polygon:

// calculate the number of crimes within that polygon
pointsWithin = turf.pointsWithinPolygon( pointsCollection, hexgrid.features[i]);

4. then (also inside the loop), make a new polygon using the turf.polygon constructor, using the coordinates from the current hexagon (hexgrid.features[i].geometry.coordinates) and adding a property called crimes which is populated with the number of crimes contained within it (crimes: pointsWithin.features.length - the length of the resulting array of points from turf.pointsWithinPolygon):

// create a new polygon, combining the old geometry with the new 'crimes' property
polygons.push( turf.polygon( hexgrid.features[i].geometry.coordinates, { crimes: pointsWithin.features.length }) );

5. Once the loop is complete, you should have an array of turf.polygon objects, just as you previously did with turf.point objects. Accordingly, you can now convert them into a turf.featureCollection in the same way.

6. Finally, change your L.GeoJson statement so that you are adding your hex grid instead of the points.

Carefully follow each of those 6 steps, making sure that you understand each one fully (ask if not).

If all goes to plan, you should have something like this:

crimes hexagon map

Adding Style and Interactivity

Nice! Now, whilst this doesn’t do much to show us crime patterns yet, all of the information that we need is in there…

Using your skills from last week and the brief notes below, colour in the crime map so that each hexagon represents a number of crimes, and hovering the mouse over a hexagon reports the number of crimes in a map like this:

Notes:

Here is the legend to use:

0 - 10 crimes
10 - 20 crimes
20 - 40 crimes
40 - 60 crimes
60 - 80 crimes
80 - 100 crimes
100 - 120 crimes
120+ crimes

Make sure thet you set the opacity in the object that you return from your style() function. This is what I used (though feel free to style it however you like!):

{
	weight: 1,
	color: 'white',
	fillOpacity: 0.6,
	fillColor: getColour(feature.properties.crimes)
};

You will have noticed by now that it takes a long time for the data to load from the police servers. To make it clear to the user that you are waiting for it to load (rather than it hasn’t worked), you can add this statement to your initMap() function (think carefully about where it should go!!). If you put it in the right place, it will mean that your map says:

Loading…

…until the data actually arrives, which is a nice touch!

// set the info box to 'loading'
info._div.innerHTML = '<h2>Loading...</h2>';

Finished!

Some of the material on these pages is derived from the excellent Leaflet Tutorials and Mozilla Developers websites. Mozilla Developers by Mozilla Contributors is licensed under CC-BY-SA 2.5.