Quantifying Green Values of Drainage Systems

This is the very first blog post about my research project. Although I am not going to write much about it this time, I do have a Prezi presentation here. Enjoy!

Visualising Crime Hotspots in England and Wales using {ggmap}

Two weeks ago, I was looking for ways to make pretty maps for my own research project. A quick search led me to some very informative blog posts by Kim Gilbert, David Smith and Max Marchi. Eventually, I Google'd the excellent crime weather map example by David Kahle and decided to stick with the gg-style approach.

Thanks to David Kahle and Hadley Wickham who ramped up that example and subsequently developed the {ggmap} package, making maps in R can be really intuitive and fun!

I wrote a wrapper function that takes a location within England and Wales, downloads crime data around that location over a certain period of time and creates crime weather plots. This blog post discusses the data used, methodology and the wrapper function with some worked examples. The codes are available here.

(Nov-2013 Update: I have updated the codes and created a web app using Shiny and ShinyApps. For more information, please read this new blog post.)

Data

The street-level crime data is one of the 9,000 datasets available from data.gov.uk. The data can be downloaded systematically via the Police API. The latest version of the API no longer requires authentication.

The following URL can be used to obtain crime records at street-level within a one-mile radius of a single point. The parameters required are latitude, longitude and month. The downloaded data is in JSON format which can be converted into R's data format using the {RJSONIO} package.

Example URL: http://data.police.uk/api/crimes-street/all-crime?lat=52.629729&lng=-1.131592&date=2012-04

Methodology

The methodology can be summarised in the following six steps:

1. Obtain latitude and longitude of a user-defined location using ggmap::geocode.
2. Download crime data via the Police API as discussed above.
3. Convert JSON into a list and then a data frame.
4. Download a base map from Google using ggmap::get_googlemap.
5. Covert the base map into a ggplot object using ggmap::ggmap.
6. Add multiple layers on top of the base map using the data frame like a normal ggplot.

For more details, check out the functions in the codes:

• "get.data" and "list2df" for steps 1, 2 and 3
• "visualise.data" for steps 4, 5 and 6

Wrapper and Worked Examples

The wrapper function looks like this ...

crimeplot.wrapper <- function(
  point.of.interest = "London Eye",  ## user-defined location
  period = c("2013-01","2013-02"),  ## period of time in YYYY-MM
  type.map = "roadmap",  ## roadmap, terrain, satellite or hybrid
  type.facet = NA,  ## options: NA, month, category or type
  type.print = NA,  ## options: NA, panel or window
  output.plot = TRUE,  ## print it to a png file?
  output.filename = "temp.png",  ## provide a filename
  output.size = c(700,700)) ## width and height setting                              

... given the location, time period and a few more graphical settings, the wrapper can produce a crime weather map. The following worked examples illustrate the usage.

Example 1 - All crimes around London Eye from Jan-2013 to Apr-2013

Comments:
Here we can see a huge crime hotspt in the Soho district of London - an area full of bars, restaurants, theatres and nightclubs (did I mention Chinatown?)

Codes:
## Define the period
ex1.period <- format(seq(as.Date("2013-01-01"),length=4,by="months"),"%Y-%m")

## Use the wrapper
ex1.plot <- crimeplot.wrapper(point.of.interest = "London Eye",
                              period = ex1.period,
                              type.map = "roadmap",
                              output.filename = "ex1.png",
                              output.size = c(700,700))

Example 2 - Typical crimes and traffic incidents around London Eye from Jan-2013 to Apr-2013

(Note: click on the image to see original image in higher resolution)

Comments:
Now we seperate the data from British Transport Police (BTP) and all other forces (Force) using the facet function in {ggplot}. We can see a traffic black spot on the other side of River Thames.

Codes:

## Define the period
ex2.period <- format(seq(as.Date("2013-01-01"),length=4,by="months"),"%Y-%m")
 
## Use the wrapper
ex2.plot <- crimeplot.wrapper(point.of.interest = "London Eye",
                              period = ex2.period,
                              type.map = "roadmap",
                              type.facet = "type",
                              output.filename = "ex2.png",
                              output.size = c(1400,700))

Example 3 - Monthly crimes in Manchester for the year 2012 on a satellite map

(Note: click on the image to see original image in higher resolution)

Comments:
Using the facet function on "month", we can look at the changes in patterns over time. Looks like there is not much seasonality in Manchester as the crime hotspots remain hot over the year.

Codes:

## Define the period
ex3.period <- format(seq(as.Date("2012-01-01"),length=12,by="months"),"%Y-%m")

## Use the wrapper
ex3.plot <- crimeplot.wrapper(point.of.interest = "Manchester",
                              period = ex3.period,
                              type.map = "satellite",
                              type.facet = "month",
                              output.filename = "ex3.png",
                              output.size = c(1400,1400))

Example 4 - Crimes by categories in Liverpool from Jan-2013 to Apr-2013 on a hybrid map

(Note: click on the image to see original image in higher resolution)

Comments:
Now we separate different categories of crimes. It is interesting to see that only a small part of the city is affected by shoplifting and other theft while burglary, arson and vehicle crimes are very common problems in Liverpool.

Codes:

## Define the period
ex4.period <- format(seq(as.Date("2013-01-01"),length=4,by="months"),"%Y-%m")
 
## Use the wrapper
ex4.plot <- crimeplot.wrapper(point.of.interest = "Liverpool",
                              period = ex4.period,
                              type.map = "hybrid",
                              type.facet = "category",
                              output.filename = "ex4.png",
                              output.size = c(1400,1400))

Further Work

Further work is needed to ...
1. optimise the codes for "list2df" transformation (At the moment it is quite slow. I tried lapply but it didn't give me back the desired data frame format but I know there must be a solution.)
2. better automate the graphical settings for output resolution, font size etc.
3. make it interactive using {Shiny}

(Nov-2013 Update: I have improved (1) using plyr::ldply and done (2 & 3). See this.)

Acknowledgement

I would like to thank Yanchang Zhao for his excellent book titled "R and Data Mining: Examples and Case Studies" which encouraged me to shift from MATLAB to R. All embedded codes were Created by Pretty R at inside-R.org.

Key References

1. David Kahle's original crime map.

2. Police API documentation.

Colour it up: my quest to master ggplot2 (part 2)

Learning-by-doing: my quest to master ggplot2 (part 1)

R, where should I start?

This is a dynamic post which I will continue to update whenever I find something new. Hope you will find the following links useful.

Online Courses for Learning the R language

1. Try R from Code School
2. Computing for Data Analysis by Roger Peng (Coursera)
3. Concepts in Computing with Data by Phil Spector (Berkeley)

Free Documentations for Learning the R Language

1. R for Beginners by Emmanuel Paradis
2. R Graphics by Paul Murrel
3. ggplot2 (official documentation)
4. Advanced R Programming by Hadley Wickham

Online Courses for Data Mining with R

1. Data Analysis by Jeff Leek (Coursera)
2. Data and Computing Fundamentals by Danny Kaplan and Libby Shoop (Macalester College)
3. Statistical Learning (Stanford OpenEdX)
4. CS109 Data Science (Harvard)

e-Books for Data Mining with R

1. R and Data Mining: Examples and Case Studies by Yanchang Zhao (Really useful worked examples!)
2. Data Mining Algorithms in R (Wikibooks)
3. The Elements of Statistical Learning by Trevor Hastie, Robert Tibshirani and Jerome Friedman
4. Introduction to Data Science by Jeffrey Stanton
5. Forecasting: Principles and Practice by Rob Hyndman and George Athanasopoulous
6. Bayesian Computation with R (Free Kindle Edition): UK Link, US Link (Aug 2013 Update: No longer free)
7. 12 Free (as in beer) Data Mining Books

R Tutorials

1. Twotorials by Anthony Damico (learning new tricks from short 2-min videos)
2. Revolution Analytics Free Webinars
3. ggplot2 Graphics Cheat Sheet
4. 10 tips for making your R graphics look their best
5. Making Maps with R
6. Compiling R 3.0.1 with MKL support
7. Flowing Data - Tutorials
8. Quick-R
9. R-Uni (A List of Free R Tutorials and Resources in University Webpages)

Interesting Blogs and Articles

1. R-bloggers
2. Statistics Blogs
3. Whizage by Thia Kai Xin
4. R Resources by Vivek Patil
5. 100 most read R posts in 2012
6. "R" you ready?
7. The Angry Statistician
8. VizWiz - Data Visualization Done Right
9. FastML - Machine learning made easy
10. ggplot2 Blog
11. Spatial.ly - Visualisation, Analysis and Resources
12. Vistat - a reproducible gallery of statistical graphics
13. The Shape of Data
14. Animated Graph for Data People
15. is.R()
16. 60+ R resources to improve your data skills

Useful R Packages

1. Ten R packages I wish I knew about earlier (Before you do anything, read this blog post first!!)
2. caret (short for Classification And REgression Training) for a simple way to train and fine-tune model using different algorithms
3. ff and bigmemory - two packages to solve memory issues with big datasets
4. quantmod for financial modelling
5. foreach and doSNOW for parallel computing in R

Interactive Development Environment

1. RStudio - a really nice IDE for R
2. RStudio Server Amazon Machine Image by Louis Aslett (Wanna run RStudio on Amazon EC2? Try this!)

Other Useful Tips

1. Re-installing packages in R version 3.0.0

Data

1. UC Irvine Machine Learning Repository
2. Time Series Data Library created by Rob Hyndman
3. Quandl (like a Wikipedia for Time Series Data)

Blend what?

Why?

Over the years I have learned quite a few things about machine learning but I have never thought of writing them down properly. Too often I can't figure out exactly what I did when I look at my old codes. The time is NOW!

More importantly, I have fallen in love with the R programming language and the massive amount of useful packages from the R community. I want to talk about tricks, tools and useful resources for data mining with R (and sometimes my old favourite Matlab) here. 

Bayesian Ensemble Learning

One of the interesting tricks I learned is called "Bayesian Ensemble Learning". It involves combining (i.e. blending) different models to improve overall prediction accuracy. Although it has its downside (e.g. computationally expensive, difficult to interpret ...), it is certainly my favourite data mining technique at the moment. I also decided to name this blog with it long before I start writing this first post!

Research

There is also a need to promote my own research project online. So I guess there will be times I talk about drainage design, green infrastructure and decision support systems. This is not the main focus of the blog but I will try to create some funky graphs and explain my research to a wider audience when the time is right (i.e. when I eventually master the art of graphics in R).

OK, so here we go, this is my journey into the wonderful world of data science!