IDC ANALYST CONNECTION March 27, 2018 The goal of Smart Cities, and the solution providers that serve them, is to transform…
Explore New York City Yellow Taxi trips Data with OpenDataSoft
Last week, New York City opened data from millions of taxi trips. 165,114,361 to be more precise. At a time when war rages between taxis and Uber, just a few days after the clash between NYC’s mayor DeBlasio and Uber, it looked like a good a idea to play with those data by uploading them into the OpenDataSoft platform and displaying them on a map and as graphs & charts!
Some quite usual facts
As we could expect, there are more and more taxi trips during the week, peeking on Saturday.
The month by month evolution seems to show more taxi trips during Spring and Fall:
It would be interesting to wait for 2015 data to see if there is a real pattern though.
Reverse Engineering of Taxi pricing
Here are, given a trip’s fare amount, the average distance and average duration of the trip:
It may be worth trying to recreate a model of pricing from those data and create an app that tells you when you are way too far the average, or by filtering on the pickup location checking if part of the New York City population is being disadvantaged as some people claim.
About data quality
For a dataset this huge, there are not a lot of errors or bad data, but absolutely clean and perfect datasets are really unusual. Data visualization and mapping are good ways to find some incorrect data, especially when you have 160M rows to check!
The first, and most basic, example is bad geo coordinates:
I know Uber has launched Uber Boat in Istanbul but most of the markers far from NYC are probably incorrectly located.
With the same idea, we can see that there are both very long trips – 13 days – that may need more investigations, and trips with negative duration, that don’t need any investigation:
Smart Cities 101
The harder the funnier, we’ve created a heatmap of every pickup locations in 2014 (~160 million, remember) and add the Subway entrances as a layer. It’s pretty amazing to navigate that easily in those huge datasets!
By implementing some route calculation between the subway stations, we could compare every taxi trip with a public transport trip and understand better how people behave, why, and what the city can do to improve their lives. That would be a nice first step in the development of a Smart City…