I recently attended the Amazon Web Services (AWS) Public Sector Summit in Washington, DC, on June 20-21, 2016. One reason was to accept an award, and the other was to connect with various AWS customers as an AWS Technology Partner. There were several great sessions dealing with Open Data, but one that has really stayed with me is the presentation by Tom Schenk (pictured above), the Chief Data Officer for the City of Chicago, on the City’s OpenGrid project.
OpenGrid seeks to bridge a gap in current Open Data practice by making real-time, event-based data accessible and useful to everyone. This is of enormous significance as local governments like the City of Chicago increasingly embrace IoT to achieve Smart City goals (see State and Local Governments Embrace IoT, Including in Smart Cities), and it’s a powerful testament to the Emanuel Administration’s commitment to leveraging Open Data.
It’s also already been done before.
It’s been done in a packaged way that makes OpenGrid-style capabilities as accessible and affordable to the Champaigns and Auroras of the world as to the Chicagos. But first, a bit of background on the OpenGrid project.
What is OpenGrid?
OpenGrid was developed in 2016 off the effort to develop a unified data platform called Windy Grid ahead of the NATO Summit held in Chicago in 2012. OpenGrid is an open source platform that excels at providing either real-time or historical insights of event-based data. The data used in OpenGrid requires a time and location. As data is ingested in OpenGrid, the data indexed by location and time, and so geospatial queries will show what data is available for a specified geometry, like a neighborhood.
Tom described a challenge the City of Chicago’s Open Data portal faces: the Open Data is in a machine readable, but not a human readable, format.
This required the integration of an additional open source platform, Plenario. Plenario ingests Open Data from Socrata and CKAN data portals. “Developed by the University of Chicago, it’s a complementary project to OpenGrid to navigate your Open Data even easier.”
Plenario is a centralized hub for open datasets from around the world, ready to search and download. All data in Plenario exists on a single map and a single timeline, making it incredibly easy to access multiple datasets at once – especially those originally housed at different data portals.
In order to include internal data into the analyses that the City of Chicago performs on OpenGrid, a MongoDB instance is required to handle the datasets for inclusion into OpenGrid.
In sum, the City of Chicago’s OpenGrid initiative encompasses four core technologies: one commercial Open Data platform (Socrata), two large scale open source programs (OpenGrid.io and Plenar.io), and the deployment and management of an unstructured database (MongoDB).
Towards a lighter approach of Open Data
It’s an impressive undertaking and a resourceful solution to the shortcomings of the available data platforms in the open data space. But my career path led me to another solution, OpenDataSoft, which also addresses these shortcomings, but does so with a single turnkey Cloud solution, accessible to all cities, even those not blessed with Chicago’s large budget and large active civic community.
In Part Two of this post, I’ll discuss the ways the journey that the City of Chicago has been on to achieve a very useful data portal mirrors the journey of my colleague Jason Hare and myself in finding a better platform to make Open Data (and internal data) more approachable and useful to everyone.