The other day, I flooded a customer website licensed for 75 datasets with 318 additional in five minutes while using the OpenDataSoft ArcGIS Harvester. Yup. I did that. I felt not so much like “Darth Vader” as much as I did “Dark Helmet” (Rick Moranis in Spaceballs, see below if this reference goes over your head).
I was using OpenDataSoft’s ArcGIS Harvester. It’s still in private beta, but will likely be available by the end of December (consider it a Christmas gift to you!) for data publishing. Luckily, the Harvester also came with a delete button so the customer was not affected. I took the usual ribbing from my co-workers. David Thoumas added that we do not need “too many datasets”.
I was awed with the power of the Harvester.
I had seen it work a year ago and take most of an afternoon to mine data. I innocently put the root of the GIS server into the Harvester, and suddenly it was pulling in data at “Ludicrous Speed”. Ian Henshaw noted we hit hyper drive and “They’ve gone to plaid”.
Several titles for this blog post came to mind:
- “With Great Power Comes Great Responsibility”
- “Young Jedi uses too much Force in Releasing Data”
- “We’re Gonna Need a Bigger Boat”
- “How Jason Hare Blew Through a 75 Dataset License in 5 minutes”
At first I felt like I was aiming the Death Star at Planet Alderaan. Heck yes I was going to show some folks how to ingest data with our giant Hoover Machine. As the datasets quickly tallied I realized I was ingesting an entire ArcGIS server. It went from 1 to 15 in about 2 minutes. By 3 minutes we were at 150 and by 5 minutes we had all 318 datasets.
What I learned and What You Should Know
The OpenDataSoft Harvest Tool can pull data, with rules, from an ArcGIS server with two clicks. We can harvest other types of data as well. This was a learning moment for me. “Choose wisely”. We can also harvest this data very quickly (remember Ludicrous Speed).
No worries. After David Thoumas grounded me from using the Harvester I got a quick lesson from our awesome developer team. I am no longer considered a threat to our Open Data community or customers.