The Memory Leak

Every weekday afternoon around 5pm or so, the quarry several miles from my house blasts away a new layer of limestone. Whenever I feel the slight tremor from the blast, I feel as if valuable data is going uncollected. Could this data somehow be used to make a 3D model of the Edwards Aquifer?

An overlooked aspect of a GPS receiver is the highly accurate clock it contains. It seems like GPS receivers (and their clocks) could be attached to low-cost seismometers. The common clock of the GPS would allow independently collected data to be synthesized. I wonder if there might be enough geo-geeks in my neighborhood to build and deploy an array of these. There’s another quarry about 10 miles away, where I presume they also blast, so not all blasting signals originate from the same location. There will also soon be a lot of highway construction starting in this area, which will likely involve yet more blasting. Once the data is collected, it could be forwarded to a central site for processing. With all the recent innovations in oil exploration seismic processing, it seems like this data could be continually synthesized for model refinement. I’m not a seismologist, but would be interested in hearing from one regarding the plausibility of this.

While the geologists over at Edwards Aquifer Authority have done an fine job, it would be nice if they could build some detailed 3D models and serve them out on the web.

In his book The Unbearable Lightness of Being, Milan Kundera makes an interesting case that kitsch is the absolute denial of shit. Lately I’ve wondered if this implies other activities that deny the existence of shit are really exercises in kitsch creation.

Is Linq Kitsch?
Google for linq and plumbing and notice how many hits you get. Plumbing code is discussed with a sort of subtle condescension. The message is that plumbing is dirty, uncreative work - something better not thought about, hidden below the surface.

While Linq looks exciting, I wonder if we run the risk of abstracting the database to the point we overlook solutions requiring an understanding of internal design. Will someday our children forget the obvious solution: sometimes you just need to jiggle the handle after flushing objects (rows). Linq must be smart, but I can’t help but think there must be cases where plumbing your own O/R mapping might work better. It’s really not that bad, if you’ve ever played around with piecing together PVC pipe you probably realize that plumbing can actually be fun, with lots of room for creativity.

And it’s not just programming, but other aspects of GIS. Physicians seem to get credit for the longer lifespans we enjoy these days, but I wonder if more credit is due to plumbers. John Snows’ cholera maps revealed a plumbing problem. As far as I can tell he wasn’t using medical knowledge so much as taking on the role of a plumber and recognizing that, yes, shit does truly exist.

But beyond Snow’s maps, sewer datasets receive little attention. Could it be our kitsch sensibilities prevent us from publishing them on the web? Where I live there is a sign saying “Entering Edwards Aquifer Recharge Zone”, but never a sign that says “Warning Sewer Line is Here”. Last year a road construction crew broke a sewer pipe over the aquifer. They reported it twice. It leaked for a couple of weeks as the bureaucrats bungled through what could best be described as kitsch performance art: two different agencies denying the existence of the sewer line. Maybe we need more sewer museums to combat the kitsch, and give plumbing the respect it deserves.

Shortly after Zillow announced they are releasing their neighborhood boundaries, The Virtual Earth blog described how this data can be chopped up by MapCruncher into tiled vectors for more efficient display.

I don’t see a realistic editing workflow though.

Let’s pretend I’m a realtor with in-depth knowledge of the boundary for a particular neighborhood and would like to edit Zillow’s boundary. Let’s assume my web map editing skills are limited to editing polygons in Google’s My Maps.

Editing the Neighborhood
I would like to go to a Zillow Web map displaying the neighborhoods (as vector tiles), choose a neighborhood, provide a Google My Maps account URL and click a button that says “edit”. Zillow would then would forward the outline of the selected neighborhood to My Maps account. I would then edit the neighborhood polygon using My Maps editing tools.

Is Google exposing My Maps in their API? When I search the Google Maps API reference for “My Maps” I don’t see anything. It would be great if I could leverage it as an editor.

Updating the Tiles
After editing the neighborhood I would then go back to the original neighborhood at the Zillow site submit the My Maps URL and click “Update”. Zillow would then retrieve the edited polygon, review it, and submit it to a MapCruncher queue. The MapCruncher process would read the queue and update the affected tiles. Maybe they could include a URL to my realty site in exchange for me improving their data. An edit history list could also be maintained.

HPC Cluster for MapCruncher
Granted there would probably not be a lot of updates to this dataset, but if there were, I think the MapCruncher queue would quickly fill up. I wonder if MapCruncher could be deployed to a cluster using Windows HPC Server 2008 cluster. If such a system could be implemented, it seems like, for example, a crowd of appraisal districts could be harnessed to maintain tiled vectors of parcels across an entire state.

I find evolution to be a fascinating design process, yet sometimes I wonder if evolution can ever be Agile. The key thing about evolution is that time is plentiful. Agile time is broken up into short sprints.

Perhaps the sprint can be thought of as punctuated equilibrium event.

Evolutionary design is not without its faults - I’m typing this on a QWERTY keyboard. But does Agile avoid QWERTY decisions? It seems like the Agile tenet of delivering working software early can easily result in QWERTY inertia. If an early iteration release delivers great enough value, users run the risk of using the tools so much that that they become ingrained with the UI (or framework).

What is to keep an arbitrary design decision in an early iteration from freezing into place and lasting for many generations?

Reading about Peter Batty playing with his IPhone makes me wonder about business models for calibration of geolocation technologies.

It seems to me that if I make a call from a phone equiped with GPS in an area covered by cell towers or WiFi, that my GPS location could be collected and used for calibration. When subsequent calls are made from that area with phones (such as IPhones) that lack GPS, a correctional offset could be applied. Since my GPS location has value, seems like I should be offered some way to sell it. Or trade it for minutes?

Is Google’s My Location already doing this? If so, then since Google owns the locational bit of info, calling it “My Location” seems a bit misleading.

Last month a spatial app, Live Traffic, won 2nd place ($5000) in the Nokia Open C challenge:

Live Traffic is a traffic assistance software that provides real-time traffic volumes, developed by a project team of four developers, led by by Pu Zhihua of Shanghai, China. Live Traffic adopts FCD (Floating Car Data) technology to acquire road traffic information anywhere anytime, and publish mapped traffic information to Nokia phone users via GPRS or EDGE connections. The developers ported 2,500 lines of code via Open C.

First place winner got $10,000 for … oh, don’t ask, suffice to say it is yet another video app.

Nokia is not a member of the Open Handset Alliance, which is promoting the Android API designed by Google, so I guess they are competing for market. With the top award for Google’s Android Developer Challenge set at $275,000, I bet Android beats out Open C.

I’m not certain, but it looks like an entry to the Android challenge could leverage the MapView. So maybe the grand prize winner will be something useful instead of yet another way to watch TV.

The Revolution will not be on Streaming Video
Like the folks at the One Laptop Per Child program, it seems like the US geospatial community is too focused on laptops/desktops as a hardware platform. At Large points out a good article about mobile phones in Africa (”Can Cell phones Save the World?”). I like the system described here by Manobi, but it seems like it needs spatial enablement.

What we need are some slick mobile geospatial apps for 3rd world development.

In a country like Bangladesh, for example, almost every everyone has a mobile phone while few have (or need?) laptops.

I suggest developing a mobile app that connects farmers, truckers, and packers. Imagine an eBay like system on mobile phones that connect these three groups of users. Location would be important since transport costs are such a large factor. Farmers (who rarely own trucks) would contract to sell produce to packers and have it delivered by truckers. The packer they choose would be based on trucking cost subtracted from the price offered by packer. Truckers would offer prices based on routing costs, (including deadhead), coordinating between multiple pickup locations.

With three people involved in a transaction (farmers, truckers, and packers), would it be possible for an eBay-like rating system?

If so, then the payoff could be substantial.

Here’s an interesting video where Google describes how they use MapReduce to build connectivity in their street data. In ESRI terminology, this how they clean and build topology using parallel processing. They also briefly mention using it to render map tiles.

They don’t go into detail, but apparently those of us outside Google could do this sort of thing using Hadoop on Amazon EC2.

A challenge with tile caches is keeping them up to date with the vectors they depict. Here is how ESRI does it. I think ESRI needs to allow us to scale tile generation across a large number of cpus the way Google does. The licensing model needs to allow this. It seems like opensource Geo software on a paid AMI could be coupled with Hadoop on EC2 to do this.

Once that happens, an agency like a state data center could rebuild tile caches on EC2/S3 nightly from, for example, a statewide vector layer of parcel maps.

I’ve heard rebuilding a geodatabase topology for the nationwide census takes over 24 hours. I bet a MapReduce approach would be much faster for this too.

As suggested by Dave Bouwman and Ron Bruder, I’ve played around a bit with ArcGIS diagrammer developed by Richie Carmichael. I like it. It would be great to see something like this offered as a fully supported product.

I’ve also watched some of the Linq to SQL videos at MSDN. Keep in mind that Linq is extensible, so while currently there are only a few providers, expect soon to see other Linq providers appearing, for example Linq to Google.

It sure would be cool if ESRI wrote a Linq to Geodatabase provider.

If ESRI doesn’t do something, I suspect many will opt to bypass ArcSDE altogether and access SQL Server directly via Linq to SQL, once it supports spatial types.

I suppose if ESRI did it right it would support a user experience similar to the one in this video, but in C# of course :). I suppose, too, it would involve writing a designer with similar look and feel as ArcGIS diagrammer.

At Large points out an interesting article at Hackszine about geolocation in Kindle.

I’ve written a proof of concept editor extension based on the ideas outlined in my previous post, plus some helpful feedback from Brian Flood (thanks!).

The solution, which includes an installer and test file gdb, has been uploaded to arcscripts, right here.

The editor extension maintains a generic List of IExtensions. The extensions in this list are instantiated at OnStartEditing from source code in a table called SourceCode in the edit workspace.

There is also a command provided on a commandbar that allows you to browse and load a source .cs file into the SourceCode table, after verifying that it compiles without errors.

Note that the “using” statements need to include a full path name to the assembly files referenced by the source code.

I’m wondering if this approach might be easier than class extensions as well as easier to maintain. I haven’t tried it, but I suppose it would be possible to have it work with shapefiles as well.

The potential uses of dynamic compilation are intriguing. I’d really like to try this in a IServerObjectExtension. More on that later.