The Memory Leak

Something doesn’t make sense here. I often have heard ESRI describe Web2.0 mapping as useful, but not really GIS. To be true GIS, ESRI claims, geoprocessing capability must be provided. But then when I look at ArcGIS Online (Beta), I don’t see any geoprocessing services. I thought I saw this demo’d somewhere, but don’t see anything. I suppose ESRI is working on this - otherwise the name should be changed to ArcData Online.

What I would really like to see is a way for ESRI Business Partners to author geoprocessing services and publish them onto ArcGIS Online. Partners should be allowed to specify a fee that is charged to the user when someone uses their GP service. Of course ESRI would get a percentage of this. I would also like to build geoprocessing services by chaining together other people’s services, maybe even following the chain of resposibility pattern.

A lot of business partners sell desktop (ArcView) for ESRI. At some point ArcGIS Online will be competing with these partners. By allowing partners the ability to author and publish geoprocessing tools on ArcGIS Online, they could continue to add value to ArcGIS.

News.com has a good update on Cloud computing.

John M Willis has posted a helpful summary of cloud computing vendors.

He doesn’t mention Sun Grid. So what’s the difference between a cloud and a grid? Looks like I’m not the only one confused.

This is an elaboration of the MyHouse concept I mentioned in a previous post.

I’d like to have my pantry and refrigerator equiped with RFID so they know what is in stock. I would have a web page where I search for dinner recipes that are presented in a ranked fashion, ordered by how few extra ingredients I need to pick up from the store. In cases when I’m away from home I’d like an option to specify a path I normally travel home on. The web site would rank the recipes based on availability at stores along the path. Since I’m making a buying decision it could be fueled by ad revenue.

It seems like the CMU coke machine should have evolved into this already.

This is an interesting video: Microsoft presenting Dryad research to Google. Dryad is claimed to be a “superset” of MapReduce.

It seems like the complexity of Dryad could be hidden behind a modelbuilder-like interface.

I wonder how valid it is to view this video and think about modelbuilder terminology, replacing the word “vertex” with “tool”, “channel” with “intermediate data”. Note that, like earlier versions of geoprocessing, Dryad cannot handle cycles (loops).

Fantom Planet’s post suggests ESRI is paying more attention to volunteered geographic information VGI. Fantom Planet’s concept of Generation M: Me, MyMaps, MySpace is useful. Once the Generation M becomes home owners, I think we will see MyHouse, a web site that lets me use geography to organize my activities in my neighborhood.

Take flood plains for example. Recently I saw in the newspaper that FEMA was updating the flood plain maps. But wait, Generation M doesn’t read newspapers - MyHouse must get GeoRSS alerts for things like this.

The old maps are inaccurate …

To see the draft updated maps go here I must go here and pull up a very Gen-X ArcIMS site.

We have 90 days to file protests against the new maps. If I had other info I could mashup against the map, it would make it much easier to explain my protest. Protestors should be allowed to mark up a public map where neighbors can review.

FEMA needs to recognize VGI and provide guidelines for the local authorities coordinating the map updates. The guidelines would encourage (RestFul) practices that would allow sites like MyHouse to integrate. The information volunteered by protestors is crucial to allow creation more accurate maps.

On the political side, there are likely other activities in FEMA that could benefit from VGI as well, like for emergency response. So maybe when the next administration cleans house VGI can be introduced comprehensively. (Now when is that California primary?)

On the technical side, once ESRI fixes ElementGraphicsLayer.WriteXml and provides the new mashup capability in 9.3 maybe MyHouse could be built with ArcGIS Server. Until then I must say building MyHouse would be easier with other tools.

On the philosophical side, by selfishly collecting data for MyHouse I would indirectly benefit my neighborhood (my herd).

“All of us have a place in history. Mine is clouds.” -Richard Brautigan

Brautigan was probably the first geospatial Software as a Service (SaaS) evangelist. We need someone alive today who can assume his role. The next big wave is cloud computing. We in the geocommunity are lagging even though it may very well offer more to our industry than any other.

Here is a laundry list of what I’ve seen in my explorations:

Google
Has MapReduce, BigTable and Sawzall. I don’t see any indication they will try to use their searchengines to perform geoprocessing.

Yahoo
Uses Hadoop, an open source implementation of MapReduce, coupled with ZooKeeper for coordination. Not sure how PIG fits into this, but looks like it might do what DryadLinq does.

Microsoft
Is extending Dryad to leverage Linq, in something called DryadLinq looks interesting. Note that Dryad uses SqlServer Data Services (SSDS) on the back end. See the Mix08 presentation. While this looks interesting, it might be a long time. Not sure how Windows Server 2008 HPC fits into this, but worth watching. Ozzie did mention MapReduce in his Mix08 keynote.

Amazon
First kid on the block with AWS, offering S3, EC2 and SimpleDB. See this post showing how Hadoop can be run on EC2 S3.

GISolve
This is the only site I found for spatial parallel processing. Don’t see much recent activity.

HyperTable
This is an open source version of BigTable.

Sun
Sun Redshift is a major grid computing effort. Sun also bought MySql and seem committed, though maybe they don’t see spatial as special.

The recently announced SQL Server Data Services (SSDS) looks exciting.

Passing OGC WKT strings to stored procedures should allow us to do Restful geoprocessing with SSDS, right?

I wish I could make the ESRI DevSummit, but it’s my kids spring break and Palm Springs just isn’t a family destination. So maybe this summer we’ll make it to San Diego - a very family friendly venue. I wonder if this somehow polarizes the demographics of the Dev Summit, resulting in attendees with either no kids or grown kids.

Keeping it Real at GITA
I’m glad to see energy policy being highlighted in the GITA conference keynote. The mayor makes some good points. Transport costs as a determinant for land use is all geography 101 though. With so many geographers around why don’t we hear more about this?

It sometimes seems ironic that thousands of people continue to fly to distant locations to talk about how the internet is going to change everything.

Keeping it Virtual at Life2.0
I wish I had time to attend the Life2.0 summit, being hosted in second life. While virtual conferences sound like the future, I wonder if they can be a forum that overcomes groupthink. On line I can subscribe to ideas that fit all my pre-conceived notions, and filter out people that annoy me. At a keynote, however I’m forced to listen to ideas that I may find disturbing. So maybe meatspace is better.

Mixing it Up at Mix08
Microsoft clearly attempted to overcome groupthink by arranging Guy Kawasaki’s interview of Steve Ballmer at Mix08. Maybe they also realized watching Ballmer’s atavar do a monkey dance in Second Life wouldn’t be so exciting.

Keeping the Faith at SxSWi
By contrast SxSWi made no such attempts. I wasn’t there, but it seems like attendees were too drunk on koolaid to tolerate someone asking Twitter founder some tough questions. Read about the SxSWi stage interview of Twitter founder by BusinessWeek’s Sarah Lacy. Given that Zuckerberg is the youngest person ever named to Forbes billionaire list, I wonder if the young crowd viewed Lacy as a traitor to the movement. Ageism matters - especially with regard to Web2.0 adoption.

In my view this shows Microsoft is more committed to open discussion of differing viewpoints than the Twitter kids at SxSWi.

Now if only James Fee could do an on stage interview with Jack Dangermond …

Ray Ozzie doesn’t mention Windows Live Core by name in his Mix08 keynote video, but he seems to suggest they are moving forward. You may recall that this is the project staffed by some heavyweights that will provide a response to EC2.

Maybe Microsoft could farm out a job via Mechanical Turk to transcribe Ray’s presentation into searchable text, so I don’t have to summarize this for you:

Minute 12 is some good stuff - he mentions Grid computing and MapReduce.
Minute 20 - utility computing in the cloud.
Minute 25 - Scott Guthrie
27:30 IE8 (stopped watching here).

A lot more presentations are posted here, some look interesting.

Remember, Microsoft is spending a lot of money on datacenters. They’ll soon complete construction on the $1B datacenter here in San Antonio - which is only one of many. Presumably these datacenters will house Windows Live Core, providing a .NET alternative to EC2. I hope Microsoft has taken a look at the experience offered by AppLogic, and is planning something equivalent.

I didn’t see much discussion on any new Microsoft technology that helps computers on a cluster talk to each other. Given the importance of Linq, it sure seems like something like InterLinq is needed.

Using Visual Studio, I’d like to build and debug remote executables that reside in a datacenter.

In my mind this is what GIS 3.0 is all about.

Most of the Linq examples I see are oriented towards SQL. Here’s something really different though -
Luke Hoban has an example of using Linq to do ray tracing. I consider Ray tracing as a special case of raster processing. It would be interesting to simplify this to show a more conventional use case for raster geoprocessing. An important part of geoprocessing though, is the ability to save expressions (or scripts). Notice that the Expression tree is both code and data. Since it is data, I should be able to serialize it, right? While Expression(TDelegate) has a ToString() method, I don’t see a constructor that accepts a string. This thread is the closest thing I found that suggests deserialization is possible.

In the ray tracing example, this means a client would build a raytracing expression then send it to a server (that has lots of cpu (or gpu?)) for execution.

Or thinking in terms of arcmap, - you can save the calculator code in a .cal file. What if there were a raster calculator that saved expression trees in a similar fashion?

Update: Looks like InterLINQ might be the ticket.