.NET Successors to VBA IDE and DOCELL are Needed

ArcGIS 10.0 (formerly known as 9.4) will be the last release to support VBA. Expect to be told to simply download Visual Studio Express for free, and create assemblies containing extensions, commands etc. that can be installed without touching the registry.

Still, I find it nice to have code that travels with the mxd. Since compilers are part of the .NET framework, it should be possible for ESRI to offer an IDE that compiles code stored in the mxd and runs it within the ArcMap (or ArcCatalog) process.

But Why Stop There?
It should also be possible to to generate MSIL for languages other than just C# and VB.NET. What would those languages look like? I suggest ESRI should provide small tightly focused languages that address specific problems – domain specific languages (DSLs) in other words. The System.Reflection.Emit or Expression Trees available in .NET could be leveraged to do this.

Take DOCELL, for example. This was a language introduced in ArcGrid (at 6.0.1?), and is nearing the end of its supported life. When ArcGIS was introduced, ESRI suggested that DOCELL scripts be replaced with ArcObjects code. A few lines of DOCELL code that could be interactively tested now require many lines of .NET code that must be compiled. A DSL would be an appropriate replacement for DOCELL.

In addition to DOCELL, DSL’s that operate on map topologies would be interesting. A lot of topology tasks involve trees. XPath is a good example of a DSL that finds nodes in trees based on patters. Suppose I have a water distribution network, and I’d like to find all valves on 6 inch pipes down stream from pumps with a particular capacity. This really isn’t much different from an XPath query. If there were an XPath equivalent for working with trees grown from map topologies, writing searches like this would be much easier. The details of recursion could be hidden.

Lex/Yacc for GeoCompilers
Maybe something similar to lex/yacc could be provided. A lex-like program would scan a geometric network (or topology) to produce tokens that a yacc-like program could use to create compilers. These compilers, in turn, would be used to compile programs written in a language specific to the geodatabase. Take the NHD geodatabase for example. Writing ArcObjects programs against this data model is a bit tricky for most ecologists whose primary skill is ecology – and not programming. Imagine a having different programming languages available specific to the NHD. This would make the ecologists’ job would be much easier. Compilers would live within the geodatabase where they would be updated when schema changes occur.

Web2.0 Routing Analysis

In a recent interview with Government Computer News, Jack Dangermond describes the differences between Google and ESRI. Google is focused on visualization, while ESRI is focused more on spatial analysis services with authoritative data.

In my opinion, Google’s iPhone routing application (described here by Peter Batty) constitutes spatial analysis for the masses – or at least for the masses interested in transit.

I think ESRI could compete with Google in this arena by providing a service where their business partners could develop Network Solvers to complement the one offered by ArcGIS Online. The architecture for integrating these solvers would be similar to that used by Network Analyst.

Instead of relying on advertising revenue, ESRI could offer something similar to Amazon Web Service’s cost calculator, but extended so that business partners could set a price for usage of their solver. This price would be added by ESRI to the ArcGIS Online bill sent to those who use the service, in addition to charges for cpu usage, geodatabase i/o etc., on a pay-as-you-go basis.

Put the Author back in Authoritative
Consider, for example a travelling salesman (TSP) solver. While there are some TSP algorithms that work on the client side, I suspect many might be too chatty and are best implemented behind the firewall. A business partner could develop a solver, expose it as a service, then sell a routing application at Salesforce.com’s AppExchange that consumes it.

If ESRI could also web-enable model builder (as I’ve describe here) the solver could also be exposed as a geoprocessing tool.

Sorting through The Big Sort

Maybe I spent too many years living in the same county as the authors, but I agree with everything Bill Bishop and Robert Cushing have written in their book The Big Sort: Why the Clustering of Like-Minded America Is Tearing Us Apart.

My only problem with this book is that it is, well, a book. While the narrative is quite readable, most of the trends probably aren’t that surprising to most readers. What would likely surprise many readers though, is how such small changes in human behavior can cause such large changes in the cultural landscape. This is difficult to communicate in a text narrative. Models are better suited for this.

There is a Big Sort website, and it even has some maps. But the website is primarily geared towards selling the book. The authors have obviously collected a lot of data regarding cultural changes over the past 50 years, why not publish that too? That way, animated maps showing historic trends, plus agent based models could be developed.

The models would let readers users make small changes to agent behaviors and watch how those changes impact settlement patterns through time.

Agent based modeling has been applied to Nobel laureate Thomas Schelling’s game theory of how racial segregation occurs.

Allen Downing has written a free online computational modeling textbook, where he points out that Schelling’s model implies that “if you observe segregation in a real city, you cannot conclude that the people in the city are racists”.

Something that was new to me in The Big Sort was the research describing how the views held by a group of people become more extreme through time – more extreme than the average of the individuals in the group. This suggests that, for example, while people may not be motivated by racism to move, living in a segregated neighborhoods might slowly cause them to become racist. This too could be modeled, by having degrees of red or blue, based on how long an agent has been surrounded by similar colors.

Downing includes this exercise:

The mechanism Bishop hypothesizes is not that people, like the agents in Schelling’s model, are more likely to move if they are isolated, but that when they move for any reason, they are likely to choose a neighborhood with people like themselves. Modify your implementation of Schelling’s model to simulate this kind of behavior and see if it yields similar degrees of segregation.

If Bishop & Cushing would publish their data on their website, students working on this exercise would be able to publish models demonstrating the processes described in the book.

Who knows, maybe even the real estate listing sites might start including a scaled index to guide buyers who are consciously searching for a diversified neighborhood.