The Memory Leak

An iterative waterfall?

While Agile is on my mind, I thought I’d write a bit about Agile Geodatabase design.

Let’s say I follow ESRI’s steps from the bottom of this page to create a geodatabase.

1. With Microsoft Visio or Rational Software Corporation’s Rational Rose, design a geodatabase in UML and export it to an XML Metadata Interchange (XMI) file or Microsoft Repository. To learn how, see http://support.esri.com/geodatabase/uml.

2. Add the Schema wizard to ArcCatalog.

3. Generate a geodatabase schema from the XMI file or Microsoft Repository with the Schema wizard.

4. Once you have generated the schema, you can modify it with tools in ArcCatalog if needed.

5. Once the schema is ready, you can load data into it.

In step 4 let’s say I decide to modify the schema. At that point my schema is out of synch with my CASE model. It can quickly become a pain keeping the model in synch with the geodatabase. Then there is also the pain of keeping Data Access Layer Components (DALCs) in synch with the geodatabase. In essence, these steps represent a waterfall.

So here’s my suggestion: ESRI should provide a Geodatabase Designer within Visual Studio. It would provide a look and feel similar to the VS Class Diagrammer, but it would not replace the Class Designer. Instead, it would provide a graphical way of editing an XMI schema containing geodatabase types. A command would allow it to generate code (.NET classes) the way Dave describes here. Likewise, there would also be a command to synchronize the XMI schema with a geodatabase, as well as with the DALC’s.

I know this is all rather vague, but my point is until we have easy-to-use tools that support the round trips needed in iterative development we’ll end up with waterfall processes. Escher notwithstanding, waterfalls are very un-agile.

Update: Here’s an example of how a designer can be built for Visual Studio.

With growing interest in Agile GIS development, this might be a good time to suggest how ESRI could make Agile easier. Since the Agile Manifesto demands frequent deliveries of working software, ease of deployment becomes more important. Below is a suggestion to make ArcGIS desktop more Agile by making it more like ArcGIS Explorer.

I really like how ArcGIS Explorer custom tasks can be installed by end users without Admin privs. Details are here. This capability is Agile friendly. With Desktop however, I don’t see an easy way to build installation packages that don’t require Admin privs for installation. Since many sites I work with require someone from the IT department (with Admin privs) to come install software updates, things can really slow down, especially around the holidays.

While it would be possible to write an ArcMap extension that discovers and loads extensions from .NET assemblies in a manner similar to ArcGIS Explorer, loading commandbars is problematic. Last time I checked, accessing IDocument.Commandbars inside of IExtension.Startup crashes arcmap. Extensions really need to all be loaded before commandbars since the commands on the commandbars often expect extensions to be loaded. So while I could have an extension that loads other extensions via Reflection, Activator.CreateInstance, and IExtensionManagerAdmin.AddExtension, I don’t see a way to do this for command bars.

The crux of the problem seems to be exclusive reliance on COM Categories for discovery of customized components. It would be great if ArcMap would also scan special file folders at startup. These folders would be named similar to the COM Categories, e.g. ESRI Mx Commandbars. Arcmap would then use Reflection to instantiate these objects at startup. I guess we might need some metadata telling it which classes I want registered, or maybe it would be OK if there is no metadata just assume that if a class implements IToolBardef, then it should get loaded.

How do we prevent someone from replacing our assemblies with malicious ones? I’m not clear if/how ArcGIS Explorer accomplishes this, but we should probably think about it. I guess strong naming could be used.

More discussion over at High Earth Orbit on neogeography definition.

While I’m sure many are tired of seeing this dead horse beaten, I do find value in discussing a use case often addressed by neogeography: crowdsourcing. As High Earth points out, the neo and paleo geographers would both be actors.

The problem is some of the tools needed to support crowdsourcing are not getting high enough priority by ESRI.

Case in point: ArcGIS Server’s GraphicsLayer.WriteToXml method would make crowdsourcing a lot easier. A Neogeographer draws graphics on the map, adds some attributes and saves it. Behind the scenes it gets saved to disk (via WriteXml, not arcsde via versioning). A Paleogeographer opens ArcEditor, retrieves graphicslayer to map, converts graphics to features, edits it and commits it to the geodatabase.

The only problem with this is a bug in WriteToXml. It was logged in August (NIM011262), but the SP4 doc doesn’t mention it as being fixed.

The slow resolution of this issue might give neogeographers the impression that ESRI doesn’t place high enough priority on crowdsourcing. The ArcGIS architecture needs to support crowdsourcing.

Pretend to Be An Architect
Speaking of architecture, have you ever noticed how so many architects live long and remain creative in their later years? Take a look at Johnson, Wright, and Venturi.

Contrast this with mathematicians, who seem to die too soon, e.g. Boole, Hamilton and Turing.

I think ageism lurks beneath the surface of the paleo/neo discussion. The GIS community is getting gray. A lot of fresh college grads focus on web design instead of cartography. If we can set an example by aging more gracefully maybe they’d be more interested in trying a few old school concepts. Perhaps the key to aging gracefully is to become more like architects and less like mathematicians.

The swiss are known for clocks and pocket knives, so why didn’t they include a watch on this pocket knife?

If you’ve worked much with GIS there’s a good chance you’ve had to go through the why-spatial-is-special routine with a DBA wanting to store geometry as numeric columns within normalized tables.

But what about time?

Say you’re using GPS clock to compute location via time difference of arrival (TDOA). Nanosecond precision is needed (speed of light = 1 foot per nanosecond), however, SQL Server doesn’t support anything finer than 3.33 microseconds. This could be overcome by introducing a time column with an ITemporalReference. Internally it would store time as a 64bit integer along with a domain and scale - just like with spatial types. ITime is to IGeometry what ITemporalReference is to ISpatialReference. A simpler (though perhaps more confusing) alternative might be to overload the M (measure) value of geometry to allow time to be stored as a measure.

On the other end of the scale is geologic time, which falls outside the .NET DateTime structure limits. In this case the domain would be much larger.

From the helpdoc:

What is the best way for storing temporal data - a netCDF file or a relational database? Which one is faster?

Storing temporal data in relational database is just as viable as using a netCDF file. ESRI’s support of netCDF is primarily to support the existing community of netCDF data and users, not to force people to learn about a new file format. The decision should be made based on how you want to create and manage data in your organization.

It looks like netCDF addresses this issue. But what if I don’t want to represent time using netCDF or date columns as ESRI suggests?

ESRI has developed some useful hydrography tools for Spatial Analyst. Maybe I’m missing something, but I don’t see any tools for cleaning up gaps and spikes in flow (time series) data. Water flow telemetry can contain a lot of subtle noise that should be cleaned before used in models.

I’m writing some tools to clean up time series data. It seems like some of the concepts of the 2 dimensional kernel used by spatial analyst could be generalized (and simplified) to work with one dimensional (time series) data. This would allow me to resample data collected at odd intervals into an evenly spaced sampling scheme. Lots of the operators available for rasters seem applicable to time series: resampling, overlays, convolution.

There appear to be a lot of tools out there written for stock market analysis, but they don’t seem concerned with cleaning up errors.

From EnchantedLearning.

Peter Batty is looking into EC2 for his new venture:

… thinking seriously about using Amazon EC2 and S3 when we roll out, especially now that Amazon has added new “extra large” servers with 15 GB of memory, 8 “EC2 Compute Units” (4 virtual cores with 2 EC2 Compute Units each), and 1690 GB of instance storage, based on a 64-bit platform - these servers should work well for serious database processing.

Amazon has details on the new instance types Peter refers to here.

With such large amounts of memory available, it seems possible to build some really killer route finding services.

Microsoft is working on something similar to EC2. I just hope ESRI provides 64-bit, and a license policy that allows cloud deployment when Microsoft comes up with something.

In response to EC2 questions, Microsoft CTO Ray Ozzie said:

Amazon Web Services [are] … showing Web 2.0 startups that there might actually be something there with regard to this utility computing model. Whether it’s the right set of services exactly, or whether the way that they’ve designed them is exactly what matches the needs of those potential developers, there are some questions. But I think they’ve done the industry a service by beginning to open people’s [in other words, Microsoft's] eyes to the potential.

I don’t have any announcements at this point in time. But directionally, I think you could see in my presentation that we believe very heavily in this utility computing fabric concept; it’s the only way, even internally focused, it’s the only way we can get scale amongst all the properties we run internally. And I think it just makes sense to offer those services to developers and to enterprise customers over time.

Sounds like the same business case Bezos made for AWS.

It’s not so much that (Amazon Web Services) has something to do with selling books. It’s the inverse: Selling books has a lot to do with this.

I really like the ArcGIS Visual Studio Integration Framework for creating new projects & classes. It sure would be nice if the tedious process of creating an installer project could be accomplished through a wizard.

SpatialDB Advisor has a good article about the confusion surrounding Oracle Spatial licensing. Given that spatial capabilities in SqlServer 2008 will be standard issue, Oracle might decide to change their licensing.

Speaking of licensing, Microsoft Windows Server 2008 will include virtualization. They are also building a $500M datacenter here in San Antonio. I’m hoping this means they will provide something similar to Amazon EC2 for .NET. If this is the case the biggest unknown will be whether ESRI offers a licensing model for 9.3 that lets me run ArcObjects in the cloud.

Image stolen from Jeremiah, who discusses more about cloud computing here.

The new ESRI Streets and Data 2007 Update now includes Streetmap Pro. It’s free, but you need to request it.

I’m trying to use StreetMap Pro data with a tracking application. The labeling is a bottleneck, to see what I’m talking about do this:

Open up the StreetmapPro.mxd that ships with ESRI Streets and Data 2007 Update. Then open up the task manager on CPU history. Pan around the map at 1:500K in an area with lots of labels. Try it with labels turned off, then with labels turned on. Notice how much extra cpu is burned when the labels are turned on.

There must be ways ESRI could speed this up. It should be possible to leverage the GPU to do this. Figuring out how to place labels so they don’t overlap is tricky, but seems parallelizable.

I wonder if Google Earth will be using Peakstream to do this. This webinar looks exciting, until you realize that shortly after it was released, Google bought Peakstream and locked them in a closet. Now thats evil.

In a previous post I incorrectly implied there is no textelement for elementgraphicslayers.

Now I see that a graphicelement may be assigned a TextMarkerSymbol, whose text may be assigned a string.

Now, if only Map.RefreshResource would let me pass an envelope so the whole layer wouldn’t flash (similar to IActiveView.PartialRefresh) my app would be safe for epileptics.