The Memory Leak

It’s helpful that ESRI has provided data models for so many industries. Some of the data models, such as ArcHydro, also include tools that operate on the data.

But from what I can tell, there are no classes provided that address the so-called “object-relational impedance mismatch”.

This excerpt from Scott Ambler’s book describes the problem pretty well.

What I think we need are some baseclasses that do simple object-relational mapping. These baseclasses would be generated from a data model designer that would run within Visual Studio. In addition to generating xml schemas from which would could create geodatabases, the designer would also generate .NET code (C# or VB.NET) which we could then inherit from and override/extend to meet particular needs. Or maybe partial classes is the way to go, as Dave Bouwman describes here. Either way, we really need some way to create a library of .NET classes that work with each of these data models.

While COM will likely be around for a lot longer, I don’t see why we can’t use .NET classes to hide much of it. Implementation inheritance can be good - let’s allow programmers to leverage use it. Maybe the goals of custom features could be addressed by this approach without requiring C++ ATL.

For example, instead of using a featurecursor to loop through a StreamGauge featureclass, I’d like to simply get a System.Generic.Collection collection of StreamGauges with getters/setters mapped to each field of the feature.

I like working on problems where I can get into a state psychologists refer to as flow. An important characteristic of being in Flow is a distorted sense of time. I like to think of flow as looking at a problem in the frequency domain. Just focusing on the here and now, listening to the rhythm of different potential solutions bouncing off each other. This is quite different from life in the time domain, where deadlines must be met.

Time vs Frequency
Solving a problem can be thought of as a journey. The journey can be viewed in two different ways - by time or by frequency. The time domain is where deadlines are set, and progress measured. This is where project managers live. The frequency domain is where work is accomplished. The steady rhythm of a each stroke of the canoe paddle, bang of the hammer, or debug/compile/build heartbeat sets the pace. This is where programmers live.

Differential Equations
The solution of complex math problems can follow a similar journey. A Laplace transform can be employed to move between time and frequency domains. Once in the frequency domain, the solution can be obvious. There’s a mathematical notion of flow too, but ironically, it is strongly rooted in the time domain.

Agile Methods
Waterfall project management seems to be more time oriented, whereas agile methods are more suited to the frequency domain. What is missing is something equivalent to a Laplace transform for project managers that allow them to move easily between time and frequency domains. It seems like this is the same design challenge faced by musicians. I wonder if composers of music could teach us ways to reconcile waterfall/agile forces.

Jazz Music
In jazz music individual band members take turns soloing while the rest play a predictable rhythm. In differential equations this is like holding all variables constant and allowing one to vary. I wonder if this suggests an approach for Agile where we might hold hold all components constant within a sprint, except for one. A pair of programmers (or more?) would push the envelope of a solo component as far as possible. Making big changes to more than one component within a sprint gets pretty complicated.

Jazz doesn’t seem to fit well with a big design up front approach. Instead, it relies on faith that if we can just open ourselves up spirtually to what is going on we will arrive at our destination on time. Nevertheless jazz composers must live in the world of time too. Record labels want repeatability. While I can’t picture a jazz band getting ISO 9001 certification, they must go through some sort of process. Like Mussolini, waterfall isn’t a fun project management method to work under, but sometimes it’s needed to get the trains to run on time. Can we have it both ways? Can we Take the A Train and be on schedule?

I find evolution to be a fascinating design process, yet sometimes I wonder if evolution can ever be Agile. The key thing about evolution is that time is plentiful. Agile time is broken up into short sprints.

Perhaps the sprint can be thought of as punctuated equilibrium event.

Evolutionary design is not without its faults - I’m typing this on a QWERTY keyboard. But does Agile avoid QWERTY decisions? It seems like the Agile tenet of delivering working software early can easily result in QWERTY inertia. If an early iteration release delivers great enough value, users run the risk of using the tools so much that that they become ingrained with the UI (or framework).

What is to keep an arbitrary design decision in an early iteration from freezing into place and lasting for many generations?

An iterative waterfall?

While Agile is on my mind, I thought I’d write a bit about Agile Geodatabase design.

Let’s say I follow ESRI’s steps from the bottom of this page to create a geodatabase.

1. With Microsoft Visio or Rational Software Corporation’s Rational Rose, design a geodatabase in UML and export it to an XML Metadata Interchange (XMI) file or Microsoft Repository. To learn how, see http://support.esri.com/geodatabase/uml.

2. Add the Schema wizard to ArcCatalog.

3. Generate a geodatabase schema from the XMI file or Microsoft Repository with the Schema wizard.

4. Once you have generated the schema, you can modify it with tools in ArcCatalog if needed.

5. Once the schema is ready, you can load data into it.

In step 4 let’s say I decide to modify the schema. At that point my schema is out of synch with my CASE model. It can quickly become a pain keeping the model in synch with the geodatabase. Then there is also the pain of keeping Data Access Layer Components (DALCs) in synch with the geodatabase. In essence, these steps represent a waterfall.

So here’s my suggestion: ESRI should provide a Geodatabase Designer within Visual Studio. It would provide a look and feel similar to the VS Class Diagrammer, but it would not replace the Class Designer. Instead, it would provide a graphical way of editing an XMI schema containing geodatabase types. A command would allow it to generate code (.NET classes) the way Dave describes here. Likewise, there would also be a command to synchronize the XMI schema with a geodatabase, as well as with the DALC’s.

I know this is all rather vague, but my point is until we have easy-to-use tools that support the round trips needed in iterative development we’ll end up with waterfall processes. Escher notwithstanding, waterfalls are very un-agile.

Update: Here’s an example of how a designer can be built for Visual Studio.

With growing interest in Agile GIS development, this might be a good time to suggest how ESRI could make Agile easier. Since the Agile Manifesto demands frequent deliveries of working software, ease of deployment becomes more important. Below is a suggestion to make ArcGIS desktop more Agile by making it more like ArcGIS Explorer.

I really like how ArcGIS Explorer custom tasks can be installed by end users without Admin privs. Details are here. This capability is Agile friendly. With Desktop however, I don’t see an easy way to build installation packages that don’t require Admin privs for installation. Since many sites I work with require someone from the IT department (with Admin privs) to come install software updates, things can really slow down, especially around the holidays.

While it would be possible to write an ArcMap extension that discovers and loads extensions from .NET assemblies in a manner similar to ArcGIS Explorer, loading commandbars is problematic. Last time I checked, accessing IDocument.Commandbars inside of IExtension.Startup crashes arcmap. Extensions really need to all be loaded before commandbars since the commands on the commandbars often expect extensions to be loaded. So while I could have an extension that loads other extensions via Reflection, Activator.CreateInstance, and IExtensionManagerAdmin.AddExtension, I don’t see a way to do this for command bars.

The crux of the problem seems to be exclusive reliance on COM Categories for discovery of customized components. It would be great if ArcMap would also scan special file folders at startup. These folders would be named similar to the COM Categories, e.g. ESRI Mx Commandbars. Arcmap would then use Reflection to instantiate these objects at startup. I guess we might need some metadata telling it which classes I want registered, or maybe it would be OK if there is no metadata just assume that if a class implements IToolBardef, then it should get loaded.

How do we prevent someone from replacing our assemblies with malicious ones? I’m not clear if/how ArcGIS Explorer accomplishes this, but we should probably think about it. I guess strong naming could be used.