The Memory Leak

The swiss are known for clocks and pocket knives, so why didn’t they include a watch on this pocket knife?

If you’ve worked much with GIS there’s a good chance you’ve had to go through the why-spatial-is-special routine with a DBA wanting to store geometry as numeric columns within normalized tables.

But what about time?

Say you’re using GPS clock to compute location via time difference of arrival (TDOA). Nanosecond precision is needed (speed of light = 1 foot per nanosecond), however, SQL Server doesn’t support anything finer than 3.33 microseconds. This could be overcome by introducing a time column with an ITemporalReference. Internally it would store time as a 64bit integer along with a domain and scale - just like with spatial types. ITime is to IGeometry what ITemporalReference is to ISpatialReference. A simpler (though perhaps more confusing) alternative might be to overload the M (measure) value of geometry to allow time to be stored as a measure.

On the other end of the scale is geologic time, which falls outside the .NET DateTime structure limits. In this case the domain would be much larger.

From the helpdoc:

What is the best way for storing temporal data - a netCDF file or a relational database? Which one is faster?

Storing temporal data in relational database is just as viable as using a netCDF file. ESRI’s support of netCDF is primarily to support the existing community of netCDF data and users, not to force people to learn about a new file format. The decision should be made based on how you want to create and manage data in your organization.

It looks like netCDF addresses this issue. But what if I don’t want to represent time using netCDF or date columns as ESRI suggests?

ESRI has developed some useful hydrography tools for Spatial Analyst. Maybe I’m missing something, but I don’t see any tools for cleaning up gaps and spikes in flow (time series) data. Water flow telemetry can contain a lot of subtle noise that should be cleaned before used in models.

I’m writing some tools to clean up time series data. It seems like some of the concepts of the 2 dimensional kernel used by spatial analyst could be generalized (and simplified) to work with one dimensional (time series) data. This would allow me to resample data collected at odd intervals into an evenly spaced sampling scheme. Lots of the operators available for rasters seem applicable to time series: resampling, overlays, convolution.

There appear to be a lot of tools out there written for stock market analysis, but they don’t seem concerned with cleaning up errors.

From EnchantedLearning.

Peter Batty is looking into EC2 for his new venture:

… thinking seriously about using Amazon EC2 and S3 when we roll out, especially now that Amazon has added new “extra large” servers with 15 GB of memory, 8 “EC2 Compute Units” (4 virtual cores with 2 EC2 Compute Units each), and 1690 GB of instance storage, based on a 64-bit platform - these servers should work well for serious database processing.

Amazon has details on the new instance types Peter refers to here.

With such large amounts of memory available, it seems possible to build some really killer route finding services.

Microsoft is working on something similar to EC2. I just hope ESRI provides 64-bit, and a license policy that allows cloud deployment when Microsoft comes up with something.

In response to EC2 questions, Microsoft CTO Ray Ozzie said:

Amazon Web Services [are] … showing Web 2.0 startups that there might actually be something there with regard to this utility computing model. Whether it’s the right set of services exactly, or whether the way that they’ve designed them is exactly what matches the needs of those potential developers, there are some questions. But I think they’ve done the industry a service by beginning to open people’s [in other words, Microsoft's] eyes to the potential.

I don’t have any announcements at this point in time. But directionally, I think you could see in my presentation that we believe very heavily in this utility computing fabric concept; it’s the only way, even internally focused, it’s the only way we can get scale amongst all the properties we run internally. And I think it just makes sense to offer those services to developers and to enterprise customers over time.

Sounds like the same business case Bezos made for AWS.

It’s not so much that (Amazon Web Services) has something to do with selling books. It’s the inverse: Selling books has a lot to do with this.

From Metropolis.

Cnet has an interesting article describing what is needed for successful citywide Wi-Fi deployment.

One of the common threads weaved through each of these [successful] deployments is that all of these cities have committed to using the Wi-Fi networks for their own purposes whether it be to provide remote access for mobile city workers, automate meter reading, control traffic congestion or enhance public safety.

If this is indeed the case, I think (near) real-time GIS mapping applications need more attention. I heard a rumor over a month ago that ESRI has bought Tracking Server from Northrop Grumman. However, I haven’t seen any press releases. I was told there are no plans to include Tracking Server as part of an EDN subscription. I guess this means Tracking Server will stay primarily in the Defense sector. That’s too bad, I think there are a lot of peaceful applications of this technology.

I really like the ArcGIS Visual Studio Integration Framework for creating new projects & classes. It sure would be nice if the tedious process of creating an installer project could be accomplished through a wizard.

If my name weren’t on the ballot, I probably would have forgotten to vote too. I suspect many other homeowners associations (HOAs) also have a hard time getting people involved. The rules of our HOA say at least 10% of the residences must vote for a an election to be valid. Since only %6 bothered to cast a vote, the voting period has been extended.

Googling for homeowners assocation + gis doesn’t turn up much. It is strange how little the internet has done for traditional neighborhood communities when compared to the way it has cultivated so many new ones that transcend geography. I would be very interested in seeing any sort of HOA website that includes interactive maps. A while back I started one here, but haven’t had time to keep it going. Maybe if I get elected I’ll put more into it. With that in mind I’ve started a list of requirements for HOA based GIS.

Identify
Encounters with some neighbors are so infrequent that sometimes it gets hard to remember names. When that happens, I’ve used the Bexar Appraisal District’s web site to find out who owns (and hopefully lives) in a particular house. Not being designed for this purpose, it is somewhat clumsy. What is needed is a very simple focused tool that allows me to click on a parcel map and get a name.

Flood Plain Proximity
Years ago FEMA did a lousy job when they delineated the floodplain extent. I forget all the details, but to avoid paying extra insurance we had to write our congressman, kindly asking him to push paperwork through FEMA to have our property amended. Others in the neighborhood need to do the same thing. This would simply be a mashup of existing FEMA maps with parcel boundaries, along with links indicating what procedures to follow to get amended (not sure if thiis is the correct term).

Crime Maps
San Antonio Police publishes locations of all crimes. These can be mapped. There has also been a lot of interest in graffiti, but I don’t think the police map that.

Crowdsourcability
Maybe that’s not a word, but you know what I mean. I think it would be interesting to be able to mark up a shared map, not just with graffiti incidents, but also garage sales, actual utilities locations, and observations of actual flood extents. Maybe its just a matter of encouraging homeowners to blog and embed Google’s MyMaps (of course only if they’re not using WordPress.com).

WebCams
This might be controversial … but what if web cams were placed at the entrance to each cul-de-sac sub-neighborhood so that motion sensors would trigger cameras to image license plate numbers? There is a very active Cellular On Patrol committee, I wonder if the same volunteers would like to sit and watch web cams.

More on this later.

From Bacteria.

A lot in the news lately about drug-resistant staph infections.

While continuity of operations planning addresses things like fires, tornadoes, and terrorism, I haven’t seen much that explicitly addresses how businesses can prepare for pandemics.

What should the CEO of a corporation do when most of his staff are either too sick - or too afraid of becoming sick - to report to the office?

Perhaps telecommuting will become part of continuity planning. Once these plans go into action many areas of residential land use will effectively become mixed use developments. My office is currently home based. If more people in my neighborhood did the same, maybe we’d have the critical mass to support a Kinko’s. Who knows, maybe even there’d be enough of us to support delivery of something other than pizza for lunch.

First Fridays

Growing up in Bryan, I watched the downtown district gradually decline.

So it is heartening to see Amazon Web Services evangelist blogging about his visit.

I flew to AggieLand - College Station and Bryan to meet with a bunch of designers and developers who had invited us to come and speak to their group. These college grads have started something really innovative - Similar to “incubator” concept, “The Creative Space” is space (yes! physical space) for start-ups. If you have an idea, you are in near Bryan, TX, this is a place to check out. They share all the resources from a Rails Developer to a designer, from artists to painters. Multiple people work on multiple concepts/ideas. Dollar Store across the street for supplies and office space in Bryan Downtown makes it the right place for people with high-energy-and-low-ego who want to get their ideas turn into action.

Just don’t tell me they’re building a Starbucks.

Is Free Data Always a Good Idea?
ESRI seems to promote the idea that software should cost money, but data should be free. At the same time OSGeo folks seem to think both software and data should be free. Is anyone asking whether free data is always a good idea ?

If you’re like a lot of GIS professionals, a big part of your job is adding value to geodata. Free data seems like a good thing at first, but without incentive for data to be collected, opportunities to add value are limited.

Hyperspectral Cube Image stolen from Duke University Math Dept.

Case in Point: Hyperspectral Imagery
In its raw form, hyperspectral imagery (HSI) is a pain to deal with. The files are huge, and difficult to visualize. To be at all useful, it must first be processed. Processing HSI is not trivial - graduate training is needed plus specialized software. So let’s suppose you take out a few college loans, get an advanced degree in remote sensing, and learn to glean information from HSI that can save the planet.

With HSI supporting so many different types of environmental analyses, you would think the world needs more HSI experts. But once you start looking for a job and get only get 4 hits when searching Dice.com for hyperspectral, all of which are military projects, you may begin to think otherwise. There’s a disconnect somewhere. I think the problem is the market, or rather the lack thereof.

HSI at USGS
USGS sells data collected by the Hyperion satellite. After using the rather lame EarthExplorer, you must purchase an image in its entirety ($250 each). I’m not sure how big the files are, but rest assured, they are big … purchase is delivered on DVD, not available via download. Just because you can buy something at there website doesn’t mean its a marketplace though. A market is a place where you can sell stuff, like eBay.

WeoGeo
Take a look at WeoGeo where buyers are able to discover and purchase an HSI with just the bands they need for an area of interest. Perhaps as importantly though, users can also participate as sellers. So after processing the imagery, a user can then log in as a seller and list a derived product for sale, choosing how royalties are to be paid back to the vendor of the parent (raw) data.

I’ve used hyperspectral imagery as an example, but I think the same forces apply to any dataset requiring expertise to process. Sensors are becoming more sophisticated and UAVs more affordable. Technology alone does not an industry make. We need to be building marketplaces for the new types of data these new platforms will be collecting.

SpatialDB Advisor has a good article about the confusion surrounding Oracle Spatial licensing. Given that spatial capabilities in SqlServer 2008 will be standard issue, Oracle might decide to change their licensing.

Speaking of licensing, Microsoft Windows Server 2008 will include virtualization. They are also building a $500M datacenter here in San Antonio. I’m hoping this means they will provide something similar to Amazon EC2 for .NET. If this is the case the biggest unknown will be whether ESRI offers a licensing model for 9.3 that lets me run ArcObjects in the cloud.

Image stolen from Jeremiah, who discusses more about cloud computing here.