Arc2Earth Cloud Layers

Brian Flood has an update on his Arc2Earth Cloud Services:

You can add “cloud layers” directly to your local map and then use the native editing tools in ArcMap to make changes. Every resource in your cloud instance is controlled by login and ACL lists so you can create groups users who all work remotely on the live data. There are also bulk upload/download tools so you can get a fresh copy of any layer anytime you need to perform heavy lifting GIS analysis.

CloudSourcing
This sounds really interesting. It seems like with this sort of infrastructure in place, a confederacy of arcgis machines could be harnessed where each machine works on part of a very large geoprocessing job, similar to Seti@home. I’m not sure if ESRI lawyers would allow it, but what if people rented out their machines during idle times?

Right now for example, I’ve got a geoprocessing job that takes about 30 hours to complete. It would be helpful to have a way to push the job onto the cloud where it would be split up into parts to be solved by different machines. As each part is processed, a master process would append each part together and notify me upon completion. It sounds like Brian’s quad-trees full of r-trees would efficiently support this sort of divide and conquer strategy. Each machine in the confederacy that performed work would then send me a bill for their services. Of course once geoprocessing itself moves to the cloud, this won’t be needed.

Advertisements

3 comments so far

  1. Brian Flood on

    hey Kirk

    very interesting idea! I think the partitioning of data could be handled very easily but the devil is obviously in the details.

    1) client side code that hosts ArcObjects and polls for work
    2) server side queue for creating and posting work chunks
    3) server side code to re-combine and de-dupe the results after all clients have finished (this would be the hardest part, thankfully GAE will support CRON jobs in the future which will help)
    4) I have no idea if ESRI would allow this type of use of client side software

    there’s a similar javascript based project here:
    http://joose-js.blogspot.com/2008/12/massively-parallel-crowd-sourced.html

    cheers
    brian

  2. Kirk Kuykendall on

    Hi Brian –

    Thanks for the info … seems like this approach could be used for distributed tile generation too.

    In cases where people editing geodata in cloud stores need to synch S3 tiles, authorized editors would also agree to allowing their machines used for distributed tile generation jobs.

    Something like a statewide cloudlayer of parcels could be maintained, and kept in synch with S3 tiles.

    This is starting to look like some very disruptive technology.

    Kirk

  3. Brian Flood on

    hey kirk

    tile generation – agreed! but I’m not sure what ESRI would think about using ArcMap in that manner (need to think about that workflow some more though…)

    right now, when edits occur for any cloud layer, a tile gen work item is posted to the main work queue. Workers running on EC2 poll for them and then create tiles and re-post back to GAE. (we keep a tileset up to date for each layer, although there is a lag time between edit and tile regen). there is a limited number of tile levels we create though, all other raster tile requests have to be drawn on demand (slow but cached afterwards)

    I really didn’t want users to have to worry about tiles at all, its the type of thing that should just be baked into the system. it’s not perfect now by any means but that’s the direction I’m going in for tile sets

    cheers
    brian


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: