1. Lost in the Clouds
An example of a schemaless database
application in the cloud
SlipstreamUSA :: March 25, 2009

2. The Project
OpenStreetMap

A free map of the planet

Free geospatial data

Licensed under CC-BY-SA

Created from

Crowd-sourced contributions

90,000 registered users

1,000+ contributors per day

Used for:

Wikipedia

Flikr

OpenCycleMap.org

OpenPisteMap.org

YourNavigation.org

Mobile devices

© Constantin Litvak

3. Coverage
CC-BY-SA 2.0 © OpenStreetMap

4. The OpenStreetMap Database Server
In the cloud

MySQL

Ways: 24,956,171

Nodes: 305,956,583

Tags: 3.6 billion

highway=motorway

name=Broadway

foo=bar

CC-BY-SA 2.0 Image © Ojw Data © OpenStreetMap

5. Interfaces
API

For updates

(1,000 users per day)
Queries of up to

0.25² degrees
Weekly Planet Dump

100Gbyte XML file

Changes

Feed of changes

Daily / Hourly / Minutely

One file per minute

(~30Kb compressed)
CC-BY-SA 2.0 Image © Ojw Data © OpenStreetMap

6. Problem Statement
Problem

Want to be able to query
the data in bigger chunks
and to do it more
selectively
Solution

Cloud based schemaless
database with simple to
use query interface and
lots of indexes
CC-BY-SA 2.0 Image © Maning Sambale Data © OpenStreetMap

7. XAPI Service
GT.M or Cache

250Gbyte database

Nodes: 30Gb

Node tags: 115Gb

Node indexes: 70Gb

Ways: 21Gb

Way tags: 10Gb

Way indexes: 18Gb

Apache + serverLink

Licensed as AGPL

CC-BY-SA 2.0 Image © Maning Sambale Data © OpenStreetMap

8. XAPI Service
In the cloud

Queries of up to 100² degrees

Query by tags and tag values

Scaleable

REST style interface

http://xapi.openstreetmap.org/api/0.6/*
[tourism=hotel]
[bbox=-75.7,40.0,-75.4,40.2]
Import planet dump

Import minute diff

CC-BY-SA 2.0 Image © Ojw Data © OpenStreetMap

9. XAPI Service
Geospatial Index – quadstrings

adaabcdcabaadab

10. XAPI Service
<way id='27016525'>
<nd ref='296138118'/>
<nd ref='296138119'/>
^way(27016525)=quot;adaabcdcabaadabquot; <nd ref='296138120'/>
^way(27016525,1)=296138118 <nd ref='296138121'/>
^way(27016525,2)=296138119 <nd ref='296138118'/>
<tag k='addr:housenumber' v='2'/>
^way(27016525,3)=296138120
<tag k='building' v='yes'/>
^way(27016525,4)=296138121
</way>
^way(27016525,5)=296138118
^waytag(27016525,quot;addr:housenumberquot;)=2
^waytag(27016525,quot;buildingquot;)=quot;yesquot;
^wayx(quot;buildingquot;,quot;*quot;,quot;adaabcdcabaadabquot;,27016525)=quot;quot;
^wayx(quot;buildingquot;,quot;*quot;,quot;adaabcdcabaadabquot;,27028298)=quot;quot;
^wayx(quot;buildingquot;,quot;*quot;,quot;adaabcdcabaadabquot;,27028299)=quot;quot;
^wayx(quot;buildingquot;,quot;*quot;,quot;adaabcdcabaadabquot;,27028326)=quot;quot;
^wayx(quot;buildingquot;,quot;*quot;,quot;adaabcdcabaadabquot;,27028327)=quot;quot;
^wayx(quot;buildingquot;,quot;*quot;,quot;adaabcdcabaadabquot;,27035972)=quot;quot;
^wayx(quot;buildingquot;,quot;*quot;,quot;adaabcdcabaadabquot;,27035973)=quot;quot;
^wayx(quot;buildingquot;,quot;*quot;,quot;adaabcdcabaadabquot;,27035974)=quot;quot;
^wayx(quot;buildingquot;,quot;*quot;,quot;adaabcdcabaadabquot;,27035975)=quot;quot;
^wayx(quot;buildingquot;,quot;*quot;,quot;adaabcdcabaadabquot;,27035984)=quot;“

11. Deployment
Currently 5 instances:

London

Paris

San Diego

Zurich

Amsterdam

(Wikimedia-de)
More servers needed:

Internet connectivity

300Gb disk storage

CC-BY-SA 2.0 Image © Nikolaj Data © OpenStreetMap

12. Wikipedia: Query to Map
{{osm-query |way=name=Arizona Canal}}
CC-BY-SA 2.0 © OpenStreetMap

13. Internals
Debian

Apache web-server

REST

URI represents a

resource
Xpath flavoring

serverLink

WebLink emulator

GT.M or Cache

POM

AGPL


14. XAPI Future
Load Sharing / balancing

More Xpath-like queries

JSON output

Generic indexing service

More servers


15. Thank you
Thank you
Questions?
CC-BY-SA 2.0 Image © ITO! Data © OpenStreetMap