3. 367 millions
route requests in 2012
4 millions
referenced business
Mappy, a brand of SoLocal Group
About 10 millions
monthly users
27 millions
monthly visits
Provide route planner and local services on the web and
mobiles devices
4. Feedback on the overhaul of
the mapping engine of Mappy
around Mapnik and PostGIS
7. The path – starting
point
Background
– Map processing for 16 years with TomTom/AND databases
– Maps are used by Mappy & French YellowPages media
– « home-made » processing mastered but ageing
The future
– Repositionnning map to fit new requirements : the purpose of the
map is now to support content
– Open-source-oriented strategy (moving towards)
– Anticipate emerging challenges (vector tiles, HD maps, relief, ...)
8. Geographic data
SDK Mobile
Mappy
API AJAX
Mappy
Stack Mappy
Engine
Templates
Stack Mappy
Cache
Data processing
Mostly pre-computation
Duration : 10 days
Non standard
proprietary format
12 zoom levels
Software system
Fully mastered (behaviour)
Some dead code
Costly evolutions
Non negociable performances
6 Billions req/month
BACK OFFICE PRODUCTION
9. The path - goals
– Create a map design able to support content
– Be ready to release with the new version of our website
fr.mappy.com
– Reduce the computational time
– address weaknesses of the platform and keep in mind its strengths
– Streamlining map tile production
– Standardizing (REST API, etc.)
– Build for the future (Able to scale and support new services)
11. Geographic
data
API AJAX
NewMappy
SDK Mobile
New Mappy
KyotoCabinetKyotoCabinet
New platform
in production
Reduced Preparation Time
2days (vector) + 2days (relief)
Standard
Live code
Mapnik contribution
Reduced operating cost
BACK OFFICE PRODUCTION
13. Our journey - Starting point
– Heterogeneous knowledge of the team
on open source technologies
– Training & getting help from expert
– Bring a culture of open source
– Split project
– Migrate brick by brick in order to learn and master the platform
– « Eat our own food »
– Test it fast
Innophi
14. Geographic
data
SDK Mobile
Mappy
API AJAX
Mappy
Stack Mappy
Engine
Templates
Stack Mappy
Cache
API AJAX
NewMappy
SDK Mobile
New Mappy
KyotoCabinetKyotoCabinet
BACK OFFICE PRODUCTION
BACK OFFICE PRODUCTION
Templates
Step 0
Map design
By marketing and Map designer
15. Geographic
data
SDK Mobile
Mappy
API AJAX
Mappy
Stack Mappy
Engine
Templates
Stack Mappy
Cache
API AJAX
NewMappy
SDK Mobile
New Mappy
KyotoCabinetKyotoCabinet
BACK OFFICE PRODUCTION
BACK OFFICE PRODUCTION
Step 1
Migration of database and map engine
Keep Front interface
Work on database performance
16. Geographic
data
SDK Mobile
Mappy
API AJAX
Mappy
Stack Mappy
Engine
Templates
Stack Mappy
Cache
API AJAX
NewMappy
SDK Mobile
New Mappy
KyotoCabinetKyotoCabinet
BACK OFFICE PRODUCTION
BACK OFFICE PRODUCTION
Step 2
Cache migration
New APIs
Add relief /HD tiles
Work on performance
and configuration (Cache/Service/APIs)
17. Geographic
data
SDK Mobile
Mappy
API AJAX
Mappy
Stack Mappy
Engine
Templates
Stack Mappy
Cache
API AJAX
NewMappy
SDK Mobile
New Mappy
KyotoCabinetKyotoCabinet
BACK OFFICE PRODUCTION
BACK OFFICE PRODUCTION
Step 3
Streamlining databases
Work on scaling
Optimizing process
18. Our journey – key points
– Be able to challenge historical choices
– Reach a better consensus with our customers (Web/Mobile)
– Less time on tools implementation than configuration
– We built prototype in two weeks
– We worked on configuration during 4 months
– Real issue : association of metadata
with map tiles
(dynamic display on map icons)
19. Our journey – technical choices
– PostGIS: transactional database/scalable/standard compliant
– Mapnik : approved by marketing and technical teams!
– Tornado :
– High performance reputation, already used by mobile team
– Varnish :
– Highyield
– Already used
by front teams
– Nothing to
prove
20. Our journey –
Contribution to Mapnik
– Mappy was legitimate to participate in an open source project
– Develop an open source policy at Mappy
– Arrange technical organization (Open Source Manager, Github
website : https://github.com/Mappy)
– Contribute to Mapnik : implementation of parallelism with PostGIS
(available on master). Performance gain: 25%
– Some tools availables on Github like pycnik
22. Feedback– Key figures
– Production platform handles 85% of the load (All media except
Mobile YP)
– 14 servers:
– Varnish : 2 / Tornik : 8 / PostGIS : 4
– Output without cache:
– Peak 240 req/s
– Average throughput 150 req/s
– Output with cache: 2500 req/s (peak)
– Cache Hit Ratios: 94%
– Cache preload: 2 millions objects
– Postgis database: 75 Go
– Relief database: 300 Go
23. Feedback… team
– Positive
– Team motivation, proud and in control of our new platform
– PostGIS and Mapnik tools
– Contribution/open source world
– Negative
– Difficult to identify correct projects splitting
– Contribute lately to Mapnik
– Contributing requires available time (discussion with
community/ pull requests)
– Deployment by our customer not as fast as expected
