SlideShare une entreprise Scribd logo
1  sur  29
Télécharger pour lire hors ligne
CloudantGeo:
Scaling geospatial for the masses
Mike Miller
CoFounder, Chief Scientist
{Problems: [‘Mobile’, ‘Data’]}




            Mobile                                 Big Data

      These break our existing models for computing

                         Mike Miller, 2013/02/27              2
{Introductions: ‘Cloudant’}




          Database that ships with a mobile strategy
                          Mike Miller, 2013/02/27      3
{Introductions: ‘Cloudant’}

• Distributed Database as a Service
  (DBaaS)
• For developers of high-velocity web and
  mobile apps
• Venture Funded, YCombinator
• Ends drudgery of SQL and scale-it-
  yourself NoSQL
• 13,000+ users
• Founded by big data scientists
• Speaks CouchDB API




                                    Mike Miller, 2013/02/27   4
{Introductions: ‘Cloudant’}




                        Mike Miller, 2013/02/27   5
{Schemas: ‘Optional’}




           Schemas & protocols can be restrictive
               and inhibit data integration



                     JSON over HTTP


                         Mike Miller, 2013/02/27    6
{Install: ‘Cloudant’}



                 You do this:


             We give you:


                                That’s It


                            Mike Miller, 2013/02/27   7
{API: ‘REST’}




                Write a doc...from the browser

                  No client install necessary

                      JSON, REST, HTTP
                          Mike Miller, 2013/02/27   8
{API: ‘Search’}

                                                             Create Secondary Indexes




Query Those indexes



                 Rinse and repeat.... billions of times per day
                                   Mike Miller, 2013/02/27                              9
{Challenge: ‘Dogma’}




         “Scaling would be a good problem to have.
                  We’ll deal with that later”


                False: Scaling Mandatory



                        Mike Miller, 2013/02/27      10
{Have: ‘Success’}




                    Mike Miller, 2013/02/27   11
{Minimize: ‘Latency’}




                  Public or Private Cloud
                        Mike Miller, 2013/02/27   12
{Write: ‘Local’, Sync: ‘Later’}




  Embedded,
  Edge, Sattelites


                         Desktop,
                         Browser




                          Mike Miller, 2013/02/27
                                                    Cloud   13
Mike Miller, 2013/02/27   14
{Example: ‘HotheadGames’}




                     Mike Miller, 2013/02/27   15
{Example: ‘HotheadGames’}




              ~100x Scaling in last 9 months


 This Database is bigger than many (most?) Hadoop clusters
                          Mike Miller, 2013/02/27            16
How do you scale a database from MB to PB?

 How do you add new features that scale?




                Mike Miller, 2013/02/27      17
What if my application cares about geospatial location?

What if I need to combine FTI/search, SELECT, and geo?

     Cloudant: MapReduce + Lucene + Geospatial



                      Mike Miller, 2013/02/27             18
{Sharding: ‘Automatic’}

                                                                                                     •   Clustering in a ring (a la Dynamo)
 PUT http://rnewson.cloudant.com/dbname/blah?w=2
                                                                                                     •   Any node can handle a request
                                                                                                     •   O(1) lookup
                            Load Balancer
                                                                                                     •   Quorum system (N, R, W)
                                                                                                     •   Views distributed like
                                                                                                         documents
                                       Node 1

               24                                           No
                                                                                                     •   Distributed Erlang
            de                  A     B     C   D             de
          No                                        B
                                                                 2
                    Z
                        A
                                                        C                                            •   Masterless
           Y                                                   D
      X                 hash(blah) = E                               E


                                                                           C          N
                                                                                       od
                                                                                         e
                                                                                  D          3

                                                                                      E
  N=3                                                                                            F

  W=2
  R=2


                                                                                                         D



                                                                                                                 No
                                                                                                                   de
                                                                                                             E



                                                                                                                      4
                                                                                                                 F
                                                                                                                     G
                                                        Mike Miller, 2013/02/27                                                       19
{Manual_Sharding: ‘Sucks’}




http://static.googleusercontent.com/external_content/untrusted_dlcp/research.google.com/en/us/archive/spanner-osdi2012.pdf



                                                       Mike Miller, 2013/02/27                                               20
{Format: ‘geojson’}




           http://geojson.org/geojson-spec.html
                        Mike Miller, 2013/02/27   21
{Format: ‘topojson’}




         https://github.com/mbostock/topojson/wiki
                         Mike Miller, 2013/02/27     22
{Index: ‘R-Tree’}




                                              Wikpedia




                    Mike Miller, 2013/02/27              23
{Index: ‘R*-Tree’}




                                               Wikpedia




                     Mike Miller, 2013/02/27              24
{Index: ‘TPR*-Tree’}

• Predictive spatio-temporal query retrieves the set of moving objects that will intersect
  a query window during a future time interval




                  http://www.cs.ust.hk/~dimitris/PAPERS/VLDB03-TPR.pdf
                                      Mike Miller, 2013/02/27                           25
{API: ‘geo’}

                                                Create Secondary Indexes




Query Those indexes


                      Mike Miller, 2013/02/27                              26
{Operators: ‘more’ }


• Disjoint                                • Crosses
• Equals                                  • Within
• DWithin                                 • Contains
• Beyond                                  • Overlaps
• Intersect                               • BBX
• Touches




                       Mike Miller, 2013/02/27         27
{Want: ‘Beta Testers’ }




     For a demo or a beta invite, pm me @mlmilleratmit




                          Mike Miller, 2013/02/27        28
{Status: [‘Now’, ‘Future’] }


• Big and Getting Bigger                   • Product Roadmap
   •Databases -- 100M+                         •Security enhancements
   •Daily Transactions -- 10s of               •Advanced geospatial
    Billions                                   •Graph engine
   •Indexed Data -- 100s TBs
   •Map Reduces/day -- Billions




                             Mike Miller, 2013/02/27                    29

Contenu connexe

En vedette

Content Methodology: A Best Practices Report (Webinar)
Content Methodology: A Best Practices Report (Webinar)Content Methodology: A Best Practices Report (Webinar)
Content Methodology: A Best Practices Report (Webinar)contently
 
How to Prepare For a Successful Job Search for 2024
How to Prepare For a Successful Job Search for 2024How to Prepare For a Successful Job Search for 2024
How to Prepare For a Successful Job Search for 2024Albert Qian
 
Social Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie InsightsSocial Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie InsightsKurio // The Social Media Age(ncy)
 
Trends In Paid Search: Navigating The Digital Landscape In 2024
Trends In Paid Search: Navigating The Digital Landscape In 2024Trends In Paid Search: Navigating The Digital Landscape In 2024
Trends In Paid Search: Navigating The Digital Landscape In 2024Search Engine Journal
 
5 Public speaking tips from TED - Visualized summary
5 Public speaking tips from TED - Visualized summary5 Public speaking tips from TED - Visualized summary
5 Public speaking tips from TED - Visualized summarySpeakerHub
 
ChatGPT and the Future of Work - Clark Boyd
ChatGPT and the Future of Work - Clark Boyd ChatGPT and the Future of Work - Clark Boyd
ChatGPT and the Future of Work - Clark Boyd Clark Boyd
 
Getting into the tech field. what next
Getting into the tech field. what next Getting into the tech field. what next
Getting into the tech field. what next Tessa Mero
 
Google's Just Not That Into You: Understanding Core Updates & Search Intent
Google's Just Not That Into You: Understanding Core Updates & Search IntentGoogle's Just Not That Into You: Understanding Core Updates & Search Intent
Google's Just Not That Into You: Understanding Core Updates & Search IntentLily Ray
 
Time Management & Productivity - Best Practices
Time Management & Productivity -  Best PracticesTime Management & Productivity -  Best Practices
Time Management & Productivity - Best PracticesVit Horky
 
The six step guide to practical project management
The six step guide to practical project managementThe six step guide to practical project management
The six step guide to practical project managementMindGenius
 
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...RachelPearson36
 
Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...
Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...
Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...Applitools
 
12 Ways to Increase Your Influence at Work
12 Ways to Increase Your Influence at Work12 Ways to Increase Your Influence at Work
12 Ways to Increase Your Influence at WorkGetSmarter
 
Ride the Storm: Navigating Through Unstable Periods / Katerina Rudko (Belka G...
Ride the Storm: Navigating Through Unstable Periods / Katerina Rudko (Belka G...Ride the Storm: Navigating Through Unstable Periods / Katerina Rudko (Belka G...
Ride the Storm: Navigating Through Unstable Periods / Katerina Rudko (Belka G...DevGAMM Conference
 
Barbie - Brand Strategy Presentation
Barbie - Brand Strategy PresentationBarbie - Brand Strategy Presentation
Barbie - Brand Strategy PresentationErica Santiago
 
Good Stuff Happens in 1:1 Meetings: Why you need them and how to do them well
Good Stuff Happens in 1:1 Meetings: Why you need them and how to do them wellGood Stuff Happens in 1:1 Meetings: Why you need them and how to do them well
Good Stuff Happens in 1:1 Meetings: Why you need them and how to do them wellSaba Software
 

En vedette (20)

Content Methodology: A Best Practices Report (Webinar)
Content Methodology: A Best Practices Report (Webinar)Content Methodology: A Best Practices Report (Webinar)
Content Methodology: A Best Practices Report (Webinar)
 
How to Prepare For a Successful Job Search for 2024
How to Prepare For a Successful Job Search for 2024How to Prepare For a Successful Job Search for 2024
How to Prepare For a Successful Job Search for 2024
 
Social Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie InsightsSocial Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie Insights
 
Trends In Paid Search: Navigating The Digital Landscape In 2024
Trends In Paid Search: Navigating The Digital Landscape In 2024Trends In Paid Search: Navigating The Digital Landscape In 2024
Trends In Paid Search: Navigating The Digital Landscape In 2024
 
5 Public speaking tips from TED - Visualized summary
5 Public speaking tips from TED - Visualized summary5 Public speaking tips from TED - Visualized summary
5 Public speaking tips from TED - Visualized summary
 
ChatGPT and the Future of Work - Clark Boyd
ChatGPT and the Future of Work - Clark Boyd ChatGPT and the Future of Work - Clark Boyd
ChatGPT and the Future of Work - Clark Boyd
 
Getting into the tech field. what next
Getting into the tech field. what next Getting into the tech field. what next
Getting into the tech field. what next
 
Google's Just Not That Into You: Understanding Core Updates & Search Intent
Google's Just Not That Into You: Understanding Core Updates & Search IntentGoogle's Just Not That Into You: Understanding Core Updates & Search Intent
Google's Just Not That Into You: Understanding Core Updates & Search Intent
 
How to have difficult conversations
How to have difficult conversations How to have difficult conversations
How to have difficult conversations
 
Introduction to Data Science
Introduction to Data ScienceIntroduction to Data Science
Introduction to Data Science
 
Time Management & Productivity - Best Practices
Time Management & Productivity -  Best PracticesTime Management & Productivity -  Best Practices
Time Management & Productivity - Best Practices
 
The six step guide to practical project management
The six step guide to practical project managementThe six step guide to practical project management
The six step guide to practical project management
 
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
 
Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...
Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...
Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...
 
12 Ways to Increase Your Influence at Work
12 Ways to Increase Your Influence at Work12 Ways to Increase Your Influence at Work
12 Ways to Increase Your Influence at Work
 
ChatGPT webinar slides
ChatGPT webinar slidesChatGPT webinar slides
ChatGPT webinar slides
 
More than Just Lines on a Map: Best Practices for U.S Bike Routes
More than Just Lines on a Map: Best Practices for U.S Bike RoutesMore than Just Lines on a Map: Best Practices for U.S Bike Routes
More than Just Lines on a Map: Best Practices for U.S Bike Routes
 
Ride the Storm: Navigating Through Unstable Periods / Katerina Rudko (Belka G...
Ride the Storm: Navigating Through Unstable Periods / Katerina Rudko (Belka G...Ride the Storm: Navigating Through Unstable Periods / Katerina Rudko (Belka G...
Ride the Storm: Navigating Through Unstable Periods / Katerina Rudko (Belka G...
 
Barbie - Brand Strategy Presentation
Barbie - Brand Strategy PresentationBarbie - Brand Strategy Presentation
Barbie - Brand Strategy Presentation
 
Good Stuff Happens in 1:1 Meetings: Why you need them and how to do them well
Good Stuff Happens in 1:1 Meetings: Why you need them and how to do them wellGood Stuff Happens in 1:1 Meetings: Why you need them and how to do them well
Good Stuff Happens in 1:1 Meetings: Why you need them and how to do them well
 

Seattle Scalability Meetup