SlideShare une entreprise Scribd logo
1  sur  30
Télécharger pour lire hors ligne
Mio
a distributed Skip Graph based orderd KVS


            Cybozu Labs, Inc.
         Taro Minowa (Higepon)
Introduce myself

   @higepon
   Mona OS
         http://www.monaos.org
   Mosh
         A fast Scheme interpreter
   Outputz
         http://outputz.com


Feb 26 2010            Mio - a Skip Graph based ordered KVS   2
Summary

   Mio is...
         a distributed orderd KVS
         memcached + range search
         Skip Graph based
         Written in Erlang
         http://github.com/higepon/mio
         In alpha quality



Feb 26 2010           Mio - a Skip Graph based ordered KVS   3
Background




Feb 26 2010   Mio - a Skip Graph based ordered KVS   4
RDBMS vs KVS
                                                 Scalability
              KVS   set/get

         volatile

                                                                            High
                                                                            functionality



                                                                           Transaction

                                                                     SQL     RDBMS


Feb 26 2010                   Mio - a Skip Graph based ordered KVS                          5
RDBMS vs KVS
                                                 Scalability
              KVS   set/get

         volatile

                                                                            High
                        Complement each other
                                                                            functionality



                                                                           Transaction

                                                                     SQL     RDBMS


Feb 26 2010                   Mio - a Skip Graph based ordered KVS                          5
Mio
                                       Scalability
              KVS



                                                           High
                                                           functionality




                                                            RDBMS


Feb 26 2010         Mio - a Skip Graph based ordered KVS                   6
Mio
                                        Scalability
              KVS   Mio

                    +Range search

                                                            High
                                                            functionality




                                                             RDBMS


Feb 26 2010          Mio - a Skip Graph based ordered KVS                   6
Mio
                                        Scalability
              KVS   Mio

                    +Range search

                       Makes RDBMS                          High
                       lighter workload                     functionality




                                                             RDBMS


Feb 26 2010          Mio - a Skip Graph based ordered KVS                   6
Range search?

   Queries
         last 7 days
         prev/next
         Top 10 ranking
   SQL
         SELECT * FROM photos WHERE date between xxx
          and xx order by date limit 10
   RDBMS handles these queires

Feb 26 2010            Mio - a Skip Graph based ordered KVS   7
Mio




Feb 26 2010   Mio - a Skip Graph based ordered KVS   8
The Challenges and Design Decisions

   Range search
         Ordered structure
         Skip Graphs algorithm
   Scale-Out
         distributed using Erlang functions
   memcached compatible I/F
   Volatile
         keep it simple

Feb 26 2010                Mio - a Skip Graph based ordered KVS   9
Skip Graphs
                  James Aspnes (2003)




Feb 26 2010   Mio - a Skip Graph based ordered KVS   10
Supported operations

   search by key
   insert (join)
   remove
   range search by key1 and key2




Feb 26 2010         Mio - a Skip Graph based ordered KVS   11
Set of sorted doubly linked lists


        Shibuya   Shinjuku    Tamachi              Ueno             Yoyogi


    Same as railway stations
           All keys (stations) consist doubly linked list
           Knows only his left and right station
           Keep sorted by key
    Search Shibuya start from Ueno
           Go to left. O(n)

Feb 26 2010                  Mio - a Skip Graph based ordered KVS            12
Make an express lane
                               Skip
 Express
                  Shinjuku                         Ueno


 Local
        Shibuya   Shinjuku    Tamachi              Ueno             Yoyogi


    Skip some stations
    Ueno -> Shinjuku -> Shibuya
    Tamachi is placed on another express
     lane
Feb 26 2010                  Mio - a Skip Graph based ordered KVS            13
Multiple lanes
 Level 2


 Level 1


 Level 0


              Shibuya   Shinjuku   Tamachi         Ueno                   Yoyogi


 Level 0 lane
       all keys are in the list
 Level n (n > 0) lane
       express lane
       n + 1 lane is more express than n lane.
Feb 26 2010                        Mio - a Skip Graph based ordered KVS            14
Search
 Level 2


 Level 1


 Level 0


              Shibuya   Shinjuku   Tamachi         Ueno                   Yoyogi


 Start from highest to lower level
 Can search from any stations
 O(log n)

Feb 26 2010                        Mio - a Skip Graph based ordered KVS            15
Range Search
 Level 2


 Level 1


 Level 0


              Shibuya   Shinjuku   Tamachi         Ueno                   Yoyogi


 Search key1
 Collect matched on Level 0
 ex. Key1 = Ueno , Key2 = Shibuya

Feb 26 2010                        Mio - a Skip Graph based ordered KVS            16
Remove

                                                             B


        A      B        C                        A               C




 Remove on each Level
       Update neighbor’s links
       Highest to lower




Feb 26 2010           Mio - a Skip Graph based ordered KVS           17
Insert

                      B


          A                      C                     A           B   C




 Insert on each Level
       Update neighbors’s links
       Lowest to higher (in reverse order to remove)
       In which express lane is a new station insereted?
              radomly located
              uniform


Feb 26 2010                 Mio - a Skip Graph based ordered KVS           18
Easy to implement?

   No
         Really simple, but ...
         We should support concurrent insert/remove
              If neighbor is removed when inserting?
              If someone inserts another to neighbor?
              Searching crash?
         Fragile linked list
         We can’t find any perfect concurrent join
          algorithm.

Feb 26 2010                 Mio - a Skip Graph based ordered KVS   19
Our concurrent algorithm

   Lock some nodes
   Please read the source code :)
   Defined three invariants

              A           B              C




                  A           C                      A                   C



                      B                                          B




Feb 26 2010                       Mio - a Skip Graph based ordered KVS       20
Implementation




Feb 26 2010   Mio - a Skip Graph based ordered KVS   21
Written in Erlang

   A station(key, value) is a process
         gen_server process
         Hold left/right on each level
         Follow left/right = gen_server:call/2
         No distinction between local and remote process
              Erlang is great!

   Ditributed with -name option
         erl -name name@FQDN


Feb 26 2010                  Mio - a Skip Graph based ordered KVS   22
Performance

   5000 qps on single node
   really slow on multiple nodes
         need less communication between nodes
         need better algorithm




Feb 26 2010           Mio - a Skip Graph based ordered KVS   23
Demo




Feb 26 2010   Mio - a Skip Graph based ordered KVS   24
Tips for practical Erlang

   Max process option +P
         Set proper value. Don’t use MAX.
   gerbage_collect()
         Fast enough, reduce memory usage.
         hibernate is slow...
   refactorerl
   fprof on gen_server shows nothing
         Use dynomite profile

Feb 26 2010            Mio - a Skip Graph based ordered KVS   25
Tips for practical Erlang

   Common test
         Coverage
         load test
   gen_server:call is slow
         Use mnesia for property access.
         Easy replication
   Easy to run
         Should users run erl with many options?
         Shell script borrowed from RabbitMQ
Feb 26 2010            Mio - a Skip Graph based ordered KVS   26
Summary, Once more

   Mio is...
         a distributed orderd KVS
         memcached + range search
         Skip Graph based
         Written in Erlang
         http://github.com/higepon/mio
         In alpha quality



Feb 26 2010           Mio - a Skip Graph based ordered KVS   27

Contenu connexe

Dernier

Strategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a FresherStrategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a FresherRemote DBA Services
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerThousandEyes
 
Tata AIG General Insurance Company - Insurer Innovation Award 2024
Tata AIG General Insurance Company - Insurer Innovation Award 2024Tata AIG General Insurance Company - Insurer Innovation Award 2024
Tata AIG General Insurance Company - Insurer Innovation Award 2024The Digital Insurer
 
HTML Injection Attacks: Impact and Mitigation Strategies
HTML Injection Attacks: Impact and Mitigation StrategiesHTML Injection Attacks: Impact and Mitigation Strategies
HTML Injection Attacks: Impact and Mitigation StrategiesBoston Institute of Analytics
 
Boost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivityBoost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivityPrincipled Technologies
 
Handwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed textsHandwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed textsMaria Levchenko
 
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, AdobeApidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobeapidays
 
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
Workshop - Best of Both Worlds_ Combine  KG and Vector search for  enhanced R...Workshop - Best of Both Worlds_ Combine  KG and Vector search for  enhanced R...
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...Neo4j
 
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
04-2024-HHUG-Sales-and-Marketing-Alignment.pptxHampshireHUG
 
Scaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organizationScaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organizationRadu Cotescu
 
Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)wesley chun
 
Finology Group – Insurtech Innovation Award 2024
Finology Group – Insurtech Innovation Award 2024Finology Group – Insurtech Innovation Award 2024
Finology Group – Insurtech Innovation Award 2024The Digital Insurer
 
2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...Martijn de Jong
 
Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024The Digital Insurer
 
Artificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and MythsArtificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and MythsJoaquim Jorge
 
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...apidays
 
TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
TrustArc Webinar - Unlock the Power of AI-Driven Data DiscoveryTrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
TrustArc Webinar - Unlock the Power of AI-Driven Data DiscoveryTrustArc
 
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...DianaGray10
 
Advantages of Hiring UIUX Design Service Providers for Your Business
Advantages of Hiring UIUX Design Service Providers for Your BusinessAdvantages of Hiring UIUX Design Service Providers for Your Business
Advantages of Hiring UIUX Design Service Providers for Your BusinessPixlogix Infotech
 
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time AutomationFrom Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time AutomationSafe Software
 

Dernier (20)

Strategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a FresherStrategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a Fresher
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
 
Tata AIG General Insurance Company - Insurer Innovation Award 2024
Tata AIG General Insurance Company - Insurer Innovation Award 2024Tata AIG General Insurance Company - Insurer Innovation Award 2024
Tata AIG General Insurance Company - Insurer Innovation Award 2024
 
HTML Injection Attacks: Impact and Mitigation Strategies
HTML Injection Attacks: Impact and Mitigation StrategiesHTML Injection Attacks: Impact and Mitigation Strategies
HTML Injection Attacks: Impact and Mitigation Strategies
 
Boost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivityBoost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivity
 
Handwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed textsHandwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed texts
 
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, AdobeApidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
 
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
Workshop - Best of Both Worlds_ Combine  KG and Vector search for  enhanced R...Workshop - Best of Both Worlds_ Combine  KG and Vector search for  enhanced R...
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
 
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
 
Scaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organizationScaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organization
 
Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)
 
Finology Group – Insurtech Innovation Award 2024
Finology Group – Insurtech Innovation Award 2024Finology Group – Insurtech Innovation Award 2024
Finology Group – Insurtech Innovation Award 2024
 
2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...
 
Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024
 
Artificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and MythsArtificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and Myths
 
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
 
TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
TrustArc Webinar - Unlock the Power of AI-Driven Data DiscoveryTrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
 
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
 
Advantages of Hiring UIUX Design Service Providers for Your Business
Advantages of Hiring UIUX Design Service Providers for Your BusinessAdvantages of Hiring UIUX Design Service Providers for Your Business
Advantages of Hiring UIUX Design Service Providers for Your Business
 
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time AutomationFrom Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
 

En vedette

Product Design Trends in 2024 | Teenage Engineerings
Product Design Trends in 2024 | Teenage EngineeringsProduct Design Trends in 2024 | Teenage Engineerings
Product Design Trends in 2024 | Teenage EngineeringsPixeldarts
 
How Race, Age and Gender Shape Attitudes Towards Mental Health
How Race, Age and Gender Shape Attitudes Towards Mental HealthHow Race, Age and Gender Shape Attitudes Towards Mental Health
How Race, Age and Gender Shape Attitudes Towards Mental HealthThinkNow
 
AI Trends in Creative Operations 2024 by Artwork Flow.pdf
AI Trends in Creative Operations 2024 by Artwork Flow.pdfAI Trends in Creative Operations 2024 by Artwork Flow.pdf
AI Trends in Creative Operations 2024 by Artwork Flow.pdfmarketingartwork
 
PEPSICO Presentation to CAGNY Conference Feb 2024
PEPSICO Presentation to CAGNY Conference Feb 2024PEPSICO Presentation to CAGNY Conference Feb 2024
PEPSICO Presentation to CAGNY Conference Feb 2024Neil Kimberley
 
Content Methodology: A Best Practices Report (Webinar)
Content Methodology: A Best Practices Report (Webinar)Content Methodology: A Best Practices Report (Webinar)
Content Methodology: A Best Practices Report (Webinar)contently
 
How to Prepare For a Successful Job Search for 2024
How to Prepare For a Successful Job Search for 2024How to Prepare For a Successful Job Search for 2024
How to Prepare For a Successful Job Search for 2024Albert Qian
 
Social Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie InsightsSocial Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie InsightsKurio // The Social Media Age(ncy)
 
Trends In Paid Search: Navigating The Digital Landscape In 2024
Trends In Paid Search: Navigating The Digital Landscape In 2024Trends In Paid Search: Navigating The Digital Landscape In 2024
Trends In Paid Search: Navigating The Digital Landscape In 2024Search Engine Journal
 
5 Public speaking tips from TED - Visualized summary
5 Public speaking tips from TED - Visualized summary5 Public speaking tips from TED - Visualized summary
5 Public speaking tips from TED - Visualized summarySpeakerHub
 
ChatGPT and the Future of Work - Clark Boyd
ChatGPT and the Future of Work - Clark Boyd ChatGPT and the Future of Work - Clark Boyd
ChatGPT and the Future of Work - Clark Boyd Clark Boyd
 
Getting into the tech field. what next
Getting into the tech field. what next Getting into the tech field. what next
Getting into the tech field. what next Tessa Mero
 
Google's Just Not That Into You: Understanding Core Updates & Search Intent
Google's Just Not That Into You: Understanding Core Updates & Search IntentGoogle's Just Not That Into You: Understanding Core Updates & Search Intent
Google's Just Not That Into You: Understanding Core Updates & Search IntentLily Ray
 
Time Management & Productivity - Best Practices
Time Management & Productivity -  Best PracticesTime Management & Productivity -  Best Practices
Time Management & Productivity - Best PracticesVit Horky
 
The six step guide to practical project management
The six step guide to practical project managementThe six step guide to practical project management
The six step guide to practical project managementMindGenius
 
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...RachelPearson36
 
Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...
Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...
Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...Applitools
 
12 Ways to Increase Your Influence at Work
12 Ways to Increase Your Influence at Work12 Ways to Increase Your Influence at Work
12 Ways to Increase Your Influence at WorkGetSmarter
 

En vedette (20)

Product Design Trends in 2024 | Teenage Engineerings
Product Design Trends in 2024 | Teenage EngineeringsProduct Design Trends in 2024 | Teenage Engineerings
Product Design Trends in 2024 | Teenage Engineerings
 
How Race, Age and Gender Shape Attitudes Towards Mental Health
How Race, Age and Gender Shape Attitudes Towards Mental HealthHow Race, Age and Gender Shape Attitudes Towards Mental Health
How Race, Age and Gender Shape Attitudes Towards Mental Health
 
AI Trends in Creative Operations 2024 by Artwork Flow.pdf
AI Trends in Creative Operations 2024 by Artwork Flow.pdfAI Trends in Creative Operations 2024 by Artwork Flow.pdf
AI Trends in Creative Operations 2024 by Artwork Flow.pdf
 
Skeleton Culture Code
Skeleton Culture CodeSkeleton Culture Code
Skeleton Culture Code
 
PEPSICO Presentation to CAGNY Conference Feb 2024
PEPSICO Presentation to CAGNY Conference Feb 2024PEPSICO Presentation to CAGNY Conference Feb 2024
PEPSICO Presentation to CAGNY Conference Feb 2024
 
Content Methodology: A Best Practices Report (Webinar)
Content Methodology: A Best Practices Report (Webinar)Content Methodology: A Best Practices Report (Webinar)
Content Methodology: A Best Practices Report (Webinar)
 
How to Prepare For a Successful Job Search for 2024
How to Prepare For a Successful Job Search for 2024How to Prepare For a Successful Job Search for 2024
How to Prepare For a Successful Job Search for 2024
 
Social Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie InsightsSocial Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie Insights
 
Trends In Paid Search: Navigating The Digital Landscape In 2024
Trends In Paid Search: Navigating The Digital Landscape In 2024Trends In Paid Search: Navigating The Digital Landscape In 2024
Trends In Paid Search: Navigating The Digital Landscape In 2024
 
5 Public speaking tips from TED - Visualized summary
5 Public speaking tips from TED - Visualized summary5 Public speaking tips from TED - Visualized summary
5 Public speaking tips from TED - Visualized summary
 
ChatGPT and the Future of Work - Clark Boyd
ChatGPT and the Future of Work - Clark Boyd ChatGPT and the Future of Work - Clark Boyd
ChatGPT and the Future of Work - Clark Boyd
 
Getting into the tech field. what next
Getting into the tech field. what next Getting into the tech field. what next
Getting into the tech field. what next
 
Google's Just Not That Into You: Understanding Core Updates & Search Intent
Google's Just Not That Into You: Understanding Core Updates & Search IntentGoogle's Just Not That Into You: Understanding Core Updates & Search Intent
Google's Just Not That Into You: Understanding Core Updates & Search Intent
 
How to have difficult conversations
How to have difficult conversations How to have difficult conversations
How to have difficult conversations
 
Introduction to Data Science
Introduction to Data ScienceIntroduction to Data Science
Introduction to Data Science
 
Time Management & Productivity - Best Practices
Time Management & Productivity -  Best PracticesTime Management & Productivity -  Best Practices
Time Management & Productivity - Best Practices
 
The six step guide to practical project management
The six step guide to practical project managementThe six step guide to practical project management
The six step guide to practical project management
 
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
 
Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...
Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...
Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...
 
12 Ways to Increase Your Influence at Work
12 Ways to Increase Your Influence at Work12 Ways to Increase Your Influence at Work
12 Ways to Increase Your Influence at Work
 

Mio - a distributed Skip Graph based orderd KVS

  • 1. Mio a distributed Skip Graph based orderd KVS Cybozu Labs, Inc. Taro Minowa (Higepon)
  • 2. Introduce myself @higepon Mona OS  http://www.monaos.org Mosh  A fast Scheme interpreter Outputz  http://outputz.com Feb 26 2010 Mio - a Skip Graph based ordered KVS 2
  • 3. Summary Mio is...  a distributed orderd KVS  memcached + range search  Skip Graph based  Written in Erlang  http://github.com/higepon/mio  In alpha quality Feb 26 2010 Mio - a Skip Graph based ordered KVS 3
  • 4. Background Feb 26 2010 Mio - a Skip Graph based ordered KVS 4
  • 5. RDBMS vs KVS Scalability KVS set/get volatile High functionality Transaction SQL RDBMS Feb 26 2010 Mio - a Skip Graph based ordered KVS 5
  • 6. RDBMS vs KVS Scalability KVS set/get volatile High Complement each other functionality Transaction SQL RDBMS Feb 26 2010 Mio - a Skip Graph based ordered KVS 5
  • 7. Mio Scalability KVS High functionality RDBMS Feb 26 2010 Mio - a Skip Graph based ordered KVS 6
  • 8. Mio Scalability KVS Mio +Range search High functionality RDBMS Feb 26 2010 Mio - a Skip Graph based ordered KVS 6
  • 9. Mio Scalability KVS Mio +Range search Makes RDBMS High lighter workload functionality RDBMS Feb 26 2010 Mio - a Skip Graph based ordered KVS 6
  • 10. Range search? Queries  last 7 days  prev/next  Top 10 ranking SQL  SELECT * FROM photos WHERE date between xxx and xx order by date limit 10 RDBMS handles these queires Feb 26 2010 Mio - a Skip Graph based ordered KVS 7
  • 11. Mio Feb 26 2010 Mio - a Skip Graph based ordered KVS 8
  • 12. The Challenges and Design Decisions Range search  Ordered structure  Skip Graphs algorithm Scale-Out  distributed using Erlang functions memcached compatible I/F Volatile  keep it simple Feb 26 2010 Mio - a Skip Graph based ordered KVS 9
  • 13. Skip Graphs James Aspnes (2003) Feb 26 2010 Mio - a Skip Graph based ordered KVS 10
  • 14. Supported operations search by key insert (join) remove range search by key1 and key2 Feb 26 2010 Mio - a Skip Graph based ordered KVS 11
  • 15. Set of sorted doubly linked lists Shibuya Shinjuku Tamachi Ueno Yoyogi Same as railway stations  All keys (stations) consist doubly linked list  Knows only his left and right station  Keep sorted by key Search Shibuya start from Ueno  Go to left. O(n) Feb 26 2010 Mio - a Skip Graph based ordered KVS 12
  • 16. Make an express lane Skip Express Shinjuku Ueno Local Shibuya Shinjuku Tamachi Ueno Yoyogi Skip some stations Ueno -> Shinjuku -> Shibuya Tamachi is placed on another express lane Feb 26 2010 Mio - a Skip Graph based ordered KVS 13
  • 17. Multiple lanes Level 2 Level 1 Level 0 Shibuya Shinjuku Tamachi Ueno Yoyogi Level 0 lane  all keys are in the list Level n (n > 0) lane  express lane  n + 1 lane is more express than n lane. Feb 26 2010 Mio - a Skip Graph based ordered KVS 14
  • 18. Search Level 2 Level 1 Level 0 Shibuya Shinjuku Tamachi Ueno Yoyogi Start from highest to lower level Can search from any stations O(log n) Feb 26 2010 Mio - a Skip Graph based ordered KVS 15
  • 19. Range Search Level 2 Level 1 Level 0 Shibuya Shinjuku Tamachi Ueno Yoyogi Search key1 Collect matched on Level 0 ex. Key1 = Ueno , Key2 = Shibuya Feb 26 2010 Mio - a Skip Graph based ordered KVS 16
  • 20. Remove B A B C A C Remove on each Level  Update neighbor’s links  Highest to lower Feb 26 2010 Mio - a Skip Graph based ordered KVS 17
  • 21. Insert B A C A B C Insert on each Level  Update neighbors’s links  Lowest to higher (in reverse order to remove)  In which express lane is a new station insereted? radomly located uniform Feb 26 2010 Mio - a Skip Graph based ordered KVS 18
  • 22. Easy to implement? No  Really simple, but ...  We should support concurrent insert/remove If neighbor is removed when inserting? If someone inserts another to neighbor? Searching crash?  Fragile linked list  We can’t find any perfect concurrent join algorithm. Feb 26 2010 Mio - a Skip Graph based ordered KVS 19
  • 23. Our concurrent algorithm Lock some nodes Please read the source code :) Defined three invariants A B C A C A C B B Feb 26 2010 Mio - a Skip Graph based ordered KVS 20
  • 24. Implementation Feb 26 2010 Mio - a Skip Graph based ordered KVS 21
  • 25. Written in Erlang A station(key, value) is a process  gen_server process  Hold left/right on each level  Follow left/right = gen_server:call/2  No distinction between local and remote process Erlang is great! Ditributed with -name option  erl -name name@FQDN Feb 26 2010 Mio - a Skip Graph based ordered KVS 22
  • 26. Performance 5000 qps on single node really slow on multiple nodes  need less communication between nodes  need better algorithm Feb 26 2010 Mio - a Skip Graph based ordered KVS 23
  • 27. Demo Feb 26 2010 Mio - a Skip Graph based ordered KVS 24
  • 28. Tips for practical Erlang Max process option +P  Set proper value. Don’t use MAX. gerbage_collect()  Fast enough, reduce memory usage.  hibernate is slow... refactorerl fprof on gen_server shows nothing  Use dynomite profile Feb 26 2010 Mio - a Skip Graph based ordered KVS 25
  • 29. Tips for practical Erlang Common test  Coverage  load test gen_server:call is slow  Use mnesia for property access.  Easy replication Easy to run  Should users run erl with many options?  Shell script borrowed from RabbitMQ Feb 26 2010 Mio - a Skip Graph based ordered KVS 26
  • 30. Summary, Once more Mio is...  a distributed orderd KVS  memcached + range search  Skip Graph based  Written in Erlang  http://github.com/higepon/mio  In alpha quality Feb 26 2010 Mio - a Skip Graph based ordered KVS 27