SlideShare une entreprise Scribd logo
1  sur  29
Tomcy Thankachan
   Introduction
   Data model
   Building Blocks
   Implementation
   Refinements
   Performance Evaluation
   Real applications
   Conclusion
   Scale is too large for most commercial Databases

   Cost would be very high

   Low-level storage optimizations help performance significantly

   Hard to map semi-structured data to relational database

   Non-uniform fields makes it difficult to insert/query data
   BigTable is a distributed storage system for managing structured data

   Bigtable does not support a full relational data model

   Scalable

   Self managing
   Used by more than 60 Google products
     Google Analytics
     Google Finance
     Personalized Search
     Google Documents
     Google Earth
     Google Fusion Tables
     …


   Used for variety of demanding workloads
     Throughput oriented batch processing
     Latency sensitive data serving
   Goals

       Wide applicability
       Scalability
       High performance
       High availability

   Simple data model that supports dynamic control over data layout and
    format
   A Bigtable is a sparse, distributed, persistent multidimensional sorted
    map.
   The map is indexed by a row key, column key, and a timestamp.
     (row:string, column:string, time:int64) → string

        Webtable
   The row keys in a table are arbitrary strings.
   Data is maintained in lexicographic order by row key
   Each row range is called a tablet, which is the unit of distribution
    and load balancing.




                 row
   Column keys are grouped into sets called column families.
   Data stored in a column family is usually of the same type
   A column key is named using the syntax: family : qualifier.
   Column family names must be printable , but qualifiers may be arbitrary
    strings.

                          columns
 Each cell in a Bigtable can contain multiple versions of the same data
 Versions are indexed by 64-bit integer timestamps
 Timestamps can be assigned:
       ▪ automatically by Bigtable , or
       ▪ explicitly by client applications




                                             timestamp
   The Bigtable API provides functions :

         Creating and deleting tables and column families.
         Changing cluster , table and column family metadata.
          Support for single row transactions
          Allows cells to be used as integer counters
          Client supplied scripts can be executed in the address space of
        servers
   Bigtable is built on several other pieces of Google
infrastructure.
 Google File system(GFS)
 SSTable : Data structure for storage




                          Index(block ranges)


                64K         64K                 64K
                                        ……….    block
                block       block


   Chubby: Distributed lock service.
   Three major components
     Library linked into every client
     Single master server
      ▪ Assigning tablets to tablet servers
      ▪ Detecting addition and expiration of tablet servers
      ▪ Balancing tablet-server load
      ▪ Garbage collection files in GFS
     Many tablet servers
      ▪ Manages a set of tablets
      ▪ Tablet servers handle read and write requests to its table
      ▪ Splits tablets that have grown too large
▪ Clients communicates directly with tablet servers for
    read/write

    Each table consists of a set of tablets
      Initially, each table have just one tablet
      Tablets are automatically split as the table grows

    Row size can be arbitrary (hundreds of GB)
   Three level hierarchy
     Level 2: Root tablet contains the location of METADATA tablets
     Level 3: Each METADATA tablet contains the location of user tablets
     Level 1: Chubby file containing location of the root tablet

       ▪ Location of tablet is stored under a row key that
         encodes table identifier and its end row
   Tablet server startup
     It creates and acquires an exclusive lock on , a uniquely
      named file on Chubby.
     Master monitors this directory to discover tablet servers.


   Tablet server stops serving tablets
     If it loses its exclusive lock.
     Tries to reacquire the lock on its file as long as the file still
      exists.
     If file no longer exists, the tablet server will never be able
      to serve again.
   Master server startup
     Grabs unique master lock in Chubby.
     Scans the tablet server directory in Chubby.
     Communicates with every live tablet server
     Scans METADATA table to learn set of tablets.


   Master is responsible for finding when tablet server is no longer serving
    its tablets and reassigning those tablets as soon as possible.
     Periodically asks each tablet server for the status of its lock
     If no reply, master tries to acquire the lock itself
     If successful to acquire lock, then tablet server is either dead or having
       network trouble
   Updates committed to a commit log

   Recently committed updates are stored in memory –memtable

   Older updates are stored in a sequence of SSTables.
   Write operation
                                              Read operation
     Server checks if it is well-formed
                                                Check well-formedness of request.
     Checks if the sender is
                                                Check authorization in Chubby file
      authorized
                                                Merge memtable and SSTables to
     Write to commit log
                                                 find data
     After commit, contents are
                                                Return data.
      inserted into Memtable
In order to control size of memtable, tablet log, and SSTable
   files, “compaction” is used.


1. Minor Compaction.-             Move data from memtable to SSTable.


2. Merging Compaction. - Merge multiple SSTables and
  memtable to a single SSTable.
3. Major Compaction. - that re-writes all SSTables into exactly
  one SSTable
   Locality groups
     Clients can group multiple column families together into a locality
      group.


   Compression
     Compression applied to each SSTable block separately
     Uses Bentley and McIlroy's scheme and fast compression algorithm


   Bloom filters
     Reduce the number of disk accesses
   Caching
     Scan Cache: a high-level cache that caches key-value pairs returned by
      the SSTable interface
     Block Cache: a lower-level cache that caches SSTable blocks read from
      file system


   Commit-log implementation
     Suppose one log per tablet rather have one log per tablet server
   Satisfies goals of high-availability, high-performance, massively scalable
    data storage.
     -It has been successfully deployed in real apps (Personalized
      Search, Orkut, GoogleMaps, …)
   Successfully used by various Google products (>60).
   It’s a distributed systems, designed to store enormous volumes of data.
   Thesesystems can be easily scaled to accommodate peta-bytes of data
    across thousands of nodes.
   Bigtable: A Distributed Storage System for Structured Data by Fay Chang, Jeffrey
    Dean, Sanjay Ghemawat, Wilson C. Hsieh, Deborah A. Wallach, Mike Burrows,
    Tushar Chandra, Andrew Fikes, Robert E. Gruber

   http://glinden.blogspot.com/2006/08/google-bigtable-paper.html


   Dean, J. 2005. BigTable: A Distributed Structured Storage System. University of
    Washington CSE Colloquia (Oct. 2005).
    http://www.uwtv.org/programs/displayevent.aspx?rID=4188
GOOGLE BIGTABLE
GOOGLE BIGTABLE

Contenu connexe

Tendances

NOSQL- Presentation on NoSQL
NOSQL- Presentation on NoSQLNOSQL- Presentation on NoSQL
NOSQL- Presentation on NoSQLRamakant Soni
 
Google Megastore
Google MegastoreGoogle Megastore
Google Megastorebergwolf
 
NoSQL databases - An introduction
NoSQL databases - An introductionNoSQL databases - An introduction
NoSQL databases - An introductionPooyan Mehrparvar
 
A tour of Amazon Redshift
A tour of Amazon RedshiftA tour of Amazon Redshift
A tour of Amazon RedshiftKel Graham
 
An Overview of Spanner: Google's Globally Distributed Database
An Overview of Spanner: Google's Globally Distributed DatabaseAn Overview of Spanner: Google's Globally Distributed Database
An Overview of Spanner: Google's Globally Distributed DatabaseBenjamin Bengfort
 
What is NoSQL and CAP Theorem
What is NoSQL and CAP TheoremWhat is NoSQL and CAP Theorem
What is NoSQL and CAP TheoremRahul Jain
 
Google Cloud Dataproc - Easier, faster, more cost-effective Spark and Hadoop
Google Cloud Dataproc - Easier, faster, more cost-effective Spark and HadoopGoogle Cloud Dataproc - Easier, faster, more cost-effective Spark and Hadoop
Google Cloud Dataproc - Easier, faster, more cost-effective Spark and Hadoophuguk
 
Google File System
Google File SystemGoogle File System
Google File Systemnadikari123
 
Introduction to Cassandra
Introduction to CassandraIntroduction to Cassandra
Introduction to CassandraGokhan Atil
 
Hadoop Distributed File System
Hadoop Distributed File SystemHadoop Distributed File System
Hadoop Distributed File SystemRutvik Bapat
 
The Google File System (GFS)
The Google File System (GFS)The Google File System (GFS)
The Google File System (GFS)Romain Jacotin
 
Google Spanner - Synchronously-Replicated, Globally-Distributed, Multi-Versio...
Google Spanner - Synchronously-Replicated, Globally-Distributed, Multi-Versio...Google Spanner - Synchronously-Replicated, Globally-Distributed, Multi-Versio...
Google Spanner - Synchronously-Replicated, Globally-Distributed, Multi-Versio...Maciek Jozwiak
 
Hadoop Distributed file system.pdf
Hadoop Distributed file system.pdfHadoop Distributed file system.pdf
Hadoop Distributed file system.pdfvishal choudhary
 
OSA Con 2022 - Apache Iceberg_ An Architectural Look Under the Covers - Alex ...
OSA Con 2022 - Apache Iceberg_ An Architectural Look Under the Covers - Alex ...OSA Con 2022 - Apache Iceberg_ An Architectural Look Under the Covers - Alex ...
OSA Con 2022 - Apache Iceberg_ An Architectural Look Under the Covers - Alex ...Altinity Ltd
 

Tendances (20)

Apache HBase™
Apache HBase™Apache HBase™
Apache HBase™
 
NOSQL- Presentation on NoSQL
NOSQL- Presentation on NoSQLNOSQL- Presentation on NoSQL
NOSQL- Presentation on NoSQL
 
Google File System
Google File SystemGoogle File System
Google File System
 
Google Megastore
Google MegastoreGoogle Megastore
Google Megastore
 
NoSQL databases - An introduction
NoSQL databases - An introductionNoSQL databases - An introduction
NoSQL databases - An introduction
 
A tour of Amazon Redshift
A tour of Amazon RedshiftA tour of Amazon Redshift
A tour of Amazon Redshift
 
An Overview of Spanner: Google's Globally Distributed Database
An Overview of Spanner: Google's Globally Distributed DatabaseAn Overview of Spanner: Google's Globally Distributed Database
An Overview of Spanner: Google's Globally Distributed Database
 
What is NoSQL and CAP Theorem
What is NoSQL and CAP TheoremWhat is NoSQL and CAP Theorem
What is NoSQL and CAP Theorem
 
Google Cloud Dataproc - Easier, faster, more cost-effective Spark and Hadoop
Google Cloud Dataproc - Easier, faster, more cost-effective Spark and HadoopGoogle Cloud Dataproc - Easier, faster, more cost-effective Spark and Hadoop
Google Cloud Dataproc - Easier, faster, more cost-effective Spark and Hadoop
 
Google File System
Google File SystemGoogle File System
Google File System
 
AlwaysON Basics
AlwaysON BasicsAlwaysON Basics
AlwaysON Basics
 
kafka
kafkakafka
kafka
 
Introduction to Cassandra
Introduction to CassandraIntroduction to Cassandra
Introduction to Cassandra
 
Amazon Redshift
Amazon Redshift Amazon Redshift
Amazon Redshift
 
The Impala Cookbook
The Impala CookbookThe Impala Cookbook
The Impala Cookbook
 
Hadoop Distributed File System
Hadoop Distributed File SystemHadoop Distributed File System
Hadoop Distributed File System
 
The Google File System (GFS)
The Google File System (GFS)The Google File System (GFS)
The Google File System (GFS)
 
Google Spanner - Synchronously-Replicated, Globally-Distributed, Multi-Versio...
Google Spanner - Synchronously-Replicated, Globally-Distributed, Multi-Versio...Google Spanner - Synchronously-Replicated, Globally-Distributed, Multi-Versio...
Google Spanner - Synchronously-Replicated, Globally-Distributed, Multi-Versio...
 
Hadoop Distributed file system.pdf
Hadoop Distributed file system.pdfHadoop Distributed file system.pdf
Hadoop Distributed file system.pdf
 
OSA Con 2022 - Apache Iceberg_ An Architectural Look Under the Covers - Alex ...
OSA Con 2022 - Apache Iceberg_ An Architectural Look Under the Covers - Alex ...OSA Con 2022 - Apache Iceberg_ An Architectural Look Under the Covers - Alex ...
OSA Con 2022 - Apache Iceberg_ An Architectural Look Under the Covers - Alex ...
 

Similaire à GOOGLE BIGTABLE

Bigtable and Boxwood
Bigtable and BoxwoodBigtable and Boxwood
Bigtable and BoxwoodEvan Weaver
 
8. column oriented databases
8. column oriented databases8. column oriented databases
8. column oriented databasesFabio Fumarola
 
storage-systems.pptx
storage-systems.pptxstorage-systems.pptx
storage-systems.pptxShimoFcis
 
Summary of "Google's Big Table" at nosql summer reading in Tokyo
Summary of "Google's Big Table" at nosql summer reading in TokyoSummary of "Google's Big Table" at nosql summer reading in Tokyo
Summary of "Google's Big Table" at nosql summer reading in TokyoCLOUDIAN KK
 
Bigtable
BigtableBigtable
Bigtableptdorf
 
google file system
google file systemgoogle file system
google file systemdiptipan
 
Bigtable osdi06
Bigtable osdi06Bigtable osdi06
Bigtable osdi06temp2004it
 
Bigtable osdi06
Bigtable osdi06Bigtable osdi06
Bigtable osdi06mrlonganh
 
Data management in cloud study of existing systems and future opportunities
Data management in cloud study of existing systems and future opportunitiesData management in cloud study of existing systems and future opportunities
Data management in cloud study of existing systems and future opportunitiesEditor Jacotech
 
MySQL 8 Server Optimization Swanseacon 2018
MySQL 8 Server Optimization Swanseacon 2018MySQL 8 Server Optimization Swanseacon 2018
MySQL 8 Server Optimization Swanseacon 2018Dave Stokes
 
MySQL 8 Tips and Tricks from Symfony USA 2018, San Francisco
MySQL 8 Tips and Tricks from Symfony USA 2018, San FranciscoMySQL 8 Tips and Tricks from Symfony USA 2018, San Francisco
MySQL 8 Tips and Tricks from Symfony USA 2018, San FranciscoDave Stokes
 

Similaire à GOOGLE BIGTABLE (20)

Bigtable and Boxwood
Bigtable and BoxwoodBigtable and Boxwood
Bigtable and Boxwood
 
8. column oriented databases
8. column oriented databases8. column oriented databases
8. column oriented databases
 
storage-systems.pptx
storage-systems.pptxstorage-systems.pptx
storage-systems.pptx
 
google Bigtable
google Bigtablegoogle Bigtable
google Bigtable
 
Bigtable_Paper
Bigtable_PaperBigtable_Paper
Bigtable_Paper
 
Summary of "Google's Big Table" at nosql summer reading in Tokyo
Summary of "Google's Big Table" at nosql summer reading in TokyoSummary of "Google's Big Table" at nosql summer reading in Tokyo
Summary of "Google's Big Table" at nosql summer reading in Tokyo
 
Google file system
Google file systemGoogle file system
Google file system
 
Big table
Big tableBig table
Big table
 
Bigtable
BigtableBigtable
Bigtable
 
google file system
google file systemgoogle file system
google file system
 
Lalit
LalitLalit
Lalit
 
Google file system
Google file systemGoogle file system
Google file system
 
Bigtable osdi06
Bigtable osdi06Bigtable osdi06
Bigtable osdi06
 
Bigtable osdi06
Bigtable osdi06Bigtable osdi06
Bigtable osdi06
 
Bigtable osdi06
Bigtable osdi06Bigtable osdi06
Bigtable osdi06
 
Data management in cloud study of existing systems and future opportunities
Data management in cloud study of existing systems and future opportunitiesData management in cloud study of existing systems and future opportunities
Data management in cloud study of existing systems and future opportunities
 
MySQL 8 Server Optimization Swanseacon 2018
MySQL 8 Server Optimization Swanseacon 2018MySQL 8 Server Optimization Swanseacon 2018
MySQL 8 Server Optimization Swanseacon 2018
 
MySQL 8 Tips and Tricks from Symfony USA 2018, San Francisco
MySQL 8 Tips and Tricks from Symfony USA 2018, San FranciscoMySQL 8 Tips and Tricks from Symfony USA 2018, San Francisco
MySQL 8 Tips and Tricks from Symfony USA 2018, San Francisco
 
Fast Analytics
Fast Analytics Fast Analytics
Fast Analytics
 
Bigtable osdi06
Bigtable osdi06Bigtable osdi06
Bigtable osdi06
 

Dernier

18-04-UA_REPORT_MEDIALITERAСY_INDEX-DM_23-1-final-eng.pdf
18-04-UA_REPORT_MEDIALITERAСY_INDEX-DM_23-1-final-eng.pdf18-04-UA_REPORT_MEDIALITERAСY_INDEX-DM_23-1-final-eng.pdf
18-04-UA_REPORT_MEDIALITERAСY_INDEX-DM_23-1-final-eng.pdfssuser54595a
 
Arihant handbook biology for class 11 .pdf
Arihant handbook biology for class 11 .pdfArihant handbook biology for class 11 .pdf
Arihant handbook biology for class 11 .pdfchloefrazer622
 
Alper Gobel In Media Res Media Component
Alper Gobel In Media Res Media ComponentAlper Gobel In Media Res Media Component
Alper Gobel In Media Res Media ComponentInMediaRes1
 
Introduction to ArtificiaI Intelligence in Higher Education
Introduction to ArtificiaI Intelligence in Higher EducationIntroduction to ArtificiaI Intelligence in Higher Education
Introduction to ArtificiaI Intelligence in Higher Educationpboyjonauth
 
POINT- BIOCHEMISTRY SEM 2 ENZYMES UNIT 5.pptx
POINT- BIOCHEMISTRY SEM 2 ENZYMES UNIT 5.pptxPOINT- BIOCHEMISTRY SEM 2 ENZYMES UNIT 5.pptx
POINT- BIOCHEMISTRY SEM 2 ENZYMES UNIT 5.pptxSayali Powar
 
Concept of Vouching. B.Com(Hons) /B.Compdf
Concept of Vouching. B.Com(Hons) /B.CompdfConcept of Vouching. B.Com(Hons) /B.Compdf
Concept of Vouching. B.Com(Hons) /B.CompdfUmakantAnnand
 
The basics of sentences session 2pptx copy.pptx
The basics of sentences session 2pptx copy.pptxThe basics of sentences session 2pptx copy.pptx
The basics of sentences session 2pptx copy.pptxheathfieldcps1
 
Sanyam Choudhary Chemistry practical.pdf
Sanyam Choudhary Chemistry practical.pdfSanyam Choudhary Chemistry practical.pdf
Sanyam Choudhary Chemistry practical.pdfsanyamsingh5019
 
Measures of Central Tendency: Mean, Median and Mode
Measures of Central Tendency: Mean, Median and ModeMeasures of Central Tendency: Mean, Median and Mode
Measures of Central Tendency: Mean, Median and ModeThiyagu K
 
Organic Name Reactions for the students and aspirants of Chemistry12th.pptx
Organic Name Reactions  for the students and aspirants of Chemistry12th.pptxOrganic Name Reactions  for the students and aspirants of Chemistry12th.pptx
Organic Name Reactions for the students and aspirants of Chemistry12th.pptxVS Mahajan Coaching Centre
 
microwave assisted reaction. General introduction
microwave assisted reaction. General introductionmicrowave assisted reaction. General introduction
microwave assisted reaction. General introductionMaksud Ahmed
 
CARE OF CHILD IN INCUBATOR..........pptx
CARE OF CHILD IN INCUBATOR..........pptxCARE OF CHILD IN INCUBATOR..........pptx
CARE OF CHILD IN INCUBATOR..........pptxGaneshChakor2
 
“Oh GOSH! Reflecting on Hackteria's Collaborative Practices in a Global Do-It...
“Oh GOSH! Reflecting on Hackteria's Collaborative Practices in a Global Do-It...“Oh GOSH! Reflecting on Hackteria's Collaborative Practices in a Global Do-It...
“Oh GOSH! Reflecting on Hackteria's Collaborative Practices in a Global Do-It...Marc Dusseiller Dusjagr
 
Incoming and Outgoing Shipments in 1 STEP Using Odoo 17
Incoming and Outgoing Shipments in 1 STEP Using Odoo 17Incoming and Outgoing Shipments in 1 STEP Using Odoo 17
Incoming and Outgoing Shipments in 1 STEP Using Odoo 17Celine George
 
Grant Readiness 101 TechSoup and Remy Consulting
Grant Readiness 101 TechSoup and Remy ConsultingGrant Readiness 101 TechSoup and Remy Consulting
Grant Readiness 101 TechSoup and Remy ConsultingTechSoup
 
Kisan Call Centre - To harness potential of ICT in Agriculture by answer farm...
Kisan Call Centre - To harness potential of ICT in Agriculture by answer farm...Kisan Call Centre - To harness potential of ICT in Agriculture by answer farm...
Kisan Call Centre - To harness potential of ICT in Agriculture by answer farm...Krashi Coaching
 
SOCIAL AND HISTORICAL CONTEXT - LFTVD.pptx
SOCIAL AND HISTORICAL CONTEXT - LFTVD.pptxSOCIAL AND HISTORICAL CONTEXT - LFTVD.pptx
SOCIAL AND HISTORICAL CONTEXT - LFTVD.pptxiammrhaywood
 
A Critique of the Proposed National Education Policy Reform
A Critique of the Proposed National Education Policy ReformA Critique of the Proposed National Education Policy Reform
A Critique of the Proposed National Education Policy ReformChameera Dedduwage
 

Dernier (20)

18-04-UA_REPORT_MEDIALITERAСY_INDEX-DM_23-1-final-eng.pdf
18-04-UA_REPORT_MEDIALITERAСY_INDEX-DM_23-1-final-eng.pdf18-04-UA_REPORT_MEDIALITERAСY_INDEX-DM_23-1-final-eng.pdf
18-04-UA_REPORT_MEDIALITERAСY_INDEX-DM_23-1-final-eng.pdf
 
Arihant handbook biology for class 11 .pdf
Arihant handbook biology for class 11 .pdfArihant handbook biology for class 11 .pdf
Arihant handbook biology for class 11 .pdf
 
Alper Gobel In Media Res Media Component
Alper Gobel In Media Res Media ComponentAlper Gobel In Media Res Media Component
Alper Gobel In Media Res Media Component
 
Introduction to ArtificiaI Intelligence in Higher Education
Introduction to ArtificiaI Intelligence in Higher EducationIntroduction to ArtificiaI Intelligence in Higher Education
Introduction to ArtificiaI Intelligence in Higher Education
 
Código Creativo y Arte de Software | Unidad 1
Código Creativo y Arte de Software | Unidad 1Código Creativo y Arte de Software | Unidad 1
Código Creativo y Arte de Software | Unidad 1
 
POINT- BIOCHEMISTRY SEM 2 ENZYMES UNIT 5.pptx
POINT- BIOCHEMISTRY SEM 2 ENZYMES UNIT 5.pptxPOINT- BIOCHEMISTRY SEM 2 ENZYMES UNIT 5.pptx
POINT- BIOCHEMISTRY SEM 2 ENZYMES UNIT 5.pptx
 
Concept of Vouching. B.Com(Hons) /B.Compdf
Concept of Vouching. B.Com(Hons) /B.CompdfConcept of Vouching. B.Com(Hons) /B.Compdf
Concept of Vouching. B.Com(Hons) /B.Compdf
 
The basics of sentences session 2pptx copy.pptx
The basics of sentences session 2pptx copy.pptxThe basics of sentences session 2pptx copy.pptx
The basics of sentences session 2pptx copy.pptx
 
Sanyam Choudhary Chemistry practical.pdf
Sanyam Choudhary Chemistry practical.pdfSanyam Choudhary Chemistry practical.pdf
Sanyam Choudhary Chemistry practical.pdf
 
Measures of Central Tendency: Mean, Median and Mode
Measures of Central Tendency: Mean, Median and ModeMeasures of Central Tendency: Mean, Median and Mode
Measures of Central Tendency: Mean, Median and Mode
 
Organic Name Reactions for the students and aspirants of Chemistry12th.pptx
Organic Name Reactions  for the students and aspirants of Chemistry12th.pptxOrganic Name Reactions  for the students and aspirants of Chemistry12th.pptx
Organic Name Reactions for the students and aspirants of Chemistry12th.pptx
 
microwave assisted reaction. General introduction
microwave assisted reaction. General introductionmicrowave assisted reaction. General introduction
microwave assisted reaction. General introduction
 
CARE OF CHILD IN INCUBATOR..........pptx
CARE OF CHILD IN INCUBATOR..........pptxCARE OF CHILD IN INCUBATOR..........pptx
CARE OF CHILD IN INCUBATOR..........pptx
 
“Oh GOSH! Reflecting on Hackteria's Collaborative Practices in a Global Do-It...
“Oh GOSH! Reflecting on Hackteria's Collaborative Practices in a Global Do-It...“Oh GOSH! Reflecting on Hackteria's Collaborative Practices in a Global Do-It...
“Oh GOSH! Reflecting on Hackteria's Collaborative Practices in a Global Do-It...
 
Incoming and Outgoing Shipments in 1 STEP Using Odoo 17
Incoming and Outgoing Shipments in 1 STEP Using Odoo 17Incoming and Outgoing Shipments in 1 STEP Using Odoo 17
Incoming and Outgoing Shipments in 1 STEP Using Odoo 17
 
Grant Readiness 101 TechSoup and Remy Consulting
Grant Readiness 101 TechSoup and Remy ConsultingGrant Readiness 101 TechSoup and Remy Consulting
Grant Readiness 101 TechSoup and Remy Consulting
 
Kisan Call Centre - To harness potential of ICT in Agriculture by answer farm...
Kisan Call Centre - To harness potential of ICT in Agriculture by answer farm...Kisan Call Centre - To harness potential of ICT in Agriculture by answer farm...
Kisan Call Centre - To harness potential of ICT in Agriculture by answer farm...
 
SOCIAL AND HISTORICAL CONTEXT - LFTVD.pptx
SOCIAL AND HISTORICAL CONTEXT - LFTVD.pptxSOCIAL AND HISTORICAL CONTEXT - LFTVD.pptx
SOCIAL AND HISTORICAL CONTEXT - LFTVD.pptx
 
A Critique of the Proposed National Education Policy Reform
A Critique of the Proposed National Education Policy ReformA Critique of the Proposed National Education Policy Reform
A Critique of the Proposed National Education Policy Reform
 
TataKelola dan KamSiber Kecerdasan Buatan v022.pdf
TataKelola dan KamSiber Kecerdasan Buatan v022.pdfTataKelola dan KamSiber Kecerdasan Buatan v022.pdf
TataKelola dan KamSiber Kecerdasan Buatan v022.pdf
 

GOOGLE BIGTABLE

  • 1.
  • 3. Introduction  Data model  Building Blocks  Implementation  Refinements  Performance Evaluation  Real applications  Conclusion
  • 4. Scale is too large for most commercial Databases  Cost would be very high  Low-level storage optimizations help performance significantly  Hard to map semi-structured data to relational database  Non-uniform fields makes it difficult to insert/query data
  • 5. BigTable is a distributed storage system for managing structured data  Bigtable does not support a full relational data model  Scalable  Self managing
  • 6. Used by more than 60 Google products  Google Analytics  Google Finance  Personalized Search  Google Documents  Google Earth  Google Fusion Tables  …  Used for variety of demanding workloads  Throughput oriented batch processing  Latency sensitive data serving
  • 7. Goals  Wide applicability  Scalability  High performance  High availability  Simple data model that supports dynamic control over data layout and format
  • 8. A Bigtable is a sparse, distributed, persistent multidimensional sorted map.  The map is indexed by a row key, column key, and a timestamp. (row:string, column:string, time:int64) → string Webtable
  • 9. The row keys in a table are arbitrary strings.  Data is maintained in lexicographic order by row key  Each row range is called a tablet, which is the unit of distribution and load balancing. row
  • 10. Column keys are grouped into sets called column families.  Data stored in a column family is usually of the same type  A column key is named using the syntax: family : qualifier.  Column family names must be printable , but qualifiers may be arbitrary strings. columns
  • 11.  Each cell in a Bigtable can contain multiple versions of the same data  Versions are indexed by 64-bit integer timestamps  Timestamps can be assigned: ▪ automatically by Bigtable , or ▪ explicitly by client applications timestamp
  • 12. The Bigtable API provides functions :  Creating and deleting tables and column families.  Changing cluster , table and column family metadata.  Support for single row transactions  Allows cells to be used as integer counters  Client supplied scripts can be executed in the address space of servers
  • 13. Bigtable is built on several other pieces of Google infrastructure.  Google File system(GFS)  SSTable : Data structure for storage Index(block ranges) 64K 64K 64K ………. block block block  Chubby: Distributed lock service.
  • 14. Three major components  Library linked into every client  Single master server ▪ Assigning tablets to tablet servers ▪ Detecting addition and expiration of tablet servers ▪ Balancing tablet-server load ▪ Garbage collection files in GFS  Many tablet servers ▪ Manages a set of tablets ▪ Tablet servers handle read and write requests to its table ▪ Splits tablets that have grown too large
  • 15. ▪ Clients communicates directly with tablet servers for read/write  Each table consists of a set of tablets  Initially, each table have just one tablet  Tablets are automatically split as the table grows  Row size can be arbitrary (hundreds of GB)
  • 16. Three level hierarchy  Level 2: Root tablet contains the location of METADATA tablets  Level 3: Each METADATA tablet contains the location of user tablets  Level 1: Chubby file containing location of the root tablet ▪ Location of tablet is stored under a row key that encodes table identifier and its end row
  • 17. Tablet server startup  It creates and acquires an exclusive lock on , a uniquely named file on Chubby.  Master monitors this directory to discover tablet servers.  Tablet server stops serving tablets  If it loses its exclusive lock.  Tries to reacquire the lock on its file as long as the file still exists.  If file no longer exists, the tablet server will never be able to serve again.
  • 18. Master server startup  Grabs unique master lock in Chubby.  Scans the tablet server directory in Chubby.  Communicates with every live tablet server  Scans METADATA table to learn set of tablets.  Master is responsible for finding when tablet server is no longer serving its tablets and reassigning those tablets as soon as possible.  Periodically asks each tablet server for the status of its lock  If no reply, master tries to acquire the lock itself  If successful to acquire lock, then tablet server is either dead or having network trouble
  • 19. Updates committed to a commit log  Recently committed updates are stored in memory –memtable  Older updates are stored in a sequence of SSTables.
  • 20. Write operation  Read operation  Server checks if it is well-formed  Check well-formedness of request.  Checks if the sender is  Check authorization in Chubby file authorized  Merge memtable and SSTables to  Write to commit log find data  After commit, contents are  Return data. inserted into Memtable
  • 21. In order to control size of memtable, tablet log, and SSTable files, “compaction” is used. 1. Minor Compaction.- Move data from memtable to SSTable. 2. Merging Compaction. - Merge multiple SSTables and memtable to a single SSTable. 3. Major Compaction. - that re-writes all SSTables into exactly one SSTable
  • 22. Locality groups  Clients can group multiple column families together into a locality group.  Compression  Compression applied to each SSTable block separately  Uses Bentley and McIlroy's scheme and fast compression algorithm  Bloom filters  Reduce the number of disk accesses
  • 23. Caching  Scan Cache: a high-level cache that caches key-value pairs returned by the SSTable interface  Block Cache: a lower-level cache that caches SSTable blocks read from file system  Commit-log implementation  Suppose one log per tablet rather have one log per tablet server
  • 24.
  • 25.
  • 26. Satisfies goals of high-availability, high-performance, massively scalable data storage. -It has been successfully deployed in real apps (Personalized Search, Orkut, GoogleMaps, …)  Successfully used by various Google products (>60).  It’s a distributed systems, designed to store enormous volumes of data.  Thesesystems can be easily scaled to accommodate peta-bytes of data across thousands of nodes.
  • 27. Bigtable: A Distributed Storage System for Structured Data by Fay Chang, Jeffrey Dean, Sanjay Ghemawat, Wilson C. Hsieh, Deborah A. Wallach, Mike Burrows, Tushar Chandra, Andrew Fikes, Robert E. Gruber  http://glinden.blogspot.com/2006/08/google-bigtable-paper.html  Dean, J. 2005. BigTable: A Distributed Structured Storage System. University of Washington CSE Colloquia (Oct. 2005). http://www.uwtv.org/programs/displayevent.aspx?rID=4188

Notes de l'éditeur

  1. The persistent state of a tablet is stored in GFS
  2. When memtable reaches thresholdFrozen memtable is converted to an SSTableSSTable written to file systemGoalsReduce memory usage of the tablet serverReduce the amount of data to read from commit log during recoveryMerging compactionReads the contents of a few SSTable and memtableWrites new SSTable