Server discovery and monitoring with MongoDB
•Télécharger en tant que PPTX, PDF•
0 j'aime•382 vues
Overview of how the drivers discover servers and recover from server failures.
Recommandé
Contenu connexe
Tendances
Tendances (12)
En vedette
En vedette (20)
Similaire à Server discovery and monitoring with MongoDB
Similaire à Server discovery and monitoring with MongoDB (20)
Plus de Joe Drumgoole
Plus de Joe Drumgoole (13)
Dernier
Dernier (20)
Server discovery and monitoring with MongoDB
- 1. Server Discovery and Monitoring Joe Drumgoole Director of Developer Advocacy, EMEA 21-July-2016 V1.0
- 2. 2 MongoDB MongoDB Query Language (MQL) + Native Drivers MongoDB Document/JSON Data Model Wired Tiger MMAP Management Security In-memory Encrypted 3rd party Shared Clusters Replica Sets
- 6. 6 Replica Set Primary Failure Driver Secondary Secondary
- 7. 7 Replica Set Election Driver Secondary Secondary
- 8. 8 Replica Set New Primary Driver Primary Secondary
- 9. 9 Replica Set Recovery Driver Primary Secondary Secondary
- 10. 10 Sharded Cluster Driver Mongod Mongod Mongod Mongod Mongod Mongod Mongod Mongod Mongod mongos mongos
- 11. 11 Driver Responsibilities https://github.com/mongodb/mongo-python-driver Driver Authentication & Security Python<->BSON Error handling & Recovery Wire Protocol Topology Management Connection Pool
- 12. 12 Driver Responsibilities https://github.com/mongodb/mongo-python-driver Driver Authentication & Security Python<->BSON Error handling & Recovery Wire Protocol Topology Management Connection Pool
- 13. 13 Example API Calls import pymongo client = pymongo.MongoClient( host=“localhost”, port=27017) database = client[ ‘test_database’ ] collection = database[ ‘test_collection’ ] collection.insert_one({ "hello" : "world" , "goodbye" : "world" } ) collection.find_one( { "hello" : "world" } ) collection.update({ "hello" : "world" }, { "$set" : { "buenos dias" : "world" }} ) collection.delete_one({ "hello" : "world" } )
- 14. 14 Start MongoClient c = MongoClient( "host1, host2", replicaSet="replset" )
- 15. 15 Client Side View Secondary host2 Secondary host3 Primary host1 Mongo Client MongoClient( "host1, host2", replicaSet="replset" )
- 16. 16 Client Side View Secondary host2 Secondary host3 Primary host1 Mongo Client Monitor Thread 1 Monitor Thread 2 { ismaster : False, secondary: True, hosts : [ host1, host2, host3 ] }
- 17. 17 What Does ismaster show? >>> pprint.pprint( db.command( "ismaster" )) {u'hosts': [u'JD10Gen-old.local:27017', u'JD10Gen-old.local:27018', u'JD10Gen-old.local:27019'], u'ismaster' : False, u'secondary': True, u'setName' : u'replset', …} >>>
- 19. 19 Client Side View Secondary host2 Secondary host3 Primary host1 Mongo Client Monitor Thread 1 Monitor Thread 2
- 20. 20 Client Side View Secondary host2 Secondary host3 Primary host1 Mongo Client Monitor Thread 1 Monitor Thread 2 Monitor Thread 3
- 21. 21 Client Side View Secondary host2 Secondary host3 Primary host1 Mongo Client Monitor Thread 1 Monitor Thread 2 Monitor Thread 3 Your Code
- 22. 22 Next Is Insert c = MongoClient( "host1, host2", replicaSet="replset" ) client.db.col.insert_one( { "a" : "b" } )
- 23. 23 Insert Will Block Secondary host2 Secondary host3 Primary host1 Mongo Client Monitor Thread 1 Monitor Thread 2 Monitor Thread 3 Your Code Insert
- 24. 24 ismaster response from Host 1 Secondary host2 Secondary host3 Primary host1 Mongo Client Monitor Thread 1 Monitor Thread 2 Monitor Thread 3 Your Code Insert ismaster
- 25. 25 Now Write Can Proceed Secondary host2 Secondary host3 Primary host1 Mongo Client Monitor Thread 1 Monitor Thread 2 Monitor Thread 3 Your Code Insert Insert
- 26. 26 Later Host 3 Responds Secondary host2 Secondary host3 Primary host1 Mongo Client Monitor Thread 1 Monitor Thread 2 Monitor Thread 3 Your Code
- 27. 27 Steady State Secondary host2 Secondary host3 Primary host1 Mongo Client Monitor Thread 1 Monitor Thread 2 Monitor Thread 3 Your Code
- 28. 28 Life Intervenes Secondary host2 Secondary host3 Primary host1 Mongo Client Monitor Thread 1 Monitor Thread 2 Monitor Thread 3 Your Code ✖
- 29. 29 Monitor may not detect Secondary host2 Secondary host3 Primary host1 Mongo Client Monitor Thread 1 Monitor Thread 2 Monitor Thread 3 Your Code ✖ Insert ConnectionFailure
- 30. 30 So Retry Secondary host2 Secondary host3 Mongo Client Monitor Thread 1 Monitor Thread 2 Monitor Thread 3 Your Code ✖ Insert
- 31. 31 Check for Primary Secondary host2 Secondary host3 Mongo Client Monitor Thread 1 Monitor Thread 2 Monitor Thread 3 Your Code ✖ Insert
- 32. 32 Host 2 Is Primary Primary host2 Secondary host3 Mongo Client Monitor Thread 1 Monitor Thread 2 Monitor Thread 3 Your Code ✖ Insert
- 33. 33 Steady State Secondary host2 Secondary host3 Primary host1 Mongo Client Monitor Thread 1 Monitor Thread 2 Monitor Thread 3 Your Code
- 34. 34 What Does This Mean? - Connect import pymongo client = pymongo.MongoClient() try: client.admin.command( "ismaster" ) except pymongo.errors.ConnectionFailure, e : print( "Cannot connect: %s" % e )
- 35. 35 What Does This Mean? - Queries import pymongo def find_with_recovery( collection, query ) : try: return collection.find_one( query ) except pymongo.errors.ConnectionFailure, e : logging.info( "Connection failure : %s" e ) return collection.find_one( query )
- 36. 36 What Does This Mean? - Inserts def insert_with_recovery( collection, doc ) : doc[ "_id" ] = ObjectId() try: collection.insert_one( doc ) except pymongo.errors.ConnectionFailure, e: logging.info( "Connection error: %s" % e ) try: collection.insert_one( doc ) except DuplicateKeyError: pass
- 37. 37 What Does This Mean? - Updates collection.update( { "_id" : 1 }, { "$inc" : { "counter" : 1 }})
- 39. 39 connectTimeoutMS Secondary host2 Secondary host3 Mongo Client Monitor Thread 1 Monitor Thread 2 Monitor Thread 3 Your Code ✖ Insert connectTimeoutMS
- 40. 40 serverTimeoutMS Secondary host2 Secondary host3 Mongo Client Monitor Thread 1 Monitor Thread 2 Monitor Thread 3 Your Code ✖ Insert serverTimeoutMS
- 41. 41 More Reading • The spec author Jess Jiryu Davis has a collection of links and his better version of this talk https://emptysqua.re/blog/server-discovery-and-monitoring-in-mongodb- drivers/ • The full server discovery and monitoring spec is on GitHub https://github.com/mongodb/specifications/blob/master/source/server- discovery-and-monitoring/server-discovery-and-monitoring.rst
- 43. 43 insert_one • Stages – Parse the parameters – Get a socket to write data on – Add the object Id – Convert the whole insert command and parameters to a SON object – Apply the writeConcern to the command – Encode the message into a BSON object – Send the message to the server via the socket (TCP/IP) – Check for writeErrors (e.g. DuplicateKeyError) – Check for writeConcernErrors (e.g.writeTimeout) – Return Result object
- 44. 44 Bulk Insert bulker = collection.initialize_ordered_bulk_op() bulker.insert( { "a" : "b" } ) bulker.insert( { "c" : "d" } ) bulker.insert( { "e" : "f" } ) try: bulker.execute() except pymongo.errors.BulkWriteError as e : print( "Bulk write error : %s" % e.detail )
- 45. 45 Bulk Write • Create Bulker object • Accumulate operations • Each operation is created as a SON object • The operations are accumulated in a list • Once execute is called – For ordered execute in order added – For unordered execute INSERT, UPDATEs then DELETE • Errors will abort the whole batch unless no write concern specified
Notes de l'éditeur
- Primary secondary, secondary
- Primary secondary, secondary
- Primary secondary, secondary
- Primary secondary, secondary
- Primary secondary, secondary
- Present a native language interface - converts python types to BSON objects Convert the JSON query language into commands for the database Convert JSON data into BSON data and vice-versa Handles interfacing to different MongoDB topologies Helps recover from server side outages/network errors Manages the client side connection pool The pymongo driver code is on Github (Apache License)
- Present a native language interface - converts python types to BSON objects Convert the JSON query language into commands for the database Convert JSON data into BSON data and vice-versa Handles interfacing to different MongoDB topologies Helps recover from server side outages/network errors Manages the client side connection pool The pymongo driver code is on Github (Apache License)
- Calls i
- Calls i
- State machine, full set of states defined in spec.
- Calls i
- Calls i
- Calls i
- Needs a primary to complete a write.
- Needs a primary to complete a write.
- Needs a primary to complete a write.
- Needs a primary to complete a write.
- Each thread wakes every 10 seconds. Runs ismaster, sleeps. We use ismaster to check latency. Keep topology description up to date.
- Each thread wakes every 10 seconds. Runs ismaster, sleeps. We use ismaster to check latency. Keep topology description up to date.
- Each thread wakes every 10 seconds. Runs ismaster, sleeps. We use ismaster to check latency. Keep topology description up to date.
- Primary is marked as unknown Wakes up all monitor threads to check for a primary.
- Primary is marked as unknown Wakes up all monitor threads to check for a primary every half second.
- Primary is marked as unknown Wakes up all monitor threads to check for a primary every half second.
- Each thread wakes every 10 seconds. Runs ismaster, sleeps. We use ismaster to check latency. Keep topology description up to date.
- Try once. This will accomdate elections. Other errore should be propagated.
- Try once. This will accomdate elections. Other errore should be propagated.
- Can you afford to over or under count. Operations need to be idempotent. Turn an update into a write of a document, cf EventSourcing. Then aggregate on the server.
- How long should a connection wait before timing out and sleeping for 10 seconds.
- How long should a connection wait before timing out and sleeping for 10 seconds.