SlideShare une entreprise Scribd logo
1  sur  26
Télécharger pour lire hors ligne
Moma-Django
Overview
Django Boston meetup, 02-27-2014
Django + MongoDB:
building a custom ORM layer
Overview of the talk:
moma-django is a MongoDB manager for Django. It provides
native Django ORM support for MongoDB documents,
including the query API and the admin interface. It was
developed as a part of two commercial products and released
as an open source.
In the talk we will review the motivation behind its
developments, its features and go through 2-3 examples of
how to use some of the features: migrating an existing model,
advanced queries and the admin interface. If time permits we
will discuss unit testing and south migrations
Who are we?
 Company: Cloudoscope.com
 What we do:
– Cloudoscope’s product enable IT vendors to automate the presales process by collecting and analyzing prospect IT
performance
– Previous product - Lucidel: B2C marketing analytics based on
website data
– Data intensive projects / sites, NoSQL, analytics focus
(as a way of funding)

 Gadi Oren:

@gadioren,

gadioren
Why moma-django?
 Certain problems can be addressed well with NoSQL
 The team wants to experiment with a NoSQL
HOWEVER:
 A lot of code needs to be rewritten
 Team  learn a new API
 Some of the tools and procedures are no longer functioning
and should be replaced
– Admin interface
– Unit testing environment

 Some of the data need to be somewhat de-normalized*
Why moma-django? (our example)
 Needed a very efficient way of processing timeseries
 The timeseries where constantly growing
 We required very detailed search/slice/dice capabilities to
find the timeseries to be processed
 Some of the data was optional (e.g. demographics
information was never complete)
 Document size, content and structure varied widely
 However, we have a small distributed team and we did not
want to create a massive project
 We started experimenting using a stub Manager doing small
iterations, adding functionality as we needed over nine
months
Other packages
 PyMongo – a dependency for moma-django
 MongoEngine – somewhat similar concepts in terms of
models

 Non relational versions of Django
“Native” - advantages
 Django packages and plugins (e.g. Admin functionality)
 Using similar code conventions
 Easier to bring in new team members
 Use the same unit testing frameworks (e.g. Jenkins)

 Simple experimentation and migration path
Let’s make it interactive
Questions Anyone??? (Example Application)
 Small question asking application
 Allows voting and adding images
 Implemented as a django application over MongoDB, using
moma-django

 Register and login at http://momadjango.org
 Ask away!
Migrating an existing model
class TstBook(models.Model):
name = models.CharField(max_length=64)
publish_date = MongoDateTimeField()
author = models.ForeignKey('testing.TstAuthor')

class Meta:
unique_together = ['name', 'author']
class TstAuthor(models.Model):
first_name = models.CharField(max_length=32)
last_name = models.CharField(max_length=32)

class TstBook(MongoModel):
name = models.CharField(max_length=64)
publish_date = MongoDateTimeField()
author = models.ForeignKey('testing.TstAuthor')

class Meta:
unique_together = ['name', 'author']
class TstAuthor(MongoModel):
first_name = models.CharField(max_length=32)
last_name = models.CharField(max_length=32)

models.signals.post_syncdb.connect(post_syncdb_
mongo_handler)
Migrating an existing model (2)
 Syncdb:

 Add objects
Migrating an existing model (2)
 Syncdb:

 Add objects
>>> TstBook(name=“Good night half moon”, publish_date=datetime.datetime(2014,2,20),
author=TstAuthor.objects.get(first_name=“Gadi”)).save()
Migrating an existing model (3)
 Breaching uniqueness  try and save the same object again:
Migrating an existing model (4)
 In Mongo: content, indexes

class Meta:
unique_together = ['name', 'author']

 Admin
New field types
 MongoIDField – Internal. Used to hold the MongoDB object
ID
 MongoDateTimeField – Used for Datetime
 ValuesField – Used to represent a list of objects of any type
 StringListField – Used for a list of strings
 DictionaryField – Used as a dictionary
 Current limitation: nested structures have limited support
Queries and update – 1: bulk insert
records.append(
{ "_id" : ObjectId("502abdabf7f16836f100285a"), "time_on_site" : 290,
"user_id" : 1154449631, "account_id" : NumberLong(5), "campaign" : "(not set)",
"first_visit_date" : ISODate("2012-07-30T17:10:06Z"), "referral_path" : "(not set)",
"source" : "google", "exit_page_path" : "/some-analysis/lion-king/", "landing_page_path" : "(not set)",
"keyword" : "wikipedia lion king", "date" : ISODate("2012-07-30T00:00:00Z"),
"visit_count" : 1, "page_views" : 3,
"visit_id" : "false---------------1154449631.1343668206",
"goal_values" : { }, "goal_starts" : { }, "demographics" : { }, "goal_completions" : { },
"location" : { "cr" : "United States", "rg" : "California", "ct" : "Pasadena" },
})

UniqueVisit.objects.filter(account__in=self.list_of_accounts).delete()
UniqueVisit.objects.bulk_insert( records )
Queries and update – 2: examples
def ISODate(timestr):
res = datetime.strptime(timestr, "%Y-%m-%dT%H:%M:%SZ")
res = res.replace(tzinfo=timezone.utc)
return res
# Datetime
qs = UniqueVisit.objects.filter( first_visit_date__lte =ISODate("2012-07-30T12:29:05Z"))
self.assertEqual( qs.query.spec, dict(
# pymongo expression
{'first_visit_date': {'$lte': datetime(2012, 7, 30, 12, 29, 5, tzinfo=timezone.utc)}})
)
# Multiple conditions
qs = UniqueVisit.objects.filter( first_visit_date__lte =ISODate("2012-07-30T12:29:05Z"),
time_on_site__gt =10,
page_views__gt =2)
self.assertEqual( qs.query.spec, dict(
# pymongo expression
{'time_on_site': {'$gt': 10.0},
'page_views': {'$gt': 2},
'first_visit_date': {'$lte': datetime(2012, 7, 30, 12, 29, 5, tzinfo=timezone.utc)}}
))
Queries and update– 3: examples
# Different query optimizations
qs = UniqueVisit.objects.filter(Q(time_on_site =10)|Q(time_on_site =25)|Q(time_on_site =275))
self.assertEqual( qs.query.spec, dict(
# pymongo expression
{'time_on_site': {'$in': [10.0, 25.0, 275.0]}}
))
# Multiple or Q expressions
qs = UniqueVisit.objects.filter(Q(time_on_site =10)|Q(time_on_site =25)|Q(time_on_site =275)|Q(source = 'bing'))
self.assertEqual( qs.query.spec, dict(
# pymongo expression
{'$or': [{'time_on_site': 10.0}, {'time_on_site': 25.0}, {'time_on_site': 275.0}, {'source': 'bing'}]}
))
# Negate Q
qs = UniqueVisit.objects.filter(~Q(first_visit_date =ISODate("2012-07-30T12:29:05Z")))
self.assertEqual( qs.query.spec, dict(
# pymongo expression
{'first_visit_date': {'$ne': datetime(2012, 7, 30, 12, 29, 5, tzinfo=timezone.utc)}}
))
Queries – 4: extensions beyond standard Django
# Dot notation
qs = UniqueVisit.objects.filter(location__rg__exact ="New York")
self.assertEqual( qs.query.spec, dict((
# pymongo expression
{'location.rg': 'New York'}
))
# Check key existence
qs = UniqueVisit.objects.filter(demographics__age__exists ="true")
self.assertEqual( qs.query.spec, dict((
# pymongo expression
{'demographics.age': {'$exists': 'true'}}
))
# variable type
qs = UniqueVisit.objects.filter(landing_page_path__type = int)
self.assertEqual( qs.query.spec, dict((
# pymongo expression
{'landing_page_path': {'$type': 16}}
))
Queries - by the structure of documents
# How many documents in the DB?
>>> UniqueVisit.objects.all().count()
20
>>> # For how many documents in the DB do we have age information?
>>> UniqueVisit.objects.filter(demographics__age__exists ="true").count()
7
>>> # For how many documents in the DB do we have gender information?
>>> UniqueVisit.objects.filter(demographics__gender__exists ="true").count()
3
>>> # For how many documents in the DB do we have gender and age information?
>>> UniqueVisit.objects.filter(demographics__age__exists ="true“,
demographics__gender__exists ="true").count()
1
>>>
Manipulating documents payload
# Model
class Question(MongoModel):

user = models.ForeignKey(User)
date = MongoDateTimeField(db_index=True)
question = models.CharField(max_length=256 )

docs = DictionaryField(models.CharField())
image = DictionaryField(models.TextField())
audio = DictionaryField()
other = DictionaryField()
vote_ids = ValuesField(models.IntegerField())
def __unicode__(self):
return u'%s[%s %s]' % (self.question, self.date,
self.user, )
class Meta:
unique_together = ['user', 'question',]

# Store an image: get the image from the “POST” upload form (snippet)
docfile = request.FILES['docfile']
question_id = form.cleaned_data['question_id']
docfile_name = docfile.name
docfile_name_changed = _replace_dots(docfile.name)
question = Question.objects.get(id=question_id)

# Store meta-data
question.docs.update({docfile_name_changed : docfile.content_type})
question.image.update(
{docfile_name_changed +'_url' : '/static/display/s_'+docfile_name,
docfile_name_changed +'_name' : docfile_name,
docfile_name_changed +'_content_type' : docfile.content_type})
# Store the actual image binary block (small scale implementation)
file_read = docfile.file.read() # Note – this is a naïve implementation!
file_data = base64.b64encode(file_read)
question.image.update({docfile_name_changed +'_data' : file_data})
question.save()
Admin interface

So – what’s next?
 Github: https://github.com/gadio/moma-django
 If you want to contribute – please contact (forking is also an
option)
 Contact: gadi.oren.1 at gmail.com or
gadi at Cloudoscope.com
Backup
South
 Dealing with apps with mixed models  South to disregard
the model
# Enabling South for the non conventional mongo model
add_introspection_rules(
[
(
(MongoIdField, MongoDateTimeField, DictionaryField ),
[],
{
"max_length": ["max_length", {"default": None}],
},
),
],
["^moma_django.fields.*",])
Unit testing
 The model name is defined in settings.py
 In unit testing run, a new mongo DB schema is created
MONGO_COLLECTION prefixed with “test_”(e.g.
test_momaexample)
MONGO_HOST = 'localhost'
MONGO_PORT = 27017
MONGO_COLLECTION = 'momaexample'
Moma-django on google…

Contenu connexe

Tendances

Internet and Web Technology (CLASS-14) [JSP] | NIC/NIELIT Web Technology
Internet and Web Technology (CLASS-14) [JSP] | NIC/NIELIT Web Technology Internet and Web Technology (CLASS-14) [JSP] | NIC/NIELIT Web Technology
Internet and Web Technology (CLASS-14) [JSP] | NIC/NIELIT Web Technology Ayes Chinmay
 
MongoDB and Ruby on Rails
MongoDB and Ruby on RailsMongoDB and Ruby on Rails
MongoDB and Ruby on Railsrfischer20
 
Internet and Web Technology (CLASS-7) [XML and AJAX] | NIC/NIELIT Web Technology
Internet and Web Technology (CLASS-7) [XML and AJAX] | NIC/NIELIT Web TechnologyInternet and Web Technology (CLASS-7) [XML and AJAX] | NIC/NIELIT Web Technology
Internet and Web Technology (CLASS-7) [XML and AJAX] | NIC/NIELIT Web TechnologyAyes Chinmay
 
Internet and Web Technology (CLASS-8) [jQuery and JSON] | NIC/NIELIT Web Tech...
Internet and Web Technology (CLASS-8) [jQuery and JSON] | NIC/NIELIT Web Tech...Internet and Web Technology (CLASS-8) [jQuery and JSON] | NIC/NIELIT Web Tech...
Internet and Web Technology (CLASS-8) [jQuery and JSON] | NIC/NIELIT Web Tech...Ayes Chinmay
 
JSONSchema with golang
JSONSchema with golangJSONSchema with golang
JSONSchema with golangSuraj Deshmukh
 
Getting started with MongoDB and Scala - Open Source Bridge 2012
Getting started with MongoDB and Scala - Open Source Bridge 2012Getting started with MongoDB and Scala - Open Source Bridge 2012
Getting started with MongoDB and Scala - Open Source Bridge 2012sullis
 
MongoDB at ZPUGDC
MongoDB at ZPUGDCMongoDB at ZPUGDC
MongoDB at ZPUGDCMike Dirolf
 
Internet and Web Technology (CLASS-6) [BOM]
Internet and Web Technology (CLASS-6) [BOM] Internet and Web Technology (CLASS-6) [BOM]
Internet and Web Technology (CLASS-6) [BOM] Ayes Chinmay
 
Using Mongoid with Ruby on Rails
Using Mongoid with Ruby on RailsUsing Mongoid with Ruby on Rails
Using Mongoid with Ruby on RailsNicholas Altobelli
 
Practical Use of MongoDB for Node.js
Practical Use of MongoDB for Node.jsPractical Use of MongoDB for Node.js
Practical Use of MongoDB for Node.jsasync_io
 
Web components
Web componentsWeb components
Web componentsMohd Saeed
 
Mongoose: MongoDB object modelling for Node.js
Mongoose: MongoDB object modelling for Node.jsMongoose: MongoDB object modelling for Node.js
Mongoose: MongoDB object modelling for Node.jsYuriy Bogomolov
 
MongoDB 3.2 - a giant leap. What’s new?
MongoDB 3.2 - a giant leap. What’s new?MongoDB 3.2 - a giant leap. What’s new?
MongoDB 3.2 - a giant leap. What’s new?Binary Studio
 
02.03.21 Collaborator.pro Webinar Решение 10 главных задач технической оптими...
02.03.21 Collaborator.pro Webinar Решение 10 главных задач технической оптими...02.03.21 Collaborator.pro Webinar Решение 10 главных задач технической оптими...
02.03.21 Collaborator.pro Webinar Решение 10 главных задач технической оптими...Vladislav Morgun
 
Comparing JSON Libraries - July 19 2011
Comparing JSON Libraries - July 19 2011Comparing JSON Libraries - July 19 2011
Comparing JSON Libraries - July 19 2011sullis
 
Canonical and robotos (2)
Canonical and robotos (2)Canonical and robotos (2)
Canonical and robotos (2)panchaloha
 

Tendances (20)

Internet and Web Technology (CLASS-14) [JSP] | NIC/NIELIT Web Technology
Internet and Web Technology (CLASS-14) [JSP] | NIC/NIELIT Web Technology Internet and Web Technology (CLASS-14) [JSP] | NIC/NIELIT Web Technology
Internet and Web Technology (CLASS-14) [JSP] | NIC/NIELIT Web Technology
 
MongoDB and Ruby on Rails
MongoDB and Ruby on RailsMongoDB and Ruby on Rails
MongoDB and Ruby on Rails
 
Internet and Web Technology (CLASS-7) [XML and AJAX] | NIC/NIELIT Web Technology
Internet and Web Technology (CLASS-7) [XML and AJAX] | NIC/NIELIT Web TechnologyInternet and Web Technology (CLASS-7) [XML and AJAX] | NIC/NIELIT Web Technology
Internet and Web Technology (CLASS-7) [XML and AJAX] | NIC/NIELIT Web Technology
 
Internet and Web Technology (CLASS-8) [jQuery and JSON] | NIC/NIELIT Web Tech...
Internet and Web Technology (CLASS-8) [jQuery and JSON] | NIC/NIELIT Web Tech...Internet and Web Technology (CLASS-8) [jQuery and JSON] | NIC/NIELIT Web Tech...
Internet and Web Technology (CLASS-8) [jQuery and JSON] | NIC/NIELIT Web Tech...
 
JSONSchema with golang
JSONSchema with golangJSONSchema with golang
JSONSchema with golang
 
JS basics
JS basicsJS basics
JS basics
 
Suggest.js
Suggest.jsSuggest.js
Suggest.js
 
Getting started with MongoDB and Scala - Open Source Bridge 2012
Getting started with MongoDB and Scala - Open Source Bridge 2012Getting started with MongoDB and Scala - Open Source Bridge 2012
Getting started with MongoDB and Scala - Open Source Bridge 2012
 
ActiveRecord vs Mongoid
ActiveRecord vs MongoidActiveRecord vs Mongoid
ActiveRecord vs Mongoid
 
MongoDB at ZPUGDC
MongoDB at ZPUGDCMongoDB at ZPUGDC
MongoDB at ZPUGDC
 
Internet and Web Technology (CLASS-6) [BOM]
Internet and Web Technology (CLASS-6) [BOM] Internet and Web Technology (CLASS-6) [BOM]
Internet and Web Technology (CLASS-6) [BOM]
 
Using Mongoid with Ruby on Rails
Using Mongoid with Ruby on RailsUsing Mongoid with Ruby on Rails
Using Mongoid with Ruby on Rails
 
Practical Use of MongoDB for Node.js
Practical Use of MongoDB for Node.jsPractical Use of MongoDB for Node.js
Practical Use of MongoDB for Node.js
 
Introduction to JSON & AJAX
Introduction to JSON & AJAXIntroduction to JSON & AJAX
Introduction to JSON & AJAX
 
Web components
Web componentsWeb components
Web components
 
Mongoose: MongoDB object modelling for Node.js
Mongoose: MongoDB object modelling for Node.jsMongoose: MongoDB object modelling for Node.js
Mongoose: MongoDB object modelling for Node.js
 
MongoDB 3.2 - a giant leap. What’s new?
MongoDB 3.2 - a giant leap. What’s new?MongoDB 3.2 - a giant leap. What’s new?
MongoDB 3.2 - a giant leap. What’s new?
 
02.03.21 Collaborator.pro Webinar Решение 10 главных задач технической оптими...
02.03.21 Collaborator.pro Webinar Решение 10 главных задач технической оптими...02.03.21 Collaborator.pro Webinar Решение 10 главных задач технической оптими...
02.03.21 Collaborator.pro Webinar Решение 10 главных задач технической оптими...
 
Comparing JSON Libraries - July 19 2011
Comparing JSON Libraries - July 19 2011Comparing JSON Libraries - July 19 2011
Comparing JSON Libraries - July 19 2011
 
Canonical and robotos (2)
Canonical and robotos (2)Canonical and robotos (2)
Canonical and robotos (2)
 

Similaire à moma-django overview --> Django + MongoDB: building a custom ORM layer

MongoDB Tick Data Presentation
MongoDB Tick Data PresentationMongoDB Tick Data Presentation
MongoDB Tick Data PresentationMongoDB
 
MongoDB hearts Django? (Django NYC)
MongoDB hearts Django? (Django NYC)MongoDB hearts Django? (Django NYC)
MongoDB hearts Django? (Django NYC)Mike Dirolf
 
Back to Basics, webinar 2: La tua prima applicazione MongoDB
Back to Basics, webinar 2: La tua prima applicazione MongoDBBack to Basics, webinar 2: La tua prima applicazione MongoDB
Back to Basics, webinar 2: La tua prima applicazione MongoDBMongoDB
 
Back to Basics 2017: Mí primera aplicación MongoDB
Back to Basics 2017: Mí primera aplicación MongoDBBack to Basics 2017: Mí primera aplicación MongoDB
Back to Basics 2017: Mí primera aplicación MongoDBMongoDB
 
MongoDB Schema Design: Practical Applications and Implications
MongoDB Schema Design: Practical Applications and ImplicationsMongoDB Schema Design: Practical Applications and Implications
MongoDB Schema Design: Practical Applications and ImplicationsMongoDB
 
Building LinkedIn's Learning Platform with MongoDB
Building LinkedIn's Learning Platform with MongoDBBuilding LinkedIn's Learning Platform with MongoDB
Building LinkedIn's Learning Platform with MongoDBJake Dejno
 
MongoDB .local London 2019: Fast Machine Learning Development with MongoDB
MongoDB .local London 2019: Fast Machine Learning Development with MongoDBMongoDB .local London 2019: Fast Machine Learning Development with MongoDB
MongoDB .local London 2019: Fast Machine Learning Development with MongoDBLisa Roth, PMP
 
MongoDB .local London 2019: Fast Machine Learning Development with MongoDB
MongoDB .local London 2019: Fast Machine Learning Development with MongoDBMongoDB .local London 2019: Fast Machine Learning Development with MongoDB
MongoDB .local London 2019: Fast Machine Learning Development with MongoDBMongoDB
 
Streaming Data Pipelines with MongoDB and Kafka at ao.com
Streaming Data Pipelines with MongoDB and Kafka at ao.comStreaming Data Pipelines with MongoDB and Kafka at ao.com
Streaming Data Pipelines with MongoDB and Kafka at ao.comMongoDB
 
MongoDB in the Middle of a Hybrid Cloud and Polyglot Persistence Architecture
MongoDB in the Middle of a Hybrid Cloud and Polyglot Persistence ArchitectureMongoDB in the Middle of a Hybrid Cloud and Polyglot Persistence Architecture
MongoDB in the Middle of a Hybrid Cloud and Polyglot Persistence ArchitectureMongoDB
 
Building your first app with MongoDB
Building your first app with MongoDBBuilding your first app with MongoDB
Building your first app with MongoDBNorberto Leite
 
MongoDB@sfr.fr
MongoDB@sfr.frMongoDB@sfr.fr
MongoDB@sfr.frbeboutou
 
Supercharging your Organic CTR
Supercharging your Organic CTRSupercharging your Organic CTR
Supercharging your Organic CTRPhil Pearce
 
Back to Basics Webinar 2 - Your First MongoDB Application
Back to  Basics Webinar 2 - Your First MongoDB ApplicationBack to  Basics Webinar 2 - Your First MongoDB Application
Back to Basics Webinar 2 - Your First MongoDB ApplicationJoe Drumgoole
 
Back to Basics Webinar 2: Your First MongoDB Application
Back to Basics Webinar 2: Your First MongoDB ApplicationBack to Basics Webinar 2: Your First MongoDB Application
Back to Basics Webinar 2: Your First MongoDB ApplicationMongoDB
 
Building an enterprise Natural Language Search Engine with ElasticSearch and ...
Building an enterprise Natural Language Search Engine with ElasticSearch and ...Building an enterprise Natural Language Search Engine with ElasticSearch and ...
Building an enterprise Natural Language Search Engine with ElasticSearch and ...Debmalya Biswas
 
CCCDjango2010.pdf
CCCDjango2010.pdfCCCDjango2010.pdf
CCCDjango2010.pdfjayarao21
 
Mongoose and MongoDB 101
Mongoose and MongoDB 101Mongoose and MongoDB 101
Mongoose and MongoDB 101Will Button
 

Similaire à moma-django overview --> Django + MongoDB: building a custom ORM layer (20)

MongoDB Tick Data Presentation
MongoDB Tick Data PresentationMongoDB Tick Data Presentation
MongoDB Tick Data Presentation
 
MongoDB hearts Django? (Django NYC)
MongoDB hearts Django? (Django NYC)MongoDB hearts Django? (Django NYC)
MongoDB hearts Django? (Django NYC)
 
Back to Basics, webinar 2: La tua prima applicazione MongoDB
Back to Basics, webinar 2: La tua prima applicazione MongoDBBack to Basics, webinar 2: La tua prima applicazione MongoDB
Back to Basics, webinar 2: La tua prima applicazione MongoDB
 
Back to Basics 2017: Mí primera aplicación MongoDB
Back to Basics 2017: Mí primera aplicación MongoDBBack to Basics 2017: Mí primera aplicación MongoDB
Back to Basics 2017: Mí primera aplicación MongoDB
 
MongoDB Schema Design: Practical Applications and Implications
MongoDB Schema Design: Practical Applications and ImplicationsMongoDB Schema Design: Practical Applications and Implications
MongoDB Schema Design: Practical Applications and Implications
 
Building LinkedIn's Learning Platform with MongoDB
Building LinkedIn's Learning Platform with MongoDBBuilding LinkedIn's Learning Platform with MongoDB
Building LinkedIn's Learning Platform with MongoDB
 
MongoDB .local London 2019: Fast Machine Learning Development with MongoDB
MongoDB .local London 2019: Fast Machine Learning Development with MongoDBMongoDB .local London 2019: Fast Machine Learning Development with MongoDB
MongoDB .local London 2019: Fast Machine Learning Development with MongoDB
 
MongoDB .local London 2019: Fast Machine Learning Development with MongoDB
MongoDB .local London 2019: Fast Machine Learning Development with MongoDBMongoDB .local London 2019: Fast Machine Learning Development with MongoDB
MongoDB .local London 2019: Fast Machine Learning Development with MongoDB
 
Streaming Data Pipelines with MongoDB and Kafka at ao.com
Streaming Data Pipelines with MongoDB and Kafka at ao.comStreaming Data Pipelines with MongoDB and Kafka at ao.com
Streaming Data Pipelines with MongoDB and Kafka at ao.com
 
MongoDB in the Middle of a Hybrid Cloud and Polyglot Persistence Architecture
MongoDB in the Middle of a Hybrid Cloud and Polyglot Persistence ArchitectureMongoDB in the Middle of a Hybrid Cloud and Polyglot Persistence Architecture
MongoDB in the Middle of a Hybrid Cloud and Polyglot Persistence Architecture
 
Building your first app with MongoDB
Building your first app with MongoDBBuilding your first app with MongoDB
Building your first app with MongoDB
 
A Brief MongoDB Intro
A Brief MongoDB IntroA Brief MongoDB Intro
A Brief MongoDB Intro
 
MongoDB@sfr.fr
MongoDB@sfr.frMongoDB@sfr.fr
MongoDB@sfr.fr
 
Supercharging your Organic CTR
Supercharging your Organic CTRSupercharging your Organic CTR
Supercharging your Organic CTR
 
Back to Basics Webinar 2 - Your First MongoDB Application
Back to  Basics Webinar 2 - Your First MongoDB ApplicationBack to  Basics Webinar 2 - Your First MongoDB Application
Back to Basics Webinar 2 - Your First MongoDB Application
 
Back to Basics Webinar 2: Your First MongoDB Application
Back to Basics Webinar 2: Your First MongoDB ApplicationBack to Basics Webinar 2: Your First MongoDB Application
Back to Basics Webinar 2: Your First MongoDB Application
 
Building an enterprise Natural Language Search Engine with ElasticSearch and ...
Building an enterprise Natural Language Search Engine with ElasticSearch and ...Building an enterprise Natural Language Search Engine with ElasticSearch and ...
Building an enterprise Natural Language Search Engine with ElasticSearch and ...
 
CCCDjango2010.pdf
CCCDjango2010.pdfCCCDjango2010.pdf
CCCDjango2010.pdf
 
MongoDB
MongoDBMongoDB
MongoDB
 
Mongoose and MongoDB 101
Mongoose and MongoDB 101Mongoose and MongoDB 101
Mongoose and MongoDB 101
 

Dernier

How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?IES VE
 
Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxUdaiappa Ramachandran
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationIES VE
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostMatt Ray
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Will Schroeder
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...Aggregage
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxMatsuo Lab
 
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDELiveplex
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6DianaGray10
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfDianaGray10
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesMd Hossain Ali
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaborationbruanjhuli
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemAsko Soukka
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsSeth Reyes
 
AI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarAI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarPrecisely
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfDaniel Santiago Silva Capera
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024SkyPlanner
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAshyamraj55
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Websitedgelyza
 

Dernier (20)

How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?
 
Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptx
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptx
 
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystem
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and Hazards
 
AI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarAI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity Webinar
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
 
20230104 - machine vision
20230104 - machine vision20230104 - machine vision
20230104 - machine vision
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Website
 

moma-django overview --> Django + MongoDB: building a custom ORM layer

  • 2. Django + MongoDB: building a custom ORM layer Overview of the talk: moma-django is a MongoDB manager for Django. It provides native Django ORM support for MongoDB documents, including the query API and the admin interface. It was developed as a part of two commercial products and released as an open source. In the talk we will review the motivation behind its developments, its features and go through 2-3 examples of how to use some of the features: migrating an existing model, advanced queries and the admin interface. If time permits we will discuss unit testing and south migrations
  • 3. Who are we?  Company: Cloudoscope.com  What we do: – Cloudoscope’s product enable IT vendors to automate the presales process by collecting and analyzing prospect IT performance – Previous product - Lucidel: B2C marketing analytics based on website data – Data intensive projects / sites, NoSQL, analytics focus (as a way of funding)  Gadi Oren: @gadioren, gadioren
  • 4. Why moma-django?  Certain problems can be addressed well with NoSQL  The team wants to experiment with a NoSQL HOWEVER:  A lot of code needs to be rewritten  Team  learn a new API  Some of the tools and procedures are no longer functioning and should be replaced – Admin interface – Unit testing environment  Some of the data need to be somewhat de-normalized*
  • 5. Why moma-django? (our example)  Needed a very efficient way of processing timeseries  The timeseries where constantly growing  We required very detailed search/slice/dice capabilities to find the timeseries to be processed  Some of the data was optional (e.g. demographics information was never complete)  Document size, content and structure varied widely  However, we have a small distributed team and we did not want to create a massive project  We started experimenting using a stub Manager doing small iterations, adding functionality as we needed over nine months
  • 6. Other packages  PyMongo – a dependency for moma-django  MongoEngine – somewhat similar concepts in terms of models  Non relational versions of Django
  • 7. “Native” - advantages  Django packages and plugins (e.g. Admin functionality)  Using similar code conventions  Easier to bring in new team members  Use the same unit testing frameworks (e.g. Jenkins)  Simple experimentation and migration path
  • 8. Let’s make it interactive Questions Anyone??? (Example Application)  Small question asking application  Allows voting and adding images  Implemented as a django application over MongoDB, using moma-django  Register and login at http://momadjango.org  Ask away!
  • 9. Migrating an existing model class TstBook(models.Model): name = models.CharField(max_length=64) publish_date = MongoDateTimeField() author = models.ForeignKey('testing.TstAuthor') class Meta: unique_together = ['name', 'author'] class TstAuthor(models.Model): first_name = models.CharField(max_length=32) last_name = models.CharField(max_length=32) class TstBook(MongoModel): name = models.CharField(max_length=64) publish_date = MongoDateTimeField() author = models.ForeignKey('testing.TstAuthor') class Meta: unique_together = ['name', 'author'] class TstAuthor(MongoModel): first_name = models.CharField(max_length=32) last_name = models.CharField(max_length=32) models.signals.post_syncdb.connect(post_syncdb_ mongo_handler)
  • 10. Migrating an existing model (2)  Syncdb:  Add objects
  • 11. Migrating an existing model (2)  Syncdb:  Add objects >>> TstBook(name=“Good night half moon”, publish_date=datetime.datetime(2014,2,20), author=TstAuthor.objects.get(first_name=“Gadi”)).save()
  • 12. Migrating an existing model (3)  Breaching uniqueness  try and save the same object again:
  • 13. Migrating an existing model (4)  In Mongo: content, indexes class Meta: unique_together = ['name', 'author']  Admin
  • 14. New field types  MongoIDField – Internal. Used to hold the MongoDB object ID  MongoDateTimeField – Used for Datetime  ValuesField – Used to represent a list of objects of any type  StringListField – Used for a list of strings  DictionaryField – Used as a dictionary  Current limitation: nested structures have limited support
  • 15. Queries and update – 1: bulk insert records.append( { "_id" : ObjectId("502abdabf7f16836f100285a"), "time_on_site" : 290, "user_id" : 1154449631, "account_id" : NumberLong(5), "campaign" : "(not set)", "first_visit_date" : ISODate("2012-07-30T17:10:06Z"), "referral_path" : "(not set)", "source" : "google", "exit_page_path" : "/some-analysis/lion-king/", "landing_page_path" : "(not set)", "keyword" : "wikipedia lion king", "date" : ISODate("2012-07-30T00:00:00Z"), "visit_count" : 1, "page_views" : 3, "visit_id" : "false---------------1154449631.1343668206", "goal_values" : { }, "goal_starts" : { }, "demographics" : { }, "goal_completions" : { }, "location" : { "cr" : "United States", "rg" : "California", "ct" : "Pasadena" }, }) UniqueVisit.objects.filter(account__in=self.list_of_accounts).delete() UniqueVisit.objects.bulk_insert( records )
  • 16. Queries and update – 2: examples def ISODate(timestr): res = datetime.strptime(timestr, "%Y-%m-%dT%H:%M:%SZ") res = res.replace(tzinfo=timezone.utc) return res # Datetime qs = UniqueVisit.objects.filter( first_visit_date__lte =ISODate("2012-07-30T12:29:05Z")) self.assertEqual( qs.query.spec, dict( # pymongo expression {'first_visit_date': {'$lte': datetime(2012, 7, 30, 12, 29, 5, tzinfo=timezone.utc)}}) ) # Multiple conditions qs = UniqueVisit.objects.filter( first_visit_date__lte =ISODate("2012-07-30T12:29:05Z"), time_on_site__gt =10, page_views__gt =2) self.assertEqual( qs.query.spec, dict( # pymongo expression {'time_on_site': {'$gt': 10.0}, 'page_views': {'$gt': 2}, 'first_visit_date': {'$lte': datetime(2012, 7, 30, 12, 29, 5, tzinfo=timezone.utc)}} ))
  • 17. Queries and update– 3: examples # Different query optimizations qs = UniqueVisit.objects.filter(Q(time_on_site =10)|Q(time_on_site =25)|Q(time_on_site =275)) self.assertEqual( qs.query.spec, dict( # pymongo expression {'time_on_site': {'$in': [10.0, 25.0, 275.0]}} )) # Multiple or Q expressions qs = UniqueVisit.objects.filter(Q(time_on_site =10)|Q(time_on_site =25)|Q(time_on_site =275)|Q(source = 'bing')) self.assertEqual( qs.query.spec, dict( # pymongo expression {'$or': [{'time_on_site': 10.0}, {'time_on_site': 25.0}, {'time_on_site': 275.0}, {'source': 'bing'}]} )) # Negate Q qs = UniqueVisit.objects.filter(~Q(first_visit_date =ISODate("2012-07-30T12:29:05Z"))) self.assertEqual( qs.query.spec, dict( # pymongo expression {'first_visit_date': {'$ne': datetime(2012, 7, 30, 12, 29, 5, tzinfo=timezone.utc)}} ))
  • 18. Queries – 4: extensions beyond standard Django # Dot notation qs = UniqueVisit.objects.filter(location__rg__exact ="New York") self.assertEqual( qs.query.spec, dict(( # pymongo expression {'location.rg': 'New York'} )) # Check key existence qs = UniqueVisit.objects.filter(demographics__age__exists ="true") self.assertEqual( qs.query.spec, dict(( # pymongo expression {'demographics.age': {'$exists': 'true'}} )) # variable type qs = UniqueVisit.objects.filter(landing_page_path__type = int) self.assertEqual( qs.query.spec, dict(( # pymongo expression {'landing_page_path': {'$type': 16}} ))
  • 19. Queries - by the structure of documents # How many documents in the DB? >>> UniqueVisit.objects.all().count() 20 >>> # For how many documents in the DB do we have age information? >>> UniqueVisit.objects.filter(demographics__age__exists ="true").count() 7 >>> # For how many documents in the DB do we have gender information? >>> UniqueVisit.objects.filter(demographics__gender__exists ="true").count() 3 >>> # For how many documents in the DB do we have gender and age information? >>> UniqueVisit.objects.filter(demographics__age__exists ="true“, demographics__gender__exists ="true").count() 1 >>>
  • 20. Manipulating documents payload # Model class Question(MongoModel):  user = models.ForeignKey(User) date = MongoDateTimeField(db_index=True) question = models.CharField(max_length=256 ) docs = DictionaryField(models.CharField()) image = DictionaryField(models.TextField()) audio = DictionaryField() other = DictionaryField() vote_ids = ValuesField(models.IntegerField()) def __unicode__(self): return u'%s[%s %s]' % (self.question, self.date, self.user, ) class Meta: unique_together = ['user', 'question',] # Store an image: get the image from the “POST” upload form (snippet) docfile = request.FILES['docfile'] question_id = form.cleaned_data['question_id'] docfile_name = docfile.name docfile_name_changed = _replace_dots(docfile.name) question = Question.objects.get(id=question_id) # Store meta-data question.docs.update({docfile_name_changed : docfile.content_type}) question.image.update( {docfile_name_changed +'_url' : '/static/display/s_'+docfile_name, docfile_name_changed +'_name' : docfile_name, docfile_name_changed +'_content_type' : docfile.content_type}) # Store the actual image binary block (small scale implementation) file_read = docfile.file.read() # Note – this is a naïve implementation! file_data = base64.b64encode(file_read) question.image.update({docfile_name_changed +'_data' : file_data}) question.save()
  • 22. So – what’s next?  Github: https://github.com/gadio/moma-django  If you want to contribute – please contact (forking is also an option)  Contact: gadi.oren.1 at gmail.com or gadi at Cloudoscope.com
  • 24. South  Dealing with apps with mixed models  South to disregard the model # Enabling South for the non conventional mongo model add_introspection_rules( [ ( (MongoIdField, MongoDateTimeField, DictionaryField ), [], { "max_length": ["max_length", {"default": None}], }, ), ], ["^moma_django.fields.*",])
  • 25. Unit testing  The model name is defined in settings.py  In unit testing run, a new mongo DB schema is created MONGO_COLLECTION prefixed with “test_”(e.g. test_momaexample) MONGO_HOST = 'localhost' MONGO_PORT = 27017 MONGO_COLLECTION = 'momaexample'