(Or, building better UX / Apps with distributed databases and data synchronisation techniques).
This was my talk at Cocoaheads Berlin 17th February 2016.
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
Data sync on iOS with Couchbase Mobile
1. Data Sync on iOS with
Couchbase Mobile
(Or, building better UX / Apps with distributed
databases and data synchronisation techniques)
http://ti.eng.br @thiago_sjc
Thiago Faria Alencar
February 17, 2016HERE Maps @ Berlin
2. What do you mean with User Experience anyway ?
Combination of technology to delivery the best user experience.
That’s why people switch / opt to services: think about
Blacklane, to Immobile scout, Soundcloud.. What can you offer
at the end of the day? Is it the best possible?
6. Let’s go back in time
< 1935 2007
Instant coffee for internal
Brazilian market.. let’s build
a system for sales …
Product’s website at:
www.cafesaojose.com.br
15. Ups, inside subway now..
1 min later.. hm.. couldn’t we show the past/found route?
16. Other examples
Spotify - offline advertising!
‘Hobby' projects (telemetry)
IoT
Even the most ‘connected' applications will
experience network issues from time to time -> make
them operate offline as much as possible
23. “A distributed database is a database in which portions of
the database are stored on multiple computers within a
network.”
24. “Users have access to the portion of the database at their
location so that they can access the data relevant to their
tasks without interfering with the work of others.”
Turns out, a similar set of features needed on of distributed
databases (computer clusters) is desired to have at the end
nodes (mobile devices) > Same fundamental design decisions.
26. CAP Theorem : set of basic requirements that
describe any distributed system (not just
storage/database systems).
Consistency - All the servers in the system will have the same
data so anyone using the system will get the same copy
regardless of which server answers their request.
27. CAP Theorem
Availability - The system will always respond to a request
(even if it's not the latest data or consistent across the system or
just a message saying the system isn't working).
28. CAP Theorem
Partition Tolerance - The system continues to operate as
a whole even if individual servers fail or can't be reached.
30. Document / NoSql databases vs. relational databases
RDMS are more or less ‘all or nothing’, transaction-based, atomic
operations oriented.
Distributed databases relaxes this ‘atomic' requirement in order to provide
better scalability. It turns out this feature is useful for offline mobile apps as
well as we’ll see when compared to relation traditional databases.
31. Document / NoSql databases vs. relational databases
‘NoSQL databases' improves scaling by relaxing
consistency in favor of availability and partition tolerance.
32. A bit or market research..
Find the right tool to do the job
33. Intro to Couchbase
• Open source
• Cross-platform (really)
• Open protocols
• Backed by company, but small ‘lock-in’
• Easy to get started
• Small foot-print on client
• Built-in sync handling
• No ‘migration-hell’ (JSON docs)
36. Instead, share a single database
Compare this to a “repository” in git: each subset of data
would be like a “repo”.. not all users will have access to all
data (e.g. ACL with gitolite)
37. The ‘traditional' way of exchanging data..
Encapsulate all of it in a SSL connection and everything will
be fine (that is, while your connection works).
38. What is happening here?
Notice how in the request – response model, data is “routed” by the different
API end-points: URI path will trigger a specific request handler in the server.
39. How such problems are solved during sync?
• Authentication
• Authorization
• Data validation / routing
40. How such problems are solved during sync?
• Data routing:
When another user connects to the same backend, we want only the
documents belonging to him to be synchronised. The term used for this in
Couchbase Mobile is data replication.
41. How such issues are resolved during sync?
That’s when Couchbase data orchestration features kicks in:
Not to depend on an active connection, what each data represents and
the respective routing are described in the document’s content itself.
-Optimal solution sits neither at one or the other end of the spectrum -
normally you need a mix of things : so normal http reqs still useful!
43. For iOS, called
‘Couchbase Lite’
-Available for many other
platforms.
-Advantages of being a
JSON database (no
migration hell)
-Sync gateway ‘like the git
merge'
44. Document: the primary entity stored in a database
is called a document instead of a “row” or “record”.
Views: ‘denormalised tables/indexes ’; dynamically
generated from JSON objects
Queries: the action of looking up results from a
view’s index
Some terminology in Couchbase Lite land..
(Roots in Map/Reduce programming model)
45. 1
2
3
4
5
6
CBLManager *manager = [CBLManager sharedInstance];
NSError *error;
CBLDatabase* _database = [manager databaseNamed: @"conektapp_db" error: &error];
if (error) { NSLog(@"error getting database %@",error); }
NSDictionary* properties = @{@"user_id": @"joe",
@"type": @"contactInfo",
@"contactData": @“+49610234192"};
CBLDocument* document = [database documentWithID:
[NSString stringWithFormat: @"%@:%@", type, user_id]];
NSError* error;
if (![document putProperties: properties error: &error]) {
[self handleError: error];
}
Saving an object in Couchbase Lite
( For questions on this see more detailed articles at http://ti.eng.br )
46. 1
2
3
4
5
6
- (CBLQuery*) queryContactInfoFromUsername:(NSString*) user
{
//1- createView
CBLView * contactInfoView = [self.database viewNamed:
@"contactDataByUsername"];
[contactInfoView setMapBlock: MAPBLOCK({
if ([doc[@"type"] isEqualToString: @"contactInfo"]) {
if (doc[@"user_id"])
emit(doc[@"user_id"], doc[@"contactData"]);
}
}) version: @"2"];
}
Retrieve an object in Couchbase Lite
47. 1
2
3
4
5
6
- (CBLQuery*) queryContactInfoFromUsername:(NSString*) user
{
//1- createView
CBLView * contactInfoView = [self.database viewNamed:
@"contactDataByUsername"];
[contactInfoView setMapBlock: MAPBLOCK({
if ([doc[@"type"] isEqualToString: @"contactInfo"]) {
if (doc[@"user_id"])
emit(doc[@"user_id"], doc[@"contactData"]);
}
}) version: @"2"];
//2 - make the query
CBLQuery* query = [contactInfoView createQuery];
NSLog(@"Querying username: %@", user);
query.startKey = user;
query.endKey = user;
return query;
}
Retrieve an object in Couchbase Lite
50. The server-side
Sync gateway configuration
Located in the sync gateway, the 'sync function' is responsible for
document authorisation, validation, routing.
51. The server-side
Sync gateway configuration
1
2
3
4
5
6
7
8
9
10
11
12
{
"interface":":4984",
"adminInterface":":4985",
"log":["REST"],
"databases":{
"sync_gateway":{
"server":"http://localhost:8091",
"bucket":"app_bucket",
"sync":`function(doc) {channel(doc.channels);}`
}
}
}
sync function, not much defined for now.. will
grow according to your application’s needs
53. More on the sync function..
function(doc, oldDoc)
{
requireUser(oldDoc.owner);
channel(doc.channel);
access(doc.members, doc.roomID);
}
Validation
54. More on the sync function..
function(doc, oldDoc)
{
requireUser(oldDoc.owner);
channel(doc.channel);
access(doc.members, doc.roomID);
}
Routing
55. More on the sync function..
function(doc, oldDoc)
{
requireUser(oldDoc.owner);
channel(doc.channel);
access(doc.members, doc.roomID);
}
Access
Control
56. About the concept of “Channels”
App server tags Documents on entry; Docs are indexed by channel for
replication.
57. Summary
Data scalability: Replicate only docs relevante to
respective user.
Access Control: Replicate only documents visible to
this user
Data Validation: Replicate only documents that
contains valid information
62. Thank you :)
Finally we have a way to automate data
synchronisation!! Please use it! :)
And let’s shift our designs / apps away from the old "LAMP style”
into something that actually works in real life scenarios also when
'on the go'..
64. One more slide to show how a more ‘elaborated'
sync function may look like..
65. function (doc, oldDoc) {
if (doc._deleted) {
// Only editors with write access can delete documents:
requireRole("role:editor");
requireUser(oldDoc.writers);
// Skip other validation because a deletion has no other properties:
return;
}
// Required properties:
if (!doc.title || !doc.creator || !doc.channels || !doc.writers) {
throw({forbidden: "Missing required properties"});
} else if (doc.writers.length == 0) {
throw({forbidden: "No writers"});
}
if (oldDoc == null) {
// Only editors can create documents:
requireRole("role:editor");
// The 'creator' property must match the user creating the document:
requireUser(doc.creator)
} else {
// Only users in the existing doc's writers list can change a document:
requireUser(oldDoc.writers);
// The "creator" property is immutable:
if (doc.creator != oldDoc.creator) {
throw({forbidden: "Can't change creator"});
}
}
// Finally, assign the document to the channels in the list:
channel(doc.channels);
}
Notes de l'éditeur
(more details can be found in my blog, Mobile Couchbase section)
-Thanks to Event Host, Organizers, audience, and Jens (couchbase)
-how many heard of… couchbase , distributed database?
-'Root thing'
Combination of technology to delivery an user experience. That’s why people switch services: think about blacklane, to immobile scout, to soundcloud. What can you offer at the end of the day? Is it the best possible?
-'Root thing'
Combination of technology to delivery an user experience. That’s why people switch services: think about blacklane, to immobile scout, to soundcloud. What can you offer at the end of the day? Is it the best possible?
Story behind it (why - people still struggle with this problem, motivation, belief of making things better)
-For years to come
-Mind the ‘unconnected' connected applications
-Data / database core to any application
All about improving UX > better design ideas lead to better systems (how) > transfer the idea to you.
Mention Git analogy.
Same set of problems of distributed databases in clusters. Same fundamental design decisions.
Small introduction to Distributed Databases.
CAP theorem : set of basic requirements that describe any distributed system (not just storage/database systems).
It's theoretically impossible to have all 3 requirements met, so a combination of 2 must be chosen and this is usually the deciding factor in what technology is used.
DD relaxes this requirement in order to provide better scalability. It turns out this feature is useful for offline mobile apps as well as we’ll see when compared to relation traditional databases.
RDMS are more or less ‘all or nothing’, transaction-based, atomic operations oriented.
DD relaxes this requirement in order to provide better scalability. It turns out this feature is useful for offline mobile apps as well as we’ll see when compared to relation traditional databases.
RDMS are more or less ‘all or nothing’, transaction-based, atomic operations oriented.
-Parse,
-Firebase (google)
-Cloudkit (?) not really.. what about the other platforms?
-Encourage to Research yourself..
Operate directly on local database. And ‘sync’ it, just like a git push/pull.
Not limited to mobile devices (smartphones), but can be also used in embedded computers which eventually get disconnected but need to continue operating.
Compare this to a “repository” in git.
Encapsulate all of it in a SSL connection and everything will be fine (that is, while your connection works).
-Authentication
-Authorization
-Data validation / routing : Notice how in the request – response model, data is “routed” by the different API end-points: URI path will trigger a specific request handler in the server.
the term used for this in Couchbase Mobile is data replication. When another user connects to the same backend, we want only the documents belonging to him to be synchronised, and the term used for this is data routing.
the term used for this in Couchbase Mobile is data replication. When another user connects to the same backend, we want only the documents belonging to him to be synchronised, and the term used for this is data routing.
That’s when Couchbase data orchestration support kicks in.
-not to depend on an active connection, what each data represents and the respective routing are described in the document’s content itself
-Neither at one or the other end of the spectrum, normal reqs still useful!
-Compare to git (how come we didn’t do it before?)
That’s when Couchbase data orchestration support kicks in.
-not to depend on an active connection, what each data represents and the respective routing are described in the document’s content itself
-Neither at one or the other end of the spectrum, normal reqs still useful!
-Compare to git (how come we didn’t do it before?)
-Available for many other platforms.
-Advantages of being a JSON database (no migration hell)
-Sync gateway ‘like the git merge'
(before going into details..)
-a mention about N1QL Couchbase server
Note the sync function:
Responsible for document authorisation, validation, routing. We will implement / explain this in step 6, for now we left here the 'default' which does no verification.
Note the sync function:
Responsible for document authorisation, validation, routing. We will implement / explain this in step 6, for now we left here the 'default' which does no verification.
App server tags Documents on entry; Docs are indexed by channel for replication.
-And let’s shift our designs / apps away from the old "LAMP style"