This document discusses the development of a backend system using Apache Thrift and MongoDB. The developers describe using Thrift for its code generation, serialization/deserialization, and high performance capabilities. MongoDB is used for its document storage and compatibility with PaaS platforms. A ThriftMongoBridge is created to allow Thrift objects to be stored in MongoDB. Core principles for the backend include scalability, adaptability, testability, use of NoSQL, and productivity. Design choices like Thrift and MongoDB are discussed. The document provides examples of using the ThriftMongoBridge in testing code and updating data. Performance tests show it outperforming alternatives like Spring Data. Future plans include integrating the libraries and ensuring service reliability in cloud deployments.

1. By @FinistSeb and @LostInBrittany

2. Sebastien Lambour
Le développeur de l'Est le plus à l'Ouest !
Java & C++ Developer, coding addict,
continuous intégration ayatollah
● Senior Software Engineer
currently, Crédit Mutuel Arkea
● FinistJug actif member
BreizhBeans community and commiter
sebastien.lambour@gmail.com
+sebastien.lambour
@FinistSeb

3. Horacio Gonzalez
Spaniard lost in Brittany, Java developer,
dreamer and all-around geek
● Senior Software Engineer at Crédit Mutuel Arkea
○ Currently...
● Leader of the FinistJUG and the
BreizhBeans community
http://lostinbrittany.org/
+Horacio.Gonzalez
@LostInBrittany

4. BreizhBeans
Developer community at the far end
of French Brittany
http://breizhbeans.org/
@BreizhBeans

5. Why ?
I help to create a startup
on social translation, would
you code with me ?
Great, I do the backend!
How many members?
Some 30.000 - 50.000
will be fine!
Ouch, is it not too small?
I like big toys!!!!
Nexus 4
Me too!
Let's think big then
OK, it will rock !

6. Backends
They often look like that !

7. Transforming it is HARD !
Next challenge :
Granite aircraft !!

8. A sexy backend?
Let's groom it!
Light weight
Data Driven
Java
Cloud Ready
OK, go around...
Test
driven
Scalable

9. Following the beaten track
Brainless route
JSF 2.0
Hibernate
Spring CRUD
(Create, Read, Update,Delete)
SQL
JPA
CXF
The usual suspects...

10. Booooooring...
And ineffective
Let's use it!

11. Design choices
Because we need some solid roots

12. Core principles
What do we want our backend to be?
●
Scalability
○
●
Adaptability
○
●
Because you want to be sure that your code works!
NoSQL (meaning Not Only SQL)
○
●
Because your business evolves, and your backend need to evolve
with it!
Testability
○
●
Because traffic growth should be a blessing, not a problem!
Because you need not only tables and foreign keys, but non-SQL
stores, search indexes, caches...
Productivity
○
Because the other 4 doesn't matter if you need to spend 4
working days to modify a service...

13. A first choice :
Apache Thrift
« Software framework for scalable and performant crosslanguage services development »
●
●
●
●
●
Interface description language
Multi-language code generation
Serialisation/deserialisation
RPC Framework
High performance
○ On transport layer
○ On data manipulation

14. Thrift, it's cool
Thrift boots productivity!

15. A second choice :
MongoDB
● We needed a NO-SQL store for documents
● We wanted to use Cassandra
○ Cassandra & Thrift, a love story
● We wanted a PaaS-ready backend
○ Neither Cloudbees nor Cloudfoundry supported Cassandra
○ But they supported MongoDB
● We loved MongoDB
○ Document DB are great for document oriented applications

16. ThriftMongoBridge
Because we wanted both
Thrift and MongoDB

17. Thrift rocks and works
But...
How to put Thrift objects into MongoDB ?

18. First bad idea!
Use Spring Data or Morphia
Thrift object mapping is bad !

19. Damned!
First bad idea!

20. Second bad idea !
Use the Mongo Driver JSON parsing
Deserialisation will be a bit hard...

21. Damned !
Second bad idea!

22. Thrift isn't cool anymore ?
We need a new idea !

23. Thrift, it's cool !
But we had work to do...
Write a TBSONProtocol !
The ThriftMongoBridge is born !

24. What does it do ?
POJO generated by the Thrift compiler
public class AnotherThrift implements org.apache.thrift.TBase... {
...
public String anotherString;
public int anotherInteger;
Thrift IDL
struct AnotherThrift {
1:string anotherString,
2:i32
anotherInteger,
}
Compile to
...
public static final Map<_Fields, org.apache.thrift.meta_data.
FieldMetaData> metaDataMap;
....
public void read(org.apache.thrift.protocol.TProtocol iprot)...
...
public void write(org.apache.thrift.protocol.TProtocol oprot) ......;
}
TBSonSerializer
TBSonProtocol
Write To BSON
TBSonDeSerializer
Read from BSON
{anotherString:"value", anotherInteger:69}

25. Really simple to use
Too simple ?
AnotherThrift anotherThrift = new AnotherThrift();
anotherThrift.setAnotherString("value");
anotherThrift.setAnotherInteger(69);
// serialize into DBObject
DBObject dbObject = ThriftMongoHelper.thrift2DBObject(anotherThrift);
{anotherString:"value", anotherInteger:69}

26. How it works
write(protocol)
Thrift object generated
TBSONSerializer
TBase
getResult
sentense directives
(begin / end)
push
Data
(read / write)
pop
Struct context
DBObject
Context Stack
TBSONProtocol
DBObject (BSON)
MongoDB object

27. The BreizhCamp demo
Because you want to see some code,
don't you?

28. The BreizhCamp demo
Given this Thrift model & service
struct BreizhCampEvent {
1:i64 date,
2:string startTime,
3:string endTime,
4:com.breizhbeans.backend.thrift.enums.model.EventTypes type,
5:string room,
6:string resume,
7:string details,
8:list<string> speakers
9:list<com.breizhbeans.backend.thrift.enums.model.Track> tracks,
}
service ProgramService {
com.....GetProgramResponse getProgram(1:com.....GetProgramRequest request)
throws (1:com....TechnicalException texp, 2:com...FunctionalException fexp)
}
struct GetProgramRequest {
1:com.breizhbeans.backend.thrift.enums.model.Track track,
2:com.breizhbeans.backend.thrift.enums.model.EventTypes type,
3:string room,
4:list<string> speakers,
}
struct GetProgramResponse {
1:list<com.breizhbeans.backend.thrift.model.BreizhCampEvent> events,
}

29. The BreizhCamp demo
First : Write a test
● One loader by collection
● One goal : Have Human readable test ;-)
● Manage automatically data insertion before test and
clean up after test
Application code to test
@Test
public void testGetEvents() throws Exception {
List<BreizhCampEvent> actualEvents = programDao.getEvent();
List<BreizhCampEvent> expectedEvents = new ArrayList<BreizhCampEvent>();
expectedEvents.add(getEventPimpMyBackend());
Assert.assertEquals(expectedEvents, actualEvents);
}
Expected data building
(used by the test and the
data loader)
Standard JUnit assertion

30. The BreizhCamp demo
Write a DAO
public List<BreizhCampEvent> getEvent() throws TechnicalException {
DBCursor cursor = null;
try {
List<BreizhCampEvent> events = new ArrayList<BreizhCampEvent>();
DBCollection collection = db.getCollection(collectionName);
DBObject query = new BasicDBObject();
// execute the query
retrieve DB collection
cursor = collection.find(query);
while(cursor.hasNext()) {
Build and execute the Query
events.add( (BreizhCampEvent) ThriftMongoHelper.DBObject2Thrift(cursor.next()));
}
return events;
} catch (Exception e) {
throw new H4TTechnicalException("Unexpected error", e, null);
Deserialize Thrift objects
} finally {
cursor.close();
}
Error handling
}
Never forget it ;-)

31. The BreizhCamp demo
Can I update thrift Object by code?
@Override
public BreizhCampEvent updateRoom(final BreizhCampEvent event) throws TechnicalException {
try {
BasicDBObject updateQuery = new BasicDBObject();
Mongo Update Query
updateQuery.append("$set", new BasicDBObject()
.append(getFields(BreizhCampEvent._Fields.ROOM), event.getRoom()));
db.getCollection(collectionName)
.update(getPrimaryKeyRequest(event), updateQuery)
.getLastError()
Thrift structure mapping
.throwOnError();
return event;
} catch (Exception e) {
Query execution
throw new H4TTechnicalException("Unexpected error", e, ObjectUtils.toString(event));
}
}
Error handling

32. The BreizhCamp demo
Can I update thrift Object by script?
Mongo DB Bson writed : > db.program.find();
{ "_id" : "id=10", "date" : NumberLong("-56433978000000"), "startTime" :
"14:45", "endTime" : "15:40", "type" : 4, "room" : "Bréhat", "resume" :
"Pimp my backend!", "details" : "Le frontend, le frontend......
...", "speakers" : [ "Sébastien Lambour", "Horacio Gonzalez" ], "tracks"
: [ 7 ], "id" : 10, "classname" : "com.breizhbeans.backend.thrift.model.
BreizhCampEvent" }
Update the field room :> db.program.update();
{ "id": 10}, { $set: { "room": "Ouessant" } }

33. And the perfs?
Because speed matters!

34. Talking about perfs
The candidates are ...
Spring Data
Morphia
And...
ThriftMongoBridge, of course

35. Spring Data vs
ThriftMongoBridge
● Write and Read 500 objects into MongoDB.
● Each document is composed with a List<> and a Set<>
of 500 objects.
● The document have a JSON size of 91613 bytes
Who will win ?

36. Code
ThriftMongoBridge
First
// Get a collection
DBCollection collection = db.getCollection("dummyColl");
Write
// Serialize the Thrift object
DBObject dbObject = ThriftMongoHelper.thrift2DBObject(thriftObject);
// Put the document
collection.insert(dbObject);
Read
DBCursor cursorDoc = collection.find();
while (cursorDoc.hasNext()) {
DBObject dbObject = cursorDoc.next();
ThriftObject thirftObject = (ThriftObject)ThriftMongoHelper.DBObject2Thrift(dbObject);
}

37. Code
Spring Data
First
// Retrieve the MongoOperations bean
MongoOperations mongoOperation =
(MongoOperations) ctx.getBean("mongoTemplate");
Write
// Serialize the Thrift object
mongoOperation.save(springObject, "springObject");
Read
List<SpringObject> listUser =
mongoOperation.findAll(SpringObject.class, "springObject");

38. Time to run !

39. Thrift over MongoDB rocks!
We need a logo for our OpenSource library...
Available now :
2.0
https://github.com/BreizhBeans/ThriftTools

40. A storage library
a backend is not,
young padawan
Because we need more things

41. At the beginning...
Pretty layers, neatly isolated,
like a Care Bears' birthday cake

42. Later in the project
Like a cake after an earthquake

43. And in the end...
As pretty as a British pudding...
and almost as tasty

44. Be Layer less
Now build the Backend !
le
b
ha
c
rea
un
● Each package have a goal
● NO package assembly rules
● Improve the complexity detection by dependencies
analysis
Goal : Fight against the "God classes" anti-pattern
m
ea
dr

45. Thrift services
Prevent protocol rupture, have a stable services signature.
com.h4t.CheckAccessResponse checkAccess
(1:com.h4t.CheckAccessRequest request) throws
(1:com.h4t.TechnicalException texp, 2:com.h4t.FunctionalException fexp)
stable method signature

46. Integrate this into
the backend
Thrift IFace implementation
Request / Response decapsulation + log & monitoring
Core Business code
Thrift object
read
write
ThriftMongoHelper
Mongo
Opérations
Query
DBObject
DBObject
Mongo Driver

47. Inversion of control
Keep simple too !
Prefer static dependencies injection.
Time to do the choice
or

48. Inversion of control
Why
Because
?
and
○ binding of the MongoDB service by the cloud
○ no production configuration required
○ only one war

49. Testability
Because without tests
you need faith

50. Test strategy
No compromise!
The rules are simple... no exception!

51. TDD is my religion
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
Execute test without setup
Execute test with your build tool
Execute test into your IDE
No injection into test framework
Reuse only application injection
Keep it simple (repeat it 7 times)
Have a loose coupling (test the smallest parts)
Test must be deterministic and reproducible
Do pair review with the tests
If the complexity is growing up refactor now
...
11. If a rule goes wrong, rewrite it !

52. From concept to reality
Load thrift object
with preset datas
into MongoDB
Get expected
Thift object
with preset
datas
Collection Loader
Thrift DataSet
ThriftMongoTestTools
Get a Thrift object from
MongoDB related to a
unique key

53. Tests have goals
DAO : Validate queries and set up the test data loaders
IFACE : Functional tests
SERVICE : Test a business centric service
TU : Unit test not related to the business (test first
approach)

54. Tests are Data Driven
@Before
public void setUp() throws Exception {
Inject Data into MongoDB
eventLoader.load(getEventPimpMyBackend());
}
private BreizhCampEvent getEventPimpMyBackend() throws Exception {
String details = "Le frontend, le frontend... .....";
BreizhCampEvent event = eventLoader.getExpected(10, "14/06/2013", "14:45", "15:
40", EventTypes.CONF, "Bréhat", "Pimp my backend!", details);
event = eventLoader.addSpeakers(event, "Sébastien Lambour", "Horacio Gonzalez");
event = eventLoader.addTracks(event, Track.CLOUD);
return event;
}
Build test Data with java code, files, BDD
or the MongoThrift Test tools (not
opensourced yet ;-) )

55. What's next?
Because our work is not completed... yet!

56. Why integrate our lib ?
#Pragmatism
●
●
●
●
Productivity oriented
Fully testable
Container less (no servlet dependencies)
Open Source : Apache 2.0
The ThriftMongoBackend Core model will born soon !
With a name: TINAF
(ThriftMongoBackend Is Not A Framework)

57. What's next ?
● monitoring Thrift services in the cloud
● load tests with vmware & 10gen
● contribute TINAF
● Develop a thrift service valve to ensure brandwitdth for
critical services

58. Thank you !