Slides from our presentation about the Lessons Learnt. See more about the event on Meetup.com www.meetup.com/BKK-Developers-and-Tech-lovers/events/222069625/

1. Lessons Learnt from
Backend Systems
Development
by Josef Nevoral @ HotelQuickly

2. About HotelQuickly
● Last minute hotel booking
● iOS, Android and BlackBerry applications
● In 15 countries and more than 200 cities
● 2 years old
● 85+ people in company, 25+ in product team

3. ● 10+ years in software development
● In HotelQuickly for 2+ years
● Many roles in the past - Titanium, iOS,
backend systems
● Currently leading backend systems
development
About me

4. Backend systems
in HQ

5. Backend systems in HQ
● 100+ repositories in github (20+ public)
● 40+ third party services integrated
● 5 programming languages
o php, javascript - NodeJS, python, ruby

6. Lessons learnt
Evolution of our development workflow
From a monolithic app to microservices
Asynchronous processing

7. Evolution of our development workflow
Lesson #1

8. Key moments
11/2012
1st
commit
03/2014
Code reviews
05/2014
Continuous
integration
08/2014
QA
10/2014
Scrum
sprints
10/2014
Retrospectives
01/2015
Kanban
20

9. Key moments
11/2012
1st
commit
03/2014
Code reviews
05/2014
Continuous
integration
08/2014
QA
10/2014
Scrum
sprints
10/2014
Retrospectives
01/2015
Kanban

11. Key moments
11/2012
1st
commit
03/2014
Code reviews
05/2014
Continuous
integration
08/2014
QA
10/2014
Scrum
sprints
10/2014
Retrospectives
01/2015
Kanban

14. Key moments
11/2012
1st
commit
03/2014
Code reviews
05/2014
Continuous
integration
08/2014
QA
10/2014
Scrum
sprints
10/2014
Retrospectives
01/2015
Kanban

16. Key moments
11/2012
1st
commit
03/2014
Code reviews
05/2014
Continuous
integration
08/2014
QA
10/2014
Scrum
sprints
10/2014
Retrospectives
01/2015
Kanban

18. Key moments
11/2012
1st
commit
03/2014
Code reviews
05/2014
Continuous
integration
08/2014
QA
10/2014
Scrum
sprints
10/2014
Retrospectives
01/2015
Kanban
20

20. Key moments
11/2012
1st
commit
03/2014
Code reviews
05/2014
Continuous
integration
08/2014
QA
10/2014
Scrum
sprints
10/2014
Retrospectives
01/2015
Kanban
20

22. Key moments
11/2012
1st
commit
03/2014
Code reviews
05/2014
Continuous
integration
08/2014
QA
10/2014
Scrum
sprints
10/2014
Retrospectives
01/2015
Kanban
20

23. From sprints to Kanban

24. From a monolithic app to microservices
Lesson #2

25. Why have we built a monolithic app?
● fast initial development
● not enough experience with large systems
● simplicity in the beginning for DevOps

26. What problems do we face?
● Huge database (250+ tables)
● A single technology stack
● Scaling
● Technical debt
● Onboarding new joiners Backend
Frontend
Hotels
extranet
Tracking
Payment
engine
Mobile API
Intranet
Notifications
engine
Callbacks

27. Cut the monolith into pieces
Backend === intranet
Frontend
Tracking
Mobile API
Callbacks
Notifications
engine
Payments
engine
Hotel
extranet

28. What microservices bring us
● Single responsibility
● Free hand in technology
● Independent scaling
● Smaller chunks to digest for new joiners
● Reducing technical debt step by step
New challenges

29. Asynchronous
processing
Lesson #3

31. Booking a hotel (synchronous)
API request from mobile app
1. Validate offer (is hotel still available?)
2. Verify that customer is not fraud
3. Charge customer’s credit card
4. Save information about the booking
5. Generate voucher and send confirmation to user (sms, email, push
notification)
6. Issue loyalty points
7. Issue referral program rewards
8. Send confirmation to hotel (email, fax)
9. Send payment to hotel (generate credit card, send email/fax)
10.Return API response to customer

32. Problems
● can be slow
● minor error can
fail booking

33. Booking a hotel (asynchronous)
API request from mobile app
1. Validate offer
(is hotel still available?)
2. Charge customer’s credit
card
3. Save informations about
the booking
4. Return API response to
customer
Processed on background
● Verify that customer is not
fraud
● Generate voucher and
send confirmation to user
● Issue loyalty points
● Issue referral program
rewards
● Send confirmation to hotel
● Send payment to hotel

34. Our solution
● Cron jobs (but it sucks)
o need to keep the state using flags in database
o waiting for cron (min 1 minute)
● Messaging queue system
o fire and forget
o highly scalable with microservices (workers)
o very fast communication
o Moving from Beanstalkd to AWS SQS
(experimenting with RabbitMQ)

35. Asynchronous processing
was not answer to all our performance problems and everything
42but it helped

36. Key takeaways
● become agile
o sprints / kanban
o retrospectives
o daily updates
● have 2 pairs of eyes for everything (not just code)
● think in microservices
● move as much as possible to background processes

37. ● Becoming agile - book about agile best practices
● 12factor App - methodology for building software-as-a-
service apps
● Joel Spolsky test - 12 steps for better code
● codebunk.com - realtime pair coding
● queues.io - list of popular job and message queues
● Top 10 uses of queues
Resources

38. See you at the next HQ meetup
Follow our blog - product.hotelquickly.com
@hotelquickly
@HQProduct
josef.nevoral@hotelquickly.com