2020 brought us massive growth in usage but big challenges came along with it (as they say, a good problem to have!). In this talk, I will share the challenges we encountered and how we responded to them, including the decisions we had to make and the processes we developed.

1. #UXDXAPAC Handling Sudden Growth
Handling Sudden Growth
Kristine Joy Paas (@kjcpaas)
Engineering Manager, Quipper Ltd.
UXDX APAC

2. #UXDXAPAC Handling Sudden Growth
Kristine Joy Paas (Joy)
- Engineering Manager at Quipper Ltd.
- Ruby, Javascript (mostly React js),
Unix, Kubernetes
- Hobbies
- Pre-COVID: travelling, swimming
- COVID: reading, watching series
and movies

3. #UXDXAPAC Handling Sudden Growth

4. #UXDXAPAC Handling Sudden Growth
Agenda | 01
02
03
04
05
06
Early 2020
Our Challenges
Our Response
Results
The Future
Summary

5. #UXDXAPAC Handling Sudden Growth
Early 2020
01

6. #UXDXAPAC Handling Sudden Growth
Early 2020
Restructuring: Splitting the Japanese business to the rest of the global business
WHY?
1. Difference in Academic calendars
2. Ensures Future business sustainability

7. #UXDXAPAC Handling Sudden Growth
If you want to place an image with full screen,
remove the top blue bar and the footer
But then on March 2020…

8. #UXDXAPAC Handling Sudden Growth
Changes in user behavior
Supplemental
service
Synchronous
education

9. #UXDXAPAC Handling Sudden Growth
Changes in user behavior
Only using
Quipper
contents
Now creating
their own
content

10. #UXDXAPAC Handling Sudden Growth
Teacher’s usage pattern (Oct 2019 vs Oct 2020)

11. #UXDXAPAC Handling Sudden Growth
Student’s usage pattern (Oct 2019 vs Oct 2020)

12. #UXDXAPAC Handling Sudden Growth
Number of Access (Teacher vs Students)
pre-covid covid

13. #UXDXAPAC Handling Sudden Growth
Our Challenges
02

14. #UXDXAPAC Handling Sudden Growth
Effects on our product
- More demand for features to facilitate online education
- Higher traffic due to more users with higher activity per user
- More bugs discovered due to increased usage
- Higher urgency in addressing and fixing bug reports

15. #UXDXAPAC Handling Sudden Growth
August 2020:
- Start of some private schools
October 2020:
- Start of public schools
Incidents Happened... Incidents

16. #UXDXAPAC Handling Sudden Growth
Incident Investigation Results
Main Symptoms:
- Slow access on some pages, leading to errors
- Late delivery of activities to students
Causes:
- Slow performance on database need further optimizations
- No CDN on some platforms set up CDN
- Autoscaling setup didn’t work improve scaling based on peak times

17. #UXDXAPAC Handling Sudden Growth
Our Response
03

18. #UXDXAPAC Handling Sudden Growth
We were not ready for this!
We didn’t expect to grow this fast!
😱

19. #UXDXAPAC Handling Sudden Growth
Platform Optimization Efforts
WHY?
- By focusing on developing features, we had less attention on scalability.
- We accumulated technical debt and now it’s the time to pay back.
- Developing new features is useless if users cannot use our core features.
Which lead to…
a freeze on developing new features

20. #UXDXAPAC Handling Sudden Growth
Platform Optimization Efforts
I was chosen to lead the efforts and we broke down our efforts into 3 main
focus:
Technical
Communications
Operations

21. #UXDXAPAC Handling Sudden Growth
Platform Optimization Efforts
Technical
Allows us to prevent these incidents from happening again
- Clean up old code
- Remove unnecessary requests
- Improve performance via caching, N+1 fixes, improving algorithms, etc
- Setup auto-scaling and scheduled scaling
- Ensure apps have enough resources so they don’t crash
- Improve database for higher capacity

22. #UXDXAPAC Handling Sudden Growth
Platform Optimization Efforts
Communications
Keeps everyone on the same page
- Agree on timeline with the stakeholders
- Communicate with users about the efforts being done
- Weekly checkup with Platform leads on the optimization efforts, to also
maintain cross-platform communication
- Compile common issues encountered so that support officers can
respond to users faster
- Create an incident management flow so that everyone is up-to-date on
status of incidents that may happen

23. #UXDXAPAC Handling Sudden Growth
Platform Optimization Efforts
Operations
Allows us to recover from incidents as quickly as possible
- Have developers on stand by (on a rotation basis) from 6:30 am for
possible incidents.

24. #UXDXAPAC Handling Sudden Growth
Results
04

25. #UXDXAPAC Handling Sudden Growth
Platform Optimization Results
- Less crashes

26. #UXDXAPAC Handling Sudden Growth
Platform Optimization Results
- Improved overall response time

27. #UXDXAPAC Handling Sudden Growth
Platform Optimization Results
- No major incidents, despite increasing usage
- Our scaling system worked perfectly
- Established cross-platform communications
- Raised awareness on how developments in different departments affect
each other

28. #UXDXAPAC Handling Sudden Growth
But the war is not over...
We managed the traffic at peak
times!!! 🎉

29. #UXDXAPAC Handling Sudden Growth
The Future
05

30. #UXDXAPAC Handling Sudden Growth
- Degradations at some points of
the day
- We found that we hit our limit
of concurrent usage.
Battle is won but the war is not yet over... Incident?

31. #UXDXAPAC Handling Sudden Growth
More Future Improvements
- Establish workflows and processes
- Improve our incident response process
- Switch from reactive to proactive
- Changes on an organizational level from the lessons learned

32. #UXDXAPAC Handling Sudden Growth
Summary
06

33. #UXDXAPAC Handling Sudden Growth
Takeaways
- Always analyze the data to make sound decisions
- Don’t procrastinate in addressing technical issues. Technical debt will
come back to bite you when you are less prepared.
- We can prepare all we want but we will always encounter situations we
are not ready for. Learn to adapt!
- Be decisive, for the users. Make a call and own it. Taking a lot of time to
cascade discussions and decisions can be costly.

34. #UXDXAPAC Handling Sudden Growth
Visit https://career.quipper.com/
Be a distributor of wisdom and join us!
We’re looking for:
● Software Engineers (Web & Mobile)
● Product Designers (UI & UX)
● SRE
󾓥 🇮🇩
🇵🇭

35. #UXDXAPAC Handling Sudden Growth
Thank you!