6/16/16 Billing&Payments Engineering Meetup III - Throttling Engine at Netflix

1. Raja Senapati
Sr. Software Engineer, Payments

2. Why Do We Want to Slow Down?
• Scaling Issues with Online Payment Processors
• Adherence to SLA

3. • Netflix will send a maximum of 24 concurrent
transactions at any time. - processor1
• We can process up to 30 transactions
simultaneously at any time except between
12:00 - 3:00 where we can process up to 5
concurrent transactions. - processor2
Typical SLA

4. • The Lowest concurrent transaction processing
supported by one of our Payment Processors
is
– 1 concurrent transaction
– 3 concurrent transactions
– 5 concurrent transactions
– 7 concurrent transactions
– 9 concurrent transactions
– 11 concurrent transactions
Quiz?

5. Problem Context
EC2 instance
Client Apps
Billing, Signup UI
Payments
Processor 1
SLA : 2
EC2 instance
EC2 instance
hi
Courtesy: Hertz Comics

6. Solution Approach
EC2 instance
Client Apps
Billing, Signup UI
Payments
Processor 1
SLA : 2
EC2 instance
EC2 instance
Permit Pool

7. Permit Pool Exhausted
EC2 instance
Client Apps
Billing, Signup UI
Payments
Processor 1
SLA : 2
EC2 instance
EC2 instance
Permit Pool
Courtesy: Hertz Comics

8. Permit Pool Reuse
EC2 instance
Client Apps
Billing, Signup UI
Payments
Processor 1
SLA : 2
EC2 instance
EC2 instance
Permit Pool
Courtesy: Hertz Comics

9. Technology Stack
• Apache Zookeeper
• Distributed
Coordinator
• Apache Curator
• Open Sourced by Netflix
• Shared Semaphore
Recipe

10. Code Snippets

11. Throttling Engine Features
• Fast Property based
• Support Combination of Processor, Country,
Transaction Type, Time of Day Processing
• THROTTLE-EVERYTHING mode and
THROTTLE-NOTHING mode in the rare case
zookeeper is down

12. Thank You