RTBkit Meetup - Developer Spotlight, Behind the Scenes of RTBkit and Intro to the Stack

Virtual Meetup – January 2015

Agenda
• Developer Spotlight – Nicolas Emiliani – RTB Dev Lead
• RTBkit Behind the Scenes
• More than a Bidder – The RTBkit Stack
• Open Q&A
Copyright © 2015 Datacratic Inc. All rights reserved

Getting an RTBKit installation running.
More info here : http://datacratic.com/site/blog/tales-rtbkit-adventure
Nicolas Emiliani, RTB dev. team Tech Leader at Motrixi.

A little about our traffic :
 We are a mobile centered DSP
 We are connected to Nexage, Rubicon, Mopub and more recently to ADX
 We take up to 40k qps in peak hours from US and Canada

Life on the Amazon (not the rain forest)
 Create a VPC (or you will regret it, I promise)
 Create all your production nodes inside that VPC
 Be smart about security : do not expose your nodes
unnecessarily to the internet.
 Use a VPN software (ie: OpenVPN) to access the
nodes safely inside your VPC.

Isolating the RTBKit stack (BR loop)
 Chose an HTTP reverse proxy (we use NGINX)
 Set up the proxy as a gateway to your RTBKit installation, we use elastic
IP s on the NGINX nodes and allow incoming traffic only for the used
HTTP ports. Elastic IPS are cool and they will let you do all sort of handy
things.
 You can use the proxy as a load balancer if you have more than one
node
 You can use the proxy to automatically reply with 204s to the exchanges
in case you need to work on your installation
 You can use the proxy to clone the traffic and send it to a staging or
development environment (we recently cracked how to do this with
NGINX)

Isolating the RTBKit stack (BR loop)

Isolating the RTBKit PAL
 As with the BR loop use a reverse proxy with an Elastic IP

Type of nodes
 Different responsibilities → Different nodes → Easier scaling

Kernel parameters
 fs.file-max=300000 : RTBKit likes to chew up your file descriptors.
 net.ipv4.tcp_tw_reuse=1 : allow reusing sockets in TIME_WAIT state for
new connections.
 net.ipv4.tcp_fin_timeout=15 Close connections in the TCP FIN timeout
state in 15 seconds ( default 60 )
Tweaking this parameters will prevent crashes due to exceptions related to
too many open file descriptors.

Use the APIs, Luke
 Banker API : comes in handy when implementing pacing strategies thus
keeping logic simple in agents.
 Agent Configuration Service API : is useful to implement agent
configuration related APIs as the Agent Gateway
(https://github.com/nemiliani/agent_gateway).

Things that will help you out
 All our processes run under a watchdog, currently we are using monit
 Zabbix is your friend, particularly when things go wrong.
 Don't ever forget about Graphite. RTBKit is a huge black box otherwise.
Always keep an eye on your metrics. Yous should be doing it right now!,
no, really!

What’s taking so long?
● Ancient Node.js version in platform-deps
o Upgrading requires fixing the bindings
o Lots of work, very little pay-off

Alternative?
● Transition to HTTP interfaces
o Support for multiple languages
o Cleaner well defined OpenRTB protocol
o Start of RTBkit 2.0 implementation

How will it work?
● Abstracted agent communication
o HTTP Bidder Interface class
o Agent config specifies bidder URL

Details
● Added two new HTTP bidder examples:
o Python bidder based on tornado
o Node.js bidder
● Detailed walkthrough on the wiki

Backwards compatibility?
● Node.js support will be dropped for 14.04
● ZMQ C++ interface will remain for now.
➔ Will require an upgrade.
➔ Should be fairly painless.

Platform-deps
● Fork of platform-deps for RTBkit
o Upgrade dependencies to 14.04
o Cleanup of pointless dependencies

How long?
● Platform-deps fork is almost completed
o Should be released in a week or two
o Work done by Michael
● Currently re-writing our agents in golang
o Will iron-out kinks in the bidder interface
o Should be done in a month or two

More Than a Bidder
A Brief Introduction to an RTB Stack

But there is more to it than that ...
Campaign
Management
UI
Campaign
Reporting UI
ETL / Log
Processing
Offline
Optimization /
Bid Strategy
Logs / Log
Management
Ad Server
Online
Audience
Data Store
Offline Data
Prep
Monitoring Alerting 24/7 Ops
Conversion
Collection
Data
Collection /
DMP

You need an RTB Solution
Requirement Provided by RTBkit?
Reliable, high-performance core bidder system Yes
Online Impression / Click / Conversion matching Yes
Pacing No
Logging Customer has to set up and manage logs
Monitoring and Alerting Some support for Carbon and Graphite
24-7 operations and support No
Capacity planning and system evolution No
Campaign management UI integrated with the bidder No
Seats on exchanges / integration with exchanges No
ETL for campaign reporting and bidding agents No

Other Issues …
Keep it Simple
● There is an abundance of inventory. Your goal is to process enough bids to
achieve your goals. Over-engineering is expensive.
● Unreliability is probably OK
o Campaigns run for a long time and buy millions of impressions. Short outages
probably don't matter
o Most partners fine with noticeable error rates (~2%)

Other Issues …
Pacing and Strategy / Optimization
● Hard to do in real time, at high volume of transactions, in a distributed
system
● Hard to understand what data and how much data to use
● Risky
o blown daily caps
o uneven or poor performance

Other Issues … Ad Ops
Existing Ad Ops
● Used to feature set of DSP UI
● Comfortable delivering campaigns with current tool set
● May be resistant to using a new system
● May be resistant to trusting a new system
● You may spend a lot of time on UI features and reports, not on your real-
time platform

Host an RTBkit Meetup!

Q & A

RTBkit Meetup - Developer Spotlight, Behind the Scenes of RTBkit and Intro to the Stack

Recommended

Recommended

More Related Content

Similar to RTBkit Meetup - Developer Spotlight, Behind the Scenes of RTBkit and Intro to the Stack

Similar to RTBkit Meetup - Developer Spotlight, Behind the Scenes of RTBkit and Intro to the Stack (20)

Recently uploaded

Recently uploaded (20)

RTBkit Meetup - Developer Spotlight, Behind the Scenes of RTBkit and Intro to the Stack