Lightning talk I gave at SurgeCon 2012

1. How netfilter saved my
bacon
Fred Moyer
@phredmoyer
Silver Lining Networks
Thursday, September 27, 12

2. Free WiFi!
Ad bar inserted at the top of page pays the WiFi bill
(Silver Lining Ad Bar shown here)
Thursday, September 27, 12

3. Others have built it, you may have used
theirs (notice this page didn’t load fully)
Thursday, September 27, 12

4. How does theirs work?
• Tinyproxy runs on the network gateway
• Inserts Javascript into the HTTP response
which splits the page into two frames, one
for the ad bar, one for the web page content
• Proxying HTTP responses through
userspace on network devices is *slow*
• Users get angry; this solution sucks
Thursday, September 27, 12

5. Ignorance is bliss; how I
built it from scratch
• iptables rules on the gateway device NAT
forwarded HTTP requests to co-located
mod_perl web proxy
• Better performance than tinyproxy
• Running all web traffic through colocation
doesn’t scale though (and is really
expensive)
Thursday, September 27, 12

6. Making it scale
• Avoid sending static content requests (images,
videos, etc) through the colocated proxy
• HTTP proxy rewrites static content links:
• http://foo.com/image.jpg =>
• http://foo.com:8135/image.jpg
• Redirect port 8135 to port 80 via router iptables
rule:
• iptables -t NAT -A PREROUTING -i $LAN -p tcp
--dport 8135 -j DNAT --to :80
Thursday, September 27, 12

7. Scalability achieved
• 95% of traffic offloaded from the co-
located proxy and fetched directly from the
destination
• Hillbilly architecture driven by desperation
and experimentation rather than elegant
planning
• Performance was much better than the
tinyproxy approach used by competitors
Thursday, September 27, 12

8. Yo dawg, I heard you
like 400s
• Whoops, it doesn’t completely work
• Apache handles http://foo.com:8135 requests
to port 80 just fine
• lighttpd throws a 400 Bad Request!
• ~20% of static content requests returning 400s
makes users (and network operators) angry
Thursday, September 27, 12

9. Linux based routers use the sk_buff socket buffer
struct in kernel space. Maybe a netfilter module can
remove the :8135 from the hostname...
Thursday, September 27, 12

10. 3 months of Netfilter coding
Thursday, September 27, 12

11. Architectural Overview
gateway iptables
redirect to proxy
GET http://foo.com/ HTTP proxy
iptables -t NAT -A PREROUTING -i $LAN -p tcp --dport 80
--dst ! 192.168.0.0/16 -j DNAT --to 69.36.240.29:80
Thursday, September 27, 12

12. Architectural Overview
gateway iptables
redirect to proxy
GET http://foo.com/ HTTP proxy
html response with ad,
subrequest hrefs on port 8135
index.html
proxied request
foo.com
Thursday, September 27, 12

13. Architectural Overview
gateway iptables
redirect to proxy
GET http://foo.com/ HTTP proxy
html response with ad,
subrequest hrefs on port 8135
index.html
browser parses page, proxied request
makes image subrequest
GET http://foo.com:8135/bar.jpg foo.com
Thursday, September 27, 12

14. Architectural Overview
gateway iptables
redirect to proxy
GET http://foo.com/ HTTP proxy
html response with ad,
subrequest hrefs on port 8135
index.html
browser parses page, proxied request
makes image subrequest
netfilter module removes
:8135 hostport
GET http://foo.com:8135/bar.jpg foo.com
subrequest bypasses proxy,
fetches image directly
Thursday, September 27, 12

15. It works (finally)
• ‘Host: foo.com:8135’ => ‘Host: foo.com’ in
kernel space. No user space copying.
• < 500 ms additional latency for main page
requests through the co-located HTTP proxy
• Blows tinyproxy out of the water
• Product didn’t survive business needs though,
the ad revenue wasn’t there :(
Thursday, September 27, 12

16. Thank you Surge 2012
• Stuck in an architectural dead end?
Maybe this approach can help you.
• github.com/redhotpenguin/SL-Kernel
• www.skbuff.net/skbuff.html
• banu.com/tinyproxy
Thursday, September 27, 12