Velocity 2011 - Our first DDoS attack
•
1 j'aime•1,676 vues
Your website just went down. As you try to understand what has gone wrong, you quickly realize something is different this time. There’s no clear reason why your site should be down, but indeed it is. This talk is about the story of our team’s first unprepared fight against a DDoS attack.
Recommandé
Contenu connexe
Tendances
Tendances (20)
En vedette
Similaire à Velocity 2011 - Our first DDoS attack
Similaire à Velocity 2011 - Our first DDoS attack (20)
Plus de Cosimo Streppone
Plus de Cosimo Streppone (11)
Dernier
Dernier (20)
Velocity 2011 - Our first DDoS attack
- 1. Our My first DDoS attack Velocity Europe 2011 – Berlin Cosimo Streppone Operations Lead
- 2. <video of Mr. Wolf going to Jimmy's house in Pulp Fiction> this couldn't fit in the PDF... sorry. http://www.youtube.com/watch?v=hsKv5d0sIlU
- 8. nginx – secret sauces? # Pavel's secret gzip tuning sauce gzip on; gzip_disable msie6; gzip_min_length 1100; gzip_buffers 16 8k; gzip_comp_level 3; gzip_types text/plain application/xml application/x-javascript text/css;
- 9. nginx – secret sauces? # Michael's secret file cache sauce open_file_cache max=1000 inactive=20s; open_file_cache_valid 30s; open_file_cache_min_uses 2; open_file_cache_errors on;
- 10. nginx – antidos.conf # More on https://calomel.org/nginx.html client_header_timeout 5; client_body_timeout 10; ignore_invalid_headers on; send_timeout 10; # To limit slowloris-like attacks client_header_buffer_size 4k; large_client_header_buffers 4 4k;
- 11. nginx – drop client connections # Cut abusive established connections, # forcing clients to reconnect location ~ ^/Ao-Trang-Oi/blog/ { return 444; }
- 12. nginx – varnish caching nginx varnish backends
- 13. iptraf
- 14. tcpdump of anomalous traffic GET /Ao-Trang-Oi/blog/show.dml/14715682 HTTP/1.1 User-Agent: 1.{RND 10}.{RND 10} Referrer: http://my.opera.com/Ao-Trang-Oi/ Cache-Control: no-cache Cookie: __utma=218314117.745395330 […] __utmz=218314117.1286774593. […] utmcsr=google|utmccn= […] utmctr=cach%20de%20hoc%20mon […] <... random high speed junk follows ...>
- 15. tcpdump of anomalous traffic GET /Ao-Trang-Oi/blog/?startidx=1295 HTTP/1.1 User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US;) Gecko/20030624 Netscape/7.1 (ax) Accept: Accept=text/html,application/xhtml+xml,... Accept-Language: Accept-Language=en-us,en;q=0.5 Accept-Encoding: gzip,deflate Accept-Charset: Accept-Charset=ISO-8859-1,... Referer: http://my.opera.com/Ao-Trang-Oi/blog/ Pragma: no-cache Keep-Alive: 300 ua-cpu: x86 Connection: close
- 16. #nginx, 14th October 2010 cosimo: we're seeing a pretty "interesting" problem within our nginx fronts BLAH BLAH BLAH cosimo: there's a few hosts sending a legitimate HTTP GET request BLAH BLAH BL cosimo: followed by a binary stream of random bytes that never ends BLAH BLAH BLAH cosimo: this is just 1 request going on and on cosimo: is there some way to alter the nginx config to shut down these client connections? OMGWTFBBQ!!!!11111 cosimo: the client is sending something like: cosimo: GET /blah HTTP/1.1 “this is nkiller2” cosimo: Host: ... cosimo: Etc: etc... cosimo: and then random bullshit vr: :) vr: this is nkiller2 vr: haproxy can fight this vr: you can set a timeout http-request vr: don't know if nginx can do this cosimo: cool
- 17. PHRACK#66
- 18. tcp window zero?
- 19. iptables -A -m u32 --u32 “6&0xFF=0x6 && 4&0x1FFF=0 && 0>>22&0x3C () 12&0xFFFF=0x0000” -j ZERO_WINDOW_RECENT
- 20. u32 zero window filter 6 & 0xFF = 0x6
- 21. u32 zero window filter 4 & 0x1FFF = 0x0
- 22. u32 zero window filter 0>>22 & 0x3C () 12 & 0xFFFF = 0x0
- 23. u32 zero window filter 0>>22 & 0x3C () 12 & 0xFFFF = 0x0 ??
- 26. 0>>22& [EMAIL PROTECTED] &0xFFFF=0x0000
- 28. u32 zero window filter 0>>22 & 0x3C @ 12 & 0xFFFF = 0x0
- 29. iptables rules - logging $ipt -N ZERO_WINDOW_RECENT $ipt -A INPUT -m u32 --u32 "6&0xFF=0x6 && 4&0x1FFF=0 && 0>>22&0x3C@12&0xFFFF=0x0000" -j ZERO_WINDOW_RECENT $ipt -A ZERO_WINDOW_RECENT -m recent --set --name ZERO_WINDOW $ipt -A ZERO_WINDOW_RECENT -m recent --update --seconds 60 --hitcount 20 --name ZERO_WINDOW -j LOG --log-level info --log-prefix "ZeroWindow"
- 31. iptables rules - blocking $ipt -N ZERO_WINDOW_RECENT $ipt -A INPUT -m u32 --u32 "6&0xFF=0x6 && 4&0x1FFF=0 && 0>>22&0x3C@12&0xFFFF=0x0000" -j ZERO_WINDOW_RECENT $ipt -A ZERO_WINDOW_RECENT -m recent –set --name ZERO_WINDOW $ipt -A ZERO_WINDOW_RECENT -m recent –update --seconds 60 --hitcount 20 --name ZERO_WINDOW -j DROP
- 32. shields-up.vcl cacheable content nginx varnish non-cacheable content backends
- 33. shields-up.vcl all HTTP content varnish nginx HTTPS-only traffic backends
- 35. Pingdom response time 20s 10s 0s
- 36. End 29-Oct-2010
- 37. Packets/s seen by firewall Start 13-Oct-2010 End 29-Oct-2010
- 40. ¿Questions?
- 41. What can we, as Ops, do better? ● Embrace failures and learn from them ● Be fast (no panic/blame, think Mr. Wolf) ● Coordinate (#ops, war rooms, ...) ● Take notes ● Learn TCP/IP ● Know your tools (tcpdump, tcpflow, strace, nc, iptraf, …)
- 42. my base_packages puppet module class base_packages { $packagelist = [ "ack-grep", "colordiff", "curl", "facter", "git-core", "htop", "iftop", "iptraf", "jed", "joe", "libwww-perl", "logrotate", "lsof", "make", "mc", "oprofile", "psmisc", "rsync", "screen", "svn", "sysstat", "tcpdump", "tcpflow", "telnet", "unzip", "vim", "zip" ] package { $packagelist: ensure => "installed", } }
- 43. Thanks to... ● ithilgore (sock-raw.org) for writing nkiller2 ● @vr in #nginx for pointing us at nkiller2 ● David Falloon for his great “untested” idea ● marc.info for correctly handling “@” in ml ● SANS Institute for the TCP/IP references ● My team at Opera
- 44. Danke!