Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.
BOSTON 10-11 SEPT 2018
My ragequit journey: configuring
Netflix tools
SARAH YOUNG
BOSTON 10-11 SEPT 2018
My ragequit journ...
BOSTON 10-11 SEPT 2018
BOSTON 10-11 SEPT 2018
whoami
• Sarah Young, Security Architect at Versent.
• I’m from Melbourne in Australia.
• I help cu...
BOSTON 10-11 SEPT 2018
If anyone knows Justin Trudeau, please let me know.
BOSTON 10-11 SEPT 2018
I am not a Christian author
BOSTON 10-11 SEPT 2018
Firstly…
• This talk is not an attack on Netflix.
• I love Netflix as both an end user of their ser...
BOSTON 10-11 SEPT 2018
Intro to Netflix tools
• I don’t have to introduce Netflix… I hope?!
• Netflix have been releasing ...
BOSTON 10-11 SEPT 2018
Just one more note…
• I’m aware that there are talks at other conferences and meetups where
compani...
BOSTON 10-11 SEPT 2018
Tools I’m going to look at
• BLESS (Bastion's Lambda Ephemeral SSH Service)
• Security Monkey
• Rep...
BOSTON 10-11 SEPT 2018
The beginning of the journey…
• I was equipped with:
• Git Readmes.
• My work’s sandbox AWS account...
BOSTON 10-11 SEPT 2018
Don’t test the demo gods
BOSTON 10-11 SEPT 2018
BLESS – Qué?
• BLESS stands for Bastion's Lambda Ephemeral SSH Service.
• It’s an Internal Certific...
BOSTON 10-11 SEPT 2018
BLESS – awscli is not my friend
• Create an AWS role, easy.
• Maybe my Python version is too new fo...
BOSTON 10-11 SEPT 2018
BLESS – saml2aws
BOSTON 10-11 SEPT 2018
Firstly…
• Cue lengthy Slack
discussion about how
Brew/Python/awscli suck.
• Let’s just reinstall a...
BOSTON 10-11 SEPT 2018
BLESS – Virtual-env is additionally not my friend
• False start, let’s go now.
• Have to force inst...
BOSTON 10-11 SEPT 2018
BLESS – Certificates, KMS and Lambda are dope
• Generate certs just fine.
• Make keys in KMS just f...
BOSTON 10-11 SEPT 2018
BLESS – OSX, you make my life hard
• BLESS should be finished.
• Now to test it.
• I don’t have Bot...
BOSTON 10-11 SEPT 2018
BLESS – Dammit Python
Credit: XKCD
BOSTON 10-11 SEPT 2018
BLESS – Dammit again Python
BOSTON 10-11 SEPT 2018
Sidenote
BOSTON 10-11 SEPT 2018
BLESS – What’s the first rule of security…?
• I don’t have creds (apparently).
• Turns out this is ...
BOSTON 10-11 SEPT 2018
BLESS – don’t do this
BOSTON 10-11 SEPT 2018
BLESS – do this
BOSTON 10-11 SEPT 2018
BLESS – Real-life issues
• Very little guidance on how to scale BLESS.
• “Deploy an Amazon Linux AM...
BOSTON 10-11 SEPT 2018
BLESS – When devs don’t do what they’re told
• Got BLESS running through Jenkins.
• Devs still used...
BOSTON 10-11 SEPT 2018
BLESS - scoreboard
• Instructions – 6/10
• Accuracy of instructions – 8/10
• Ease of configuration ...
BOSTON 10-11 SEPT 2018
Security Monkey – Qué?
• Security Monkey is a tool that monitors/alerts/reports one or more AWS
acc...
BOSTON 10-11 SEPT 2018
Security Monkey – Deployment structure
BOSTON 10-11 SEPT 2018
Security Monkey - Hurrah, instructions!
• Hey, this one looks like it has a
decent walkthrough on G...
BOSTON 10-11 SEPT 2018
Security Monkey - Ah, maybe not yay after all.
• Oh wait… it’s kind of out of date…
M1 instances do...
BOSTON 10-11 SEPT 2018
Security Monkey - When your lab messes things up
• Pro tip: never use a lab your colleague has only...
BOSTON 10-11 SEPT 2018
Security Monkey – Let’s build this
• Now for the interesting stuff.
• Let’s install this thing.
• P...
BOSTON 10-11 SEPT 2018
Security Monkey - Why doesn’t my instance recognise loopback?
• All going well until sudo keeps fai...
BOSTON 10-11 SEPT 2018
Security Monkey - Python isn’t working
• When a guide posts something like this, you should probabl...
BOSTON 10-11 SEPT 2018
Security Monkey - Je ne parle pas anglais.
• Running in the virtual environment shell now, my bad.
...
BOSTON 10-11 SEPT 2018
Security Monkey - Who doesn’t love a 404?
• No idea why, but I had to re-generate the en_US locales...
BOSTON 10-11 SEPT 2018
Security Monkey – I spoke too soon
• Now everything should be running, right?
BOSTON 10-11 SEPT 2018
Security Monkey – Mysterious directories
• NGNIX can’t find the UI pages to load.
• Much searching,...
BOSTON 10-11 SEPT 2018
Security Monkey – Damn those SSL certs
• Generate self-signed SSL certs.
• Getting an error from Ch...
BOSTON 10-11 SEPT 2018
Security Monkey – Hello web UI!
• …aaaaand:
BOSTON 10-11 SEPT 2018
Security Monkey – Dude, where’s my login server?
• Pretty sure I’m supposed to have a login screen?...
BOSTON 10-11 SEPT 2018
Security Monkey – Success!
BOSTON 10-11 SEPT 2018
Security Monkey – Production issues
• Issue lists aren’t very detailed.
• Dashboard scores for the ...
BOSTON 10-11 SEPT 2018
Security Monkey - Scoreboard
• Instructions – 8/10
• Accuracy of instructions – 7/10
• Ease of conf...
BOSTON 10-11 SEPT 2018
Repokid - Qué?
• Repokid uses Access Advisor provided by Aardvark to remove
permissions granting ac...
BOSTON 10-11 SEPT 2018
Repokid - Wow, these instructions are pretty light.
• Even by Netflix standards, these
are pretty l...
BOSTON 10-11 SEPT 2018
Repokid – The downsides of using a lab
• Run out of elastic IPs.
• Reassign one, but now my termina...
BOSTON 10-11 SEPT 2018
Repokid – More Python woes…
• Instance says virtual env isn’t there (apparently).
• Instance also s...
BOSTON 10-11 SEPT 2018
Repokid – Only one thing to do
TERMINATE
BOSTON 10-11 SEPT 2018
Repokid – Git and SSH key troubles
• Wash, rinse, repeat the previous slides.
• Accessing Git repo…
BOSTON 10-11 SEPT 2018
Repokid – Let’s try that again
• Generate fresh SSH keys.
• Add to my agent.
• Upload to Github.
BOSTON 10-11 SEPT 2018
Repokid – Never mentioned I needed a database
• Apparently I need a Dynamo DB.
• Use a small local ...
BOSTON 10-11 SEPT 2018
Repokid – Readmes with footnotes
• The footnotes for Repokid are important.
• Describe what roles n...
BOSTON 10-11 SEPT 2018
Repokid - Fine tuning JSON
• Fine tune the JSON config file.
• Point at Aardvark, Dynamo DB and IAM...
BOSTON 10-11 SEPT 2018
Repokid – Production issues
• role.policies only checks inline policies. Attached policies are igno...
BOSTON 10-11 SEPT 2018
Repokid - Scoreboard
• Instructions – 2/10
• Accuracy of instructions – 2/10
• Ease of configuratio...
BOSTON 10-11 SEPT 2018
And I’m finished!
BOSTON 10-11 SEPT 2018
Lessons learned
• Read what instructions you have carefully…
• … but don’t be entirely beholden to ...
BOSTON 10-11 SEPT 2018
What’s next?
• Diffy!
• Diffy is a triage tool to help digital forensics
and quickly identify compr...
BOSTON 10-11 SEPT 2018
Documents and links
• Netflix Open Source Software Center - https://netflix.github.io/
• Netflix te...
BOSTON 10-11 SEPT 2018
Thanks for not ragequitting
on my talk and going to
happy hour.
Questions?
@_sarahyo
Prochain SlideShare
Chargement dans…5
×

DevSecCon Boston 2018: My rage quit journey: configuring Netflix tools by Sarah Young

637 vues

Publié le

DevSecCon Boston 2018: My rage quit journey: configuring Netflix tools by Sarah Young

Publié dans : Technologie
  • Soyez le premier à commenter

DevSecCon Boston 2018: My rage quit journey: configuring Netflix tools by Sarah Young

  1. 1. BOSTON 10-11 SEPT 2018 My ragequit journey: configuring Netflix tools SARAH YOUNG BOSTON 10-11 SEPT 2018 My ragequit journey: configuring Netflix tools SARAH YOUNG
  2. 2. BOSTON 10-11 SEPT 2018
  3. 3. BOSTON 10-11 SEPT 2018 whoami • Sarah Young, Security Architect at Versent. • I’m from Melbourne in Australia. • I help customers move their stuff into the cloud securely. • Worked in tech for the past 9ish years. • I’ve worked in Europe, New Zealand and Australia. • I overuse memes and GIFs. • Wannabe crazy bird lady.
  4. 4. BOSTON 10-11 SEPT 2018 If anyone knows Justin Trudeau, please let me know.
  5. 5. BOSTON 10-11 SEPT 2018 I am not a Christian author
  6. 6. BOSTON 10-11 SEPT 2018 Firstly… • This talk is not an attack on Netflix. • I love Netflix as both an end user of their service and a consumer of their SecOps tools. • Alas, I am also not on commission from Netflix. • The aim of this talk is to demonstrate how everyone struggles with tools from time-to-time. • I want to try to reduce “FOFU”, “fear of F!%*ing up”.
  7. 7. BOSTON 10-11 SEPT 2018 Intro to Netflix tools • I don’t have to introduce Netflix… I hope?! • Netflix have been releasing Open Source tools since 2014. • They release numerous types of tools: • Big data • Content encoding • Insight, reliability and performance monitoring • … and much more • I’m going to focus on some of their security tools.
  8. 8. BOSTON 10-11 SEPT 2018 Just one more note… • I’m aware that there are talks at other conferences and meetups where companies and individuals talk about successful implementations of these tools. • This is not one of those talks. • I will link to some of the happier Hollywood stories at the end of talk.
  9. 9. BOSTON 10-11 SEPT 2018 Tools I’m going to look at • BLESS (Bastion's Lambda Ephemeral SSH Service) • Security Monkey • Repokid
  10. 10. BOSTON 10-11 SEPT 2018 The beginning of the journey… • I was equipped with: • Git Readmes. • My work’s sandbox AWS account. • Google. • Slightly rusty Linux skills. • Unlimited cans of fizzy drinks from the fridge. • My patience.
  11. 11. BOSTON 10-11 SEPT 2018 Don’t test the demo gods
  12. 12. BOSTON 10-11 SEPT 2018 BLESS – Qué? • BLESS stands for Bastion's Lambda Ephemeral SSH Service. • It’s an Internal Certificate Authority. • Inside a Lambda function. • Issues short-lived certificates for EC2 access. • Certificates have 120 seconds validity by default.
  13. 13. BOSTON 10-11 SEPT 2018 BLESS – awscli is not my friend • Create an AWS role, easy. • Maybe my Python version is too new for awscli? • Let’s uninstall Python3.
  14. 14. BOSTON 10-11 SEPT 2018 BLESS – saml2aws
  15. 15. BOSTON 10-11 SEPT 2018 Firstly… • Cue lengthy Slack discussion about how Brew/Python/awscli suck. • Let’s just reinstall awscli.
  16. 16. BOSTON 10-11 SEPT 2018 BLESS – Virtual-env is additionally not my friend • False start, let’s go now. • Have to force install virtual-env. • I’m using Docker. • All goes well here.
  17. 17. BOSTON 10-11 SEPT 2018 BLESS – Certificates, KMS and Lambda are dope • Generate certs just fine. • Make keys in KMS just fine. • Make Lambda function just fine. • Things are going too well… surely?! Accurate depiction of me at this point
  18. 18. BOSTON 10-11 SEPT 2018 BLESS – OSX, you make my life hard • BLESS should be finished. • Now to test it. • I don’t have Boto3… • … except I do. • Dammit Python dependencies!
  19. 19. BOSTON 10-11 SEPT 2018 BLESS – Dammit Python Credit: XKCD
  20. 20. BOSTON 10-11 SEPT 2018 BLESS – Dammit again Python
  21. 21. BOSTON 10-11 SEPT 2018 Sidenote
  22. 22. BOSTON 10-11 SEPT 2018 BLESS – What’s the first rule of security…? • I don’t have creds (apparently). • Turns out this is a bug in saml2aws. • I should have updated to 2.7.0 before I started. I deserve Trump shame for this fail.
  23. 23. BOSTON 10-11 SEPT 2018 BLESS – don’t do this
  24. 24. BOSTON 10-11 SEPT 2018 BLESS – do this
  25. 25. BOSTON 10-11 SEPT 2018 BLESS – Real-life issues • Very little guidance on how to scale BLESS. • “Deploy an Amazon Linux AMI” isn’t super helpful. • Re-scaling the application takes downtime. • Debugging BLESS sucks. • When pen testing BLESS, we had to expose Unicreds. • Defeats the object of pen testing somewhat.
  26. 26. BOSTON 10-11 SEPT 2018 BLESS – When devs don’t do what they’re told • Got BLESS running through Jenkins. • Devs still used our manually deployed bastion. • ”Make it easy to do the right thing and hard to do the wrong thing”. • Resources who maintained BLESS rolled off projects. • Nuances introduced by devs could cause problems.
  27. 27. BOSTON 10-11 SEPT 2018 BLESS - scoreboard • Instructions – 6/10 • Accuracy of instructions – 8/10 • Ease of configuration – 5/10 • Ragequit score – 7/10
  28. 28. BOSTON 10-11 SEPT 2018 Security Monkey – Qué? • Security Monkey is a tool that monitors/alerts/reports one or more AWS accounts for anomalies. • Part of a larger suite of tools from Netflix known as the Simian Army.
  29. 29. BOSTON 10-11 SEPT 2018 Security Monkey – Deployment structure
  30. 30. BOSTON 10-11 SEPT 2018 Security Monkey - Hurrah, instructions! • Hey, this one looks like it has a decent walkthrough on Github. • Let’s give it a go.
  31. 31. BOSTON 10-11 SEPT 2018 Security Monkey - Ah, maybe not yay after all. • Oh wait… it’s kind of out of date… M1 instances don’t exist any more. Decide to wing it and pick an M5. This is not free tier.
  32. 32. BOSTON 10-11 SEPT 2018 Security Monkey - When your lab messes things up • Pro tip: never use a lab your colleague has only half configured. • Instance was not accessible from external bastion host. • Bastion host wouldn’t forward SSH keys to the Security Monkey instance. • Cue numerous error messages and troubleshooting of security groups and NACLs.
  33. 33. BOSTON 10-11 SEPT 2018 Security Monkey – Let’s build this • Now for the interesting stuff. • Let’s install this thing. • Pull all the files from Git… • Oops, in my enthusiasm I ran the commands for GCP and Openstack.
  34. 34. BOSTON 10-11 SEPT 2018 Security Monkey - Why doesn’t my instance recognise loopback? • All going well until sudo keeps failing. • My instance does not know it’s own loopback. • Bad Ubuntu! • Change to /etc/hosts fixed this.
  35. 35. BOSTON 10-11 SEPT 2018 Security Monkey - Python isn’t working • When a guide posts something like this, you should probably pay attention to it: • Because when you don’t, you get this:
  36. 36. BOSTON 10-11 SEPT 2018 Security Monkey - Je ne parle pas anglais. • Running in the virtual environment shell now, my bad. • Run the commands to compile the web interface. • Isn’t this installed by default?! • This makes no sense. • Rage level getting critical at this point. • Accurate representation of my face.
  37. 37. BOSTON 10-11 SEPT 2018 Security Monkey - Who doesn’t love a 404? • No idea why, but I had to re-generate the en_US locales. • Then, success!
  38. 38. BOSTON 10-11 SEPT 2018 Security Monkey – I spoke too soon • Now everything should be running, right?
  39. 39. BOSTON 10-11 SEPT 2018 Security Monkey – Mysterious directories • NGNIX can’t find the UI pages to load. • Much searching, much raging. • Transpires that the NGNIX location path was incorrect. • Files had been copied as /usr/local/src/security_monkey/security_monkey/static… There it is
  40. 40. BOSTON 10-11 SEPT 2018 Security Monkey – Damn those SSL certs • Generate self-signed SSL certs. • Getting an error from Chrome, success! • STILL GETTING A 404. • Remove SSL from the config, for now. • I appreciate the irony as a security professional.
  41. 41. BOSTON 10-11 SEPT 2018 Security Monkey – Hello web UI! • …aaaaand:
  42. 42. BOSTON 10-11 SEPT 2018 Security Monkey – Dude, where’s my login server? • Pretty sure I’m supposed to have a login screen? • That red error doesn’t look great. • The Googles reveals that file permissions are a common cause of this issue. • Also need to restart the supervisor service.
  43. 43. BOSTON 10-11 SEPT 2018 Security Monkey – Success!
  44. 44. BOSTON 10-11 SEPT 2018 Security Monkey – Production issues • Issue lists aren’t very detailed. • Dashboard scores for the high score view are not update, but show fine on the summary page. • Daily summary emails don’t get sent out.
  45. 45. BOSTON 10-11 SEPT 2018 Security Monkey - Scoreboard • Instructions – 8/10 • Accuracy of instructions – 7/10 • Ease of configuration – 5/10 • Ragequit score – 8/10
  46. 46. BOSTON 10-11 SEPT 2018 Repokid - Qué? • Repokid uses Access Advisor provided by Aardvark to remove permissions granting access to unused services from the inline policies of IAM roles in an AWS account. • “When used together, Aardvark and Repokid help us get closer to the principle of least privilege without sacrificing speed or introducing heavy process.” - Netflix
  47. 47. BOSTON 10-11 SEPT 2018 Repokid - Wow, these instructions are pretty light. • Even by Netflix standards, these are pretty light… • Pull repo from Git. • Create database. • Create IAM roles.
  48. 48. BOSTON 10-11 SEPT 2018 Repokid – The downsides of using a lab • Run out of elastic IPs. • Reassign one, but now my terminal is angry with me. *sigh*
  49. 49. BOSTON 10-11 SEPT 2018 Repokid – More Python woes… • Instance says virtual env isn’t there (apparently). • Instance also says there is no Git. • Fair enough. • Pull Git package. • Try to pull repo from Github. • There’s already a repokid directory?! Me
  50. 50. BOSTON 10-11 SEPT 2018 Repokid – Only one thing to do TERMINATE
  51. 51. BOSTON 10-11 SEPT 2018 Repokid – Git and SSH key troubles • Wash, rinse, repeat the previous slides. • Accessing Git repo…
  52. 52. BOSTON 10-11 SEPT 2018 Repokid – Let’s try that again • Generate fresh SSH keys. • Add to my agent. • Upload to Github.
  53. 53. BOSTON 10-11 SEPT 2018 Repokid – Never mentioned I needed a database • Apparently I need a Dynamo DB. • Use a small local one for dev purposes. • Pull Java packages, etc. to run it. • This seems to be working fine.
  54. 54. BOSTON 10-11 SEPT 2018 Repokid – Readmes with footnotes • The footnotes for Repokid are important. • Describe what roles need to be set up for the instance to work. • Might have been useful further up the document…
  55. 55. BOSTON 10-11 SEPT 2018 Repokid - Fine tuning JSON • Fine tune the JSON config file. • Point at Aardvark, Dynamo DB and IAM role. • Aaaaand…
  56. 56. BOSTON 10-11 SEPT 2018 Repokid – Production issues • role.policies only checks inline policies. Attached policies are ignored. • Generates heaps of alerts/errors in Lightsail. • The advice for the moment is… just put up and shut up. • (unless you’re going to write your own code to fix)
  57. 57. BOSTON 10-11 SEPT 2018 Repokid - Scoreboard • Instructions – 2/10 • Accuracy of instructions – 2/10 • Ease of configuration – 7/10 • Ragequit score – 7/10
  58. 58. BOSTON 10-11 SEPT 2018 And I’m finished!
  59. 59. BOSTON 10-11 SEPT 2018 Lessons learned • Read what instructions you have carefully… • … but don’t be entirely beholden to them. • Get your base packages and dependencies in order with your code. • Have your supporting tools (terminal, Github, etc.) all in order. • Don’t be afraid to try to run things slightly differently if it works better for your environment. • It’s not failing to ask for help if you’re really stuck.
  60. 60. BOSTON 10-11 SEPT 2018 What’s next? • Diffy! • Diffy is a triage tool to help digital forensics and quickly identify compromised hosts on which to focus their response. • Diffy finds outliers among a group of very similar hosts and highlights those for a human investigator, who can then examine those hosts more closely. • So far… so little instruction.
  61. 61. BOSTON 10-11 SEPT 2018 Documents and links • Netflix Open Source Software Center - https://netflix.github.io/ • Netflix tech blog - https://medium.com/netflix-techblog • Netflix Git repository - https://github.com/Netflix • Lyft’s implementation of BLESS - https://www.youtube.com/watch?v=PMlT1raRMA0 • Versent’s saml2aws repository - https://github.com/Versent/saml2aws • Versent’s unicreds repository - https://github.com/Versent/unicreds • Sethkor’s BLESS repository – https://github.com/sethkor/blesskor • Risky Business #486 Repokid episode - https://risky.biz/RB486/ • Netflix Security’s YouTube Channel - https://www.youtube.com/channel/UCCic- LGj5o892PhU_xrWq-g
  62. 62. BOSTON 10-11 SEPT 2018 Thanks for not ragequitting on my talk and going to happy hour. Questions? @_sarahyo

×