Recommandé
Recommandé
Contenu connexe
Tendances
Tendances (20)
Similaire à Hackfest 2019 Talk
Similaire à Hackfest 2019 Talk (20)
Dernier
Dernier (20)
Hackfest 2019 Talk
- 1. Why Johnny Still Can’t Pentest: A Comparative Analysis of Open-source Web Application Vulnerability Scanners @rana__khali By: Rana Khalil
- 2. whoami @rana__khali • Security assessment analyst • Master in Computer Science under the supervision of Dr. Carlisle Adams • Speaker at BSides, ISSA and OWASP Ottawa, and WiCyS
- 3. Let me weave you a tale…. @rana__khali
- 5. @rana__khali Company X You need to develop secure software!
- 6. @rana__khali
- 7. @rana__khali “Commercial scanner X ensures web application security by securing your website and web applications against hacker attacks.” “Commercial scanner Y dead accurate web vulnerability scanner to identify vulnerabilities in your websites…” “Open-source scanner Z provides first-class coverage, vulnerability detection and accuracy for modern web application technologies.” “Open-source scanner W can help you find and validate SQL Injection, Cross-Site Scripting (XSS), inadvertently disclosed sensitive information, and other vulnerabilities.”
- 8. @rana__khali Vulnerabilities? Develop code Run web application vulnerability scanner on application Yes Yaaay! No
- 9. @rana__khali I’m freaking amazing! I can do it all! And that worked out great for Johnny
- 10. @rana__khali And that worked out great for Johnny….until it didn’t Vulnerability #1 Vulnerability #2 Vulnerability #3 ………… Vulnerability ∞ Pentest Findings
- 11. @rana__khali • How do these tools work? • Do they require special configuration? • How much coverage do they achieve? • What vulnerabilities can they find? • What vulnerabilities can they NOT find?
- 12. Let the research begin…. @rana__khali
- 13. How do these scanners work? @rana__khali Web Application Vulnerability Scanners have three modules: Crawler Attacker Analysis *XSS found* *SQLi found* *LFI found* *RFI found*
- 14. How are these scanners used? @rana__khali Option #1: Point and Shoot (PaS) Option #2: Trained/Configured domain-name • Scanner is given only root URL • Default configuration unchanged • Minimal human interference • Manually visit every page of the application in proxy mode. • Change configuration & train scanner
- 15. Tool Selection @rana__khali • Chen’s evaluation • Consultation with penetration testers Name Version License Price Arachni 1.5.1-0.5.12 Arachni Public Source v1.0 N/A Burp Pro 1.7.35 Commercial $349/year Skipfish 2.10b Apache v2.0 N/A Vega 1.0 MIT N/A Wapiti 3.0.1 GNU GPL v2 N/A ZAP 2.7.0 Apache v2.0 N/A
- 16. Environment Setup @rana__khali Tools Applications * VM restored to initial state before every test run
- 17. If it sounds too good to be true, it probably is.…. @rana__khali
- 18. Vulnerability Detection @rana__khali Vulnerabilities in WackoPicko that were not detected by any scanners: 1. Weak authentication credentials • admin/admin
- 19. Vulnerability Detection @rana__khali Vulnerabilities in WackoPicko that were not detected by any scanners: 2. Parameter Manipulation Sample user: WackoPicko/users/sample.php?userid=1 Real user: WackoPicko/users/sample.php?userid=2
- 20. Vulnerability Detection @rana__khali Vulnerabilities in WackoPicko that were not detected by any scanners: 3. Forceful Browsing • Access to a link that contains a high quality version of a picture without authentication
- 21. Vulnerability Detection @rana__khali Vulnerabilities in WackoPicko that were not detected by any scanners: 4. Logic Flaw • Coupon management functionality
- 22. @rana__khali Note: This slide is shamelessly stolen from David Caissy’s 2017 Appsec Talk. Can scanners catch everything?
- 23. Vulnerability Detection @rana__khali On average scanners found only 40% of the vulnerabilities. 0 10 20 30 40 50 60 70 80 90 100 Arachni Burp Skipfish Wapiti Vega ZAP %ofDetectedVulnerabilities
- 24. Crawling Challenges @rana__khali Features that scanners found difficult to crawl in WackoPicko: 1. Uploading a file • All scanners were not able to upload a picture in PaS mode • Burp and ZAP were able to in Trained mode
- 25. Crawling Challenges @rana__khali Features that scanners found difficult to crawl in WackoPicko: 2. Authentication • All scanners except for Wapiti successfully created accounts • None of the scanners used the created accounts to authenticate Scanner # of Accounts Arachni 202 Burp 113 Skipfish 364 Vega 117 Wapiti 0 ZAP 111
- 26. Crawling Challenges @rana__khali Features that scanners found difficult to crawl in WackoPicko: 3. Multi-step processes • All scanners were not able to complete the process in PaS mode • Burp and ZAP were able to in Trained mode
- 27. Crawling Challenges @rana__khali Features that scanners found difficult to crawl in WackoPicko: 4. Infinite websites • All scanners recognized the infinite loop except Arachni ….. /calendar.php?date=1541454543 /calendar.php?date=1541540943 /calendar.php?date=1541627343
- 28. Crawling Challenges @rana__khali Features that scanners found difficult to crawl in WackoPicko: 5. State awareness • In PaS mode none of the scanners discovered any of the vulnerabilities that require authentication • Vulnerabilities that require authentication were only discovered in Trained mode
- 29. Crawling Challenges @rana__khali Features that scanners found difficult to crawl: 6. Client-side Code • Standard anchor links • Links created dynamically using JavaScript • Multi-page forms • Links in comments • Links embedded in Flash objects • Links within AJAX requests 0 10 20 30 40 50 60 70 80 90 100 Arachni Burp Skipfish Wapiti Vega ZAP %ofWIVETTestsPassed
- 30. What now? Should Johnny even bother using an automated scanner? @rana__khali
- 31. Johnny definitely should! @rana__khali • Scanners DO NOT replace a skilled pentester, but can aid the pentester • Vulnerability scan is NOT EQUIVALENT to a vulnerability assessment • Using a vulnerability scanner requires skill • A fool with a tool is still a fool • Configure your scanner! Never run your scanner in PaS • Specify the target • Set Login / logout conditions • Set the scanner in proxy mode and visit every page of the application • Configure scenarios (business flows) and cleanup b/w scenarios • Monitor and review the requests of your scan • After all that work, you’re only protected against script kiddies
- 32. That’s why Johnny still can’t pentest. @rana__khali
Notes de l'éditeur
- Hi everyone, thank you for attending my talk.
- I’ll start off with a little bit about myself. This is me. Every time I visit a new city, I take this exact picture with my Palestinian flag. This is one of my latest ones and it’s in Beijing on top of the great wall of China. It’s my first time in Quebec, so I can’t wait add another picture to my collection. When I’m not travelling, I’m working. I work as a security assessment analyst and really most of my job is conducting vulnerability assessments on web applications. Super fun. Before that, I did a masters in Computer Science on the topic I’ll be presenting today and that was done under the supervision of Dr. Carlisle Adams. For those that know him, you’ll know that not only is he one of the nicest guys you’ll meet but he’s also one of the smartest guys. He created an algorithm CAST which competed with the Advanced Encryption Standard (AES) before that got standardized. Very interesting stuff. If you’re ever in the same place as him, you should definitely strike up a conversation. Anyways, so I spoke about my research at several places and every time I would present it in the exact way the research was done – so through the methodology that we followed. And by the end of the presentation I felt like I lost most of the audience, so instead
- Today I’m going to narrate it in the form of a story. I had a lot of fun with these slides.
- So this is Johnny. He’s a fresh off the boat recent computer science graduate.
- Who got recently hired at a small start up company as a web developer. And as part of his job he needed to develop secure web applications. Of course, given that the educational system has failed Johnny, he knew how to code but he was never taught secure coding practices. So when his boss asked him to develop secure web apps, he did what every computer science student does when he didn’t know how to do it.
- He googled it.
- And lo and behold he found the answers he was looking for. There was this magical software called a web application vulnerability scanner that crawled your web application and found all the vulnerabilities. And boy did these scanners promise so much. And to top all that off, not only were there commercial ones but there was also open-source ones! Johnny or his company didn’t even have to pay for it.
- So Johnny had this genius idea. All he has to do is develop the code, deploy it, give the web application URL to the scanner and ask it to scan the site. If the scanner doesn’t find any vulnerabilities, then he’s good to go. His application is bullet proof. If it did find vulnerabilities, all he has to do is change his code to fix his code, run the scanner again on the new version of the web application and go through this cycle over and over again until the scanner shows no vulnerabilities.
- And this worked out great for Johnny. Not only was he a great software developer but now he also added security assessment specialist to his title. In fact, this made Johnny wonder why we even pay for pentesters – I mean it’s a relatively simple job.
- However, his whole world came crashing down when the company decided to hire an external company to do a vulnerability assessment and pentest on his web app. The vulnerabilities found were way too many to even count.
- This left Johnny very confused? He ran the web application vulnerability scanner, so why didn’t it report any of these vulnerabilities. So instead of blindly trusting something, Johnny had to ask some difficult questions. Like how do these tools work? Do they require special configuration? How much of the application can they actually cover. What vulnerabilities can they find? And more importantly what vulnerabilities can they not find?
- So Johnny decided to embark on a research journey in hopes of finding the answers he’s looking for.
- The first step was to understand the different components that make up a scanner. At a high-level simplified over view, scanners are made up of 3 components. A crawler module, an attacker module and an analysis module. The crawler module maps the application by taking in a URL or a set of URLs that are treated as roots of the application. It starts at those URLs and follows any reachable links and redirects in the application. The attacker module analyzes the discovered URLs and input vectors from the crawler module and generates attack payloads for each entry point to test for vulnerabilities. The crafted payloads are either predefined values or values that are generated using heuristic algorithms if the scanner is a bit more advanced. Then the analysis module takes the response page from the attacker module and analyzes them to determine if there is a vulnerability. And that’s at a high level overview the different components that make up a scanner.
- Now the next question you should be asking yourself is how are these scanners used. There’s generally two approaches. The first approach is known as point and shoot. This is the approach that is really advertised by vendors. All you have to do is specify the domain you want to test and launch the scanner on that domain. As you can see this requires very minimal human interference – all your doing is pointing the scanner on your application by clicking a button. This unfortunately is how most of these scanners are used and you can’t really bash on the people that use them this way because that’s how the scanners are advertised. The second and better approach is using the scanner in trained or configured mode. What that entails is setting the scanner in proxy mode while you visit every page of the application. For those of you that don’t know what that means, its when the scanner sits in the middle of your browser and your application. So any request you browse to passes through the scanner proxy first and then the application and any response from the application passes first through the scanner proxy and then to your browser. And the reason you would want to set your scanner in proxy mode is so that when you visit a page in the application the link to that page is recorded in the scanner. This way the scanner is aware of all the links in the application. You also would want to change the configuration of the scanner to better fit the needs of your application, train it to recognize the business flows that your application employs. We’ll see examples of that in the upcoming slides. Running the scanners in this mode is not an easy thing to do, and that’s why most people revert to the point and shoot approach.
- Now that we know how the scanners are used, it’s time to decide which scanners to evaluate. The selection was done by reviewing the feature comparison that Shay Chen does every couple of years on about 64 scanners both commercial and opensource. You can see the comparison on his website sectool market. Based on his results, we picked the top 10 performing scanners and then narrowed it down to 6 by asking pentesters which of the 10 are commonly used in corporate environments. We end up with 5 open source – Arachni, skipfish, vega, wapiti and zap and one commercial Burp suite pro – b/c portswigger was nice to give us a license. All the other commercial scanner vendors we asked said no, for obvious reasons that we’ll see in the upcoming slides.
- We then set up a test environment, where we had a VM that was reset before every test run. It contained the scanners we wanted to test and the applications that we used to test the scanners on. The first application WAVSEP is the one that Chen uses to evaluate scanners. We saw that in the previous slide. It contains over a thousand vulnerabilities that fall into several categories. The WIVET application, is a tool that is used to test the crawling coverage of the scanner. It contains