Analyzing the Effectivess of Web Application Firewalls

Larry Suto – November 2011 Analyzing The Effectiveness of Web Application Firewalls

Analyzing the Effectiveness of Web Application
Firewalls
A benchmark study of eight web application firewalls and intrusion prevention systems

by Larry Suto
Application Security Consultant
San Francisco
November 2011

1


Table Of Contents
Analyzing the Effectiveness of Web Application Firewalls
Table Of Contents
1. Executive Summary
2. Key Findings
Overall Findings
Findings: Baseline Tuned
Findings: DAST/Scanner Tuned
3. Evaluation Criteria & Results
Evaluation Notes
Barracuda Networks: Barracuda 660
Citrix: NetScaler NS9.3 Build 48.6.nc
DenyAll: rWeb 4.0 - Core manager v0.3.0.220
F5: ASM 3900
Imperva: SecureSphere 8.0
Trustwave: ModSecurity 2.6.2
SourceFire: SourceFire 3D Sensor 2100 - 4.9.1.6 build 150
Unnamed IPS
4. Conclusions
5. Challenges
False Positive Risk
WAF Evasion
6. Background
7. Methodology
Overview
Constraints and Assumptions
Targets of Attack (TOA)
Testing Tools
Experimental Tests Setup
TEST BL — Baseline Test
Vulnerabilities Included
Configuration Notes
8. Future Directions
9. References
10. Bibliography
11. Acknowledgements
Appendix A: Definition of Terms and Acronyms
Appendix B: Test sites and vulnerability data

2


1. Executive Summary
This study evaluated the effectiveness of Web Application Firewalls (WAFs) and Intrusion Prevention
System (IPS) devices at protecting web applications against external attacks in comparison to each
other. In addition, this study provides general guidelines about the ease of use, and factors affecting the
effectiveness of web application protection solutions.

The study examined the following cross-section of modern WAFs and IPSs both proprietary and open
source. (in alphabetical order):
● WAF’s: Barracuda 660, Citrix NetScaler, DenyAll rWeb, F5 ASM, Imperva SecureSphere,
ModSecurity
● IPS’s: Sourcefire 3D, Unnamed IPS

Each of the eight systems was evaluated in two separate tests. Each of the tests leveraged the devices in
different configurations. The tests were as follows:
1. Baseline Tuned: Tuned configuration requiring 1 day or less of an experienced expert’s time
2. DAST : Trained solutions by application generated filters from Dynamic Application Security
Testing (DAST) software products.

DAST tools perform automated web application vulnerability scans. Some of the DAST solutions now
generate filters which can be imported into WAF and IPS protection policies.The NTODefend DAST
tuning solution was used in this study to generate filters for five of the eight tested platforms. The other
three platforms are not yet supported the DAST tool used in this study. There are other DAST vendors
that have WAF integration such as Cenzic and Whitehat Security but they were not used due to time and
logistical considerations. Future testing should compare these solutions as well.

The following graph (Figure 1) shows a high-level summary of the test results - the % vulnerabilities found
with baseline tuning versus the % vulnerabilities found when tuned with DAST generated filters.

Figure 1

3


The following table shows the same results of both tests in a table format and includes both the actual
numeric results as well as the percentages.

Figure 2

2. Key Findings
Overall Findings
The most significant conclusions of the study are as follows:

● To maximize effectiveness, WAF solutions must be tuned by a trained professional. The
study found that basic tuning takes an average 3.5 hours by a informed user or expert. (see
Figure 6)

● For the solutions supported, there was an average improvement of 36% more vulnerabilities
blocked when the DAST generated filters were applied. (see Figure 3 below)

● IPS solutions are not designed for and are not very effective at blocking web application
vulnerabilities with their default settings. However; when tuned by DAST tool generated filters,
IPS solutions can be as or more effective than several of the WAF’s. (see Figure 3 below)

NTODefend does not yet support for Barracuda, Citrix, or F5 Figure 3

Additional general conclusions from this study are as follows:

● A web application firewall is a recommended element of the security defense strategy.
● WAF and IPS solutions are both capable of providing effective protection, but neither solution is
a sufficient application defense strategy in isolation. They need to be accompanied by solutions
& processes such as DAST and Static Analysis Security Testing (SAST) tools, risk management
etc.
● Organizations should understand and test the differences of their solution’s detection and
blocking effectiveness before and after training with DAST generated filters.
● Rule creation requires detailed knowledge of both the application and the WAF or IPS solution.
It is not practical for a general user to produce effective filters in a reasonable amount of time.
Using an automated solution made the rules generation part of the experiment manageable,
otherwise any manual attempt at generation of such rules would require a deep understanding
of regular expressions, several target systems such as SQL databases, and a deep consistency
across all rules to ensure that they do not violate each other, etc.
● Training with a DAST tool as only as effective as the tool’s output. Care should be taken that the
tool has a quality ruleset and that false positives have been remediated.

4


● DAST tuning solutions can significantly improve the effectiveness of WAFs if the WAFs allow the
DAST to create new customer rules (e.g. Sourcefire and ModSecurity). Solutions that only allow
DAST solutions to turn on existing proprietary WAF rules (e.g. Imperva) do not benefit from such
improvement. The improvement of the WAFs and IPS solutions is, however, dependent on the
quality of the DAST tuning system.

Findings: Baseline Tuned
From the first test, it became clear that considerable vendor-specific knowledge and understanding is
required to properly configure and tune the WAF and IPS solutions. The key findings from the first test
where the solutions were tuned by an experienced expert were as follows:

● The WAFs are fairly effective at detecting and defending web attacks on their own; the most
effective solution found 88% of the vulnerabilities known in the test applications while the
average effectiveness across all WAF solutions was 62%.
● The IPS solutions are not designed to protect web applications and were not very effective at
defending them during this test.

Findings: DAST/Scanner Tuned
Test two yielded the following results when WAF and IPS solutions were trained by DAST generated
filters. (See Figure 3 below)

● The IPS solutions improved by an average of 60% bringing up their performance at-par or
better than the trained/configured WAFs; with their overall blocking effectiveness averaging
82%.

● In this study, the supported WAF’s that were tuned with DAST solution improved an average of
19% from their baseline tuned state.

3. Evaluation Criteria & Results
Throughout the study, results and information were captured according to the study’s methodology. Three
criteria were defined by which to score the solutions, then the weighting and specific scoring guidelines
were defined for each criteria. The following table details the evaluation criteria and their weightings:

Figure 4

5


The results closely matched the results from the percentage blocked scores

Figure 5

The following table details the summary of the results from both tests:

Figure 6

Evaluation Notes
List of the defensive products used and experiences using them for the testing.

Barracuda Networks: Barracuda 660
● Pros: The WAF is fairly full featured and has a responsive and well laid out web GUI. Basic
signatures are easy to enable. It offers above average protection against attacks. With some
work, advanced signatures can be added.
● Cons: Advanced tuning and signature enablement is a bit counterintuitive and not easy to enable
or execute. The signature strategy seems to be focused mostly on regular expressions which
sometimes are difficult to get right. The default signatures/basic tuning processes were fairly
effective but the strict mode did not seem to improve blocking accuracy. Learning mode is not
recommended out of the box because of performance issues.
● Support: Barracuda’s support is very responsive and adequately addressed all operational
issues. They assisted in turning on all the default necessary signatures and demonstrated how to
enable and apply the stricter signatures as well.

6


Citrix: NetScaler NS9.3 Build 48.6.nc
● Pros: The evaluation version of the Netscaler platinum is full featured and includes the web
application firewall feature. The setup was straightforward. Accuracy of blocking was decent in
the tests.
● Cons: Management interface could be a little more intuitive. Policy configuration and
management could be better.
● Support: Relatively little support was required to perform the setup.

DenyAll: rWeb 4.0 - Core manager v0.3.0.220
● Pros: The web interface is easy to use, and the default blacklist is easy to configure.
● Cons: While the interface does allow for custom rules, it does not have a method to optimally
import a custom blacklist such as from NTODefend.
● Support: DenyAll support was very helpful in providing instructions to install a custom blacklist
using the command line interface.

F5: ASM 3900
● Pros: Excellent appliance quality. Great intuitive and snappy web based interface. Once basic
configuration is accomplished, there are a lot of options for tuning the protection policies. With
basic tuning F5 performed very well in the tests. The F5 sits on a very advance load balancing
platform and provides other advanced security features such as DDOS protection. It is supposed
to do well against HTTP resource exhaustion attacks.
● Cons: For users not familiar with the F5 product line, the setup can be quite challenging.
● Support: F5 was very helpful and gave a comprehensive walk-through of the tool including the
basic setup and policy configuration.

Imperva: SecureSphere 8.0
● Pros: Imperva has a high quality protection engine with a very robust set of basic policies. It is
also one of the easier WAFs to configure and provides the most value straight out of the box.
● Cons: The Imperva UI while providing a lot of information is a bit difficult to navigate. Modifying
existing policies to handle exceptions can be unwieldy. Creating custom policies can be difficult.
System can be be chatty with false positives during learning.
● Support: Imperva was very helpful in assisting with setting up the appliance and ensuring that
everything was properly configured.The recommended implementation is to allow the appliance
to collect traffic to develop and application profile before stricter rules are applied. Thus there
is a bit of a grey area between what a default policy and a stricter applied policy might be.
Nevertheless, after turning on a reasonable amount of the policies, it became clear that it was a
very effective solution.

Trustwave: ModSecurity 2.6.2
● Pros: Free open sourced software makes it lowest acquisition cost. Easy to integrate with
Apache. An expert user can modify filters to accomplish nearly any task
● Cons: Limited only to Apache users, and is installed on the application server, thereby limiting
scalability options. There is no user interface. Requires command-line activity and numerous
config files.
● Support: Did not use anything other than the projects forums and mailing lists. Live support is
purchased separately from 3rd party consulting companies.

7


SourceFire: SourceFire 3D Sensor 2100 - 4.9.1.6 build 150
● Pros: The web interface is very full featured, and importing/enabling custom rules is easy to
accomplish. SourceFire supports the SourceFire/NTODefend solution as a WAF for PCI section
6.6 compliance.
● Cons: Not useful on its own to function as a WAF. Had some troubles with clearing rules and
reloading them, but their support team was able to clear up the problem quickly.
● Support: Sourcefire was very helpful when needed. There is also a very active community of
support around the snort project which includes volunteers as well as Sourcefire paid staff.

Unnamed IPS
This IPS vendor did not give authorization to be named in this study. The IPS solution supports Snort 2.8
rules, which was used for this testing.
● Pros: The UI was very compact, appealing, and visually oriented.
● Cons: Not useful on its own to function as a WAF.
● Support: Their support team was very helpful in overcoming the few configuration difficulties
encountered.

4. Conclusions
WAFs are a controversial technology. While some security professionals argue that they are easily
avoided and the only remediation to web application vulnerabilities is to fix the code [7], others argue the
opposite [6]. Manual code reviews and code fixes are essential, however they take considerable amount
of time: days, weeks or even months once a vulnerability becomes known to the developers. In case
where users do not have access to source code for any number of reasons, (3rd party software, legacy
software, external service etc.) there is no manual recourse for fixing bugs. For some organizations,
contractual issues and elaborate change control processes means that known vulnerable applications
can and do long periods without being remediated. This problem, where for a period of time a vulnerability
is known, but not fixed at the source, is addressed by "well-trained" defensive tools like WAFs and IPS’s
where a vulnerability can be effectively patched while the code is being repaired.

This study clearly demonstrated that using WAFs/IPS devices in default or minimal mode to protect web
applications are often not useful, and WAFs configured with the recommended policies and tuned can
often be bypassed by a moderately skilled hacker or an automated hacking tool. This has been well
documented (WAF Evasion section in this document). What is interesting is that training these tools with
a DAST filter generator can dramatically increase their effectiveness and make them a far more useful
part of an enterprise's application security strategy. Additionally, trained IPS devices can be effective
at protecting Web Applications. And finally are we staring at the abyss of a flawed model at detecting
attacks? If WAFs could be made more intelligent somehow the future is bright for this technology.

5. Challenges
The WAF vendors have spent a significant amount of time making claims of one sort or another as to why
their technology is better, and this data may not speak to all of the various claims and may not be fair to
some of the individual strengths of each technology. However, it is the fairest and most impartial model
that the study could come up with to test these technologies in a vendor-neutral way. It is also important
to note that while the tests are complete as far as they go, that this test is not an exhaustive study and
only begins to scratch the surface of the problem.

In particular, WAF evasion research was completely left out, due to time constraints and to keep the
scope of the study reasonable. Even with this limited level of testing, several conclusions can be made
without doubt. For instance, it is almost certain that even with a reasonable amount of learning and tuning
most WAFs will miss well-know and well understood vulnerabilities. Suffice it to say that the problem
escalates polynomially once we start throwing in complications due to AJAX/JSON type sites or various
Java, ActiveX and Flash plugins.

8


False Positive Risk
Although there was not time to complete a comprehensive study of false positives (the blocking of good
traffic) A limited amount of effort was used to try an eliminated false positives, but in future tests this area
should have more attention. The primary focus in this study was blocking of known vulnerabilities.

WAF Evasion
After a cursory review it can be discovered that there are numerous examples in the field where it
becomes obvious that the status-quo solutions to run-time attack detection and containment technology
are routinely by-passed. At the time of the writing of this paper, the following examples were retrieved
from the Web:

● Example 1: ModSecurity SQL Injection challenge held by Trustwave's SpiderLabs. In this
challenge the ModSecurity WAF was tuned and set up to protect a set of four of the scanner
vendor vulnerable websites. Coincidentally, this was a subset of the flawed websites used in the
current study.

So lets take a look at what happened. In one of the successful bypasses of the ModSecurity
inbound CRS SQL Injection rules, an individual by the name of Johannes Dahse used a
combination of MySQL comments and new line characters to evade the CRS SQL Injection
filters.

The problem was not in the actual filter itself, which used regular expressions by the way, but
in the transformation function -t:replaceComments (Modsecurity transformation functions are a
pipeline of normalization functions that modify the input so that the rule can detect the attack).
In this case -treplaceComments altered the input (by ignoring standalone termination */ and
simultaneously replacing unterminated /* comments with a space( 0x20)) so that the filter could
not recognize it as an attack. In the end it was not possible to create a rule to remove SQL
comments reliably so another rule with a complex regular expression was created instead to
detect the presence of SQL comments themselves.

● Example 2: The second example comes of failed context-free matching techniques comes to us
from a paper from the recent Defcon 19. The talk discussed the paper titled "Bypassing PHPIDS
0.6.5." One of the attacks presented bypassed "ALL" the PHPIDS rules as of 0.6.5. How could
of this happened you ask? The flaw was the incorrect assumption that "attacks can never be
repetitive". The heart of the problem was the following bad expression:
Line 90 in ./lib/IDS/Converter.php
(?:(.{2, }) 1{32,})
The regular expression matches any two characters which are repeated 33 or more times.The
first match is a back reference created with the 1 and then 32 additional matches are required to
trigger the regex. Even though the attack string “<script>alert(1)</script> is matched by 4 filter
sets, if it is repeated 33 times it will bypass PHPIDS entirely.

● Example 3: JSReg issues has had issues as well. The details are here "http://code.google.com/
p/jsreg/wiki/Exploits", Not notice that one of them is a regular expression rewrite error: "the failure
to strip the single line comment which then fools the regex rule into thinking that the code is a
regex object and not function calls".

The commonality between these examples is that all them are using pattern matching—in particular
regular expression type of context free matching—with a little bit of AI and boundary restriction of
parameter values thrown in to detect attacks. We have already seen that pattern matching does not yield
optimum results elsewhere in the industry, I refer to Anti-virus applications of signature detection, where
the upper bound of successful detection seems unable to cross the 60% mark. I think that a defense

9


mechanism that can only protect against 70-80% of the attacks is like a house that’s missing a wall. We
have a long way to go, and while the state-of-art against attackers is not where we need it to be, I think
that next section discusses some future directions that have promise.

6. Background
When comparing web application security solutions amongst each other for the purpose of assessing
which one to choose and why consumers are faced with several difficulties. These difficulties arise
from the inability to compare apples to apples when it comes to comparing Web Application Firewalls
(WAFs), network Intrusion Prevention Systems (IPSs), and similar defense technologies that purport
to safeguard against Web attacks [1]. Each Web application firewall vendor delivers different strengths
and weaknesses, and presents varying philosophies of how Web attacks should be defended against
[2]. Security certification standards such as Common Criteria, PCI DSS 6.6, DIACAP etc. focus on the
theoretical for the most part, and do not provide a reasonable guideline for comparison, despite their
scorecard nature [1, 2, 3].

Security pundits are at odds with each other on whether the problem can be solved using fully automatic
techniques such as WAFs and IPSs or is it only solvable using partial automated tools such as static-
analysis tools in conjunction with manual code review and remediation strategies. There is a list of
do’s and don’ts; with advice like “DO consider WAF for performance monitoring”; “DON’T think it’s set
and forget” [1]. Non-profit consortia provide an invaluable resource to the consumer, however they are
very sensitive to vendor-neutrality, and their core approach of staying that way is to keep out of the
comparative-studies field all together. It remains difficult to find standards that provide direct comparative
quantitative measures, and especially difficult to find ones that can explain it to average users of this
technology. So, as things stand, it falls on to the consumer to fund an internal study on some standards
basis.

The most promising guide for end-users seems to conduct such a study seems to be The Web
Application Firewall Evaluation Criteria (WAFEC) and its corresponding WAFEC Evaluation Response
Matrix. However, WAFEC only provides only qualitative comparative analysis between potential
candidate solutions. They still require extensive knowledge of each aspect of web technology and this
author found that Evaluation Response Matrix has no mention of relative importance of each criterion, nor
any method of how to measure evaluate over-all effectiveness of the targets of evaluation [5,6].

Previous research on WAFs have been confined to vendor organizations and customer evaluation
engagements. There have been some publications discussing the functionality of WAFs but most have
been limited to discussions on defining WAF functionality and providing advice on what keep in mind
when conducting private evaluations and deployments. See the appendix for some references.

To the best of the author’s knowledge this is the first study to conduct a general purpose evaluation of
WAF and IPS systems’ ability to protect web applications using a uniform testbed with a simple and easily
reproducible methodology.

An interesting thing to keep in mind is that in the literature a WAF is referred to as security mechanism
with an intimate understanding of the nature of HTTP traffic and web application vulnerabilities. The
consensus is that a WAF is an HTTP focused implementation of general purpose IPS technology.

This study is also one of the first to provide clear empirical evidence that without specialized configuration
and tuning, a general purpose IPS provides limited protection against web application attacks.

10


7. Methodology
Overview
The study tested each solution against the same set of websites and web application prototypes to
ensure that the experiments were instantiated against well-known and well understood vulnerabilities.
This provided a uniform baseline for comparison.

This experimental study provides a quantitative comparison between various web application security
solutions that include both Web Application Firewalls and Intrusion Prevention Systems and evaluates
their relative effectiveness in detecting, reporting and thwarting web attacks. The purpose of this study is
to create a foundation for such comparative studies.

Constraints and Assumptions
The experiments in the study were designed with following constraints and assumptions:

● A typical end-user (hereinafter user) wants to protect a vulnerable web application or website,
hereinafter called the security target.
● The user can detect vulnerabilities within the security target, using an automated tool.
● The user, while knowledgeable about the Web application security principles, is not familiar with
any vendor specific details for the products being evaluated.
● The user does not fully understand the underlying technological details such-as deep
understanding of HTTP protocol negotiations.
● The user only has limited time and budget to perform these evaluations.
● This study is not an exhaustive dissertation of various advanced capabilities of the evaluated web
application firewalls and intrusion prevention systems.

Targets of Attack (TOA)
The study is a comparative analysis of primary results from a set of controlled tests conducted on a
collection of vulnerable site. The vulnerable site used is called Target of Attack (TOA). The following
TOAs were used in these experiments:

● Acunetix AspNet - http://testaspnet.vulnweb.com
● Acunetix TestPHP - http://testphp.vulnweb.com
● Cenzic Crackme - http://crackme.cenzic.com
● HP zerowebappsecurity - http://zero.webappsecurity.com
● IBM Testfire - http://demo.testfire.net
● NTO Web scan test - http://www.webscantest.com

These sites demonstrate manifestations of well known and well understood vulnerabilities.

Testing Tools
The study required use of at least one DAST product capable of creating filters for as many as possible of
the above listed defensive products.

The primary DAST products (vulnerability scanners) used were NT OBJECTive’s NTOSpider and
Mavituna’s Netsparker. NT OBJECTive’s NTODefend product was used for filter generation.

There are other DAST vendors that have WAF integration such as Cenzic and Whitehat Security, and
stand alone applications such as ThreadFix (previously called Vulnerability Manager) from Denim Group.

11


NT OBJECTive’s NTODefend product includes a QuickScan feature which tests for false positives (the
blocking of good traffic) by sending both legitimate and attack payloads to confirm that only attack traffic
is being blocked this was only used on two sites as a sanity check but expanding false positive would be
important in future tests.
The other tools and features were not used due to time and logistical considerations. Future testing
should compare these solutions as well.

Experimental Tests Setup
The evaluation tests used the DAST scanners independently, as shown in Figure 1 to detect exploitable
vulnerabilities in the TOAs. The study is comprised of the following tests:

TEST BL — Baseline Test
A security scan the security targets of vulnerable systems for bugs using two These Scanners are the
used for subsequent tests for each of the Evaluation Target WAF or IPS.

Figure 7

Figure 1 (above) depicts the WAF setup as it relates to the scanner and protected application location for
Tests I and II.

TEST I Scanning the security targets with a WAF/IPS in front of the vulnerable applications. In this test
the WAF/IPS is configured to protect the application with approximately of one half day of effort. Vendor
requested assistance is provided as necessary to ensure that the recommended protection policies are in
place. This is the Tuned/Trained mode.

TEST II Scanning the security targets with a WAF in front of the vulnerable applications. In this test the
WAF is configured to protect the application with a minimum of one half day effort. Vendor requested
assistance is provided as necessary to ensure that the recommended protection policies are in place. A
third-party rule generator is then used to import policies in order to improve the blocking capability of the
system. This is the DAST or Scanner Trained mode.

Vulnerabilities Included
Each WAF/IPS is different in its specific methodology for finding and preventing attacks. For the purposes
of this report, the types of vulnerabilities that were counted were those that are useful against custom
applications, and which most users care about. These are:

● SQL Injection / Blind SQL Injection
● Cross Site Scripting / Persistent Cross Site Scripting
● Command Injection
● CSRF / HTTP Response Splitting
● Arbitrary File Upload attacks
● Remote File Include (PHP Code Injection)

12


The vulnerabilities were recorded to cross reference which WAF's blocked which vulnerabilities in a
simple format to track and compare the results.

A complete listing of vulnerabilities blocked by each WAF was recorded to track the effectiveness of each
technology. This resulted in a fairly high degree of confidence that the extent of Blocked and Missed
numbers are complete.

Configuration Notes
For tuning and training the basic policy was enabled plus any settings that were recommended by the
vendor in order to optimally configure the system in a reasonably effective protection mode. For many of
the systems, I received direct assistance from the vendor in order to accomplish this.

To then see how well customized training from an outside source would improve the protection, custom
filters/rules were generated using the DAST tuning solution for each supported WAF and loaded them in
and repeated the test. Again, this was done mostly when it was obvious that the basic tuning process of
the WAF was going to involve a significant effort. The assumption was that protection would improve with
training, and this was clearly proven during testing. The collected data is being made freely available for
anyone to review and re-create.

Additionally, it should be understood that more manual configuration effort could likely be used to improve
the results of the WAF's further, but time constraints are always going to be an issue in real world use, as
well as when compiling a study like this.

8. Future Directions
So what are the future directions? Surely we cannot do manual defense. Even today, there is a strong
debate on whether or not programmers can be taught to create flawlessly secure programs [6]. As we
have seen, automated defense systems that perform strictly signature or pattern based detection only
achieve partial success at best, and will probably reach an upper bound that is well below the required
threshold of risk mitigation. So what does it all mean? Is run-time automated defense doomed to failure?
I don’t think so. There are entities and companies doing research in interesting directions in computer-
sciences, mathematics, artificial intelligence and other related areas that may prove to be much more
powerful weapons in fight against attacks. I think that the true solutions lies not just in ever intractable
construction of more and more complex pattern- matching systems, but in approaches that fundamentally
redefine the problem and by doing so discover innovative approaches to solving this very real, very
important issue of protecting Web Application Assets.

13


9. References
[1] Web Application Firewalls; What’s in a name?; Ramon Krikken, Copyright 2009 Burton Group; Presentation at
InfoSec 2009

[2] Web App Firewalls: How to Evaluate, Buy, Implement; Mary Brandel; 2009-06-11; http://www.cio.com.au/article/
307044/web_app_firewalls_how_evaluate_buy_implement

[3] Brian Soliman Blog, Technical Articles, 2011-07-25; Common Criteria (CC) for Information Technology
Security Evaluation; http://bryansoliman.wordpress.com/2011/07/25/common-criteria-cc-for-information-
technology-security-evaluation/

[4] WebAppSec.org; 2006; Web Application Firewall Evaluation Criteria (WAFEC) 1.0; Project Coordinator: Ofre
Shezaf; http://projects.webappsec.org/f/wasc-wafec-v1.0.pdf

[5] WebAppSec.org; 2006; WAFEC Response Matrix; http://projects.webappsec.org/f/
WAFEC_Evaluation_Response_Matrix_1.0.xls

[6] OWASP AppSec USA 2010 Keynote; Application Security — It’s a jungle out there

[7] Time to Automate Web Defenses?; Robert Lemos; 2011-10-25;
http://www.darkreading.com/vulnerability-management/167901026/security/attacks-breaches/231901651/time-to-
automate-web-defenses.html

10. Bibliography
Brian Soliman Blog, Technical Articles, 2011-07-25; Common Criteria (CC) for Information Technology Security
Evaluation http://bryansoliman.wordpress.com/2011/07/25/common-criteria-cc-for-information-technology-security-
evaluation/

Brickman, J. (2010) Common Criteria – a good concept in transformation http://community.ca.com/blogs/iam/
archive/2010/03/25/common-criteria-a-good-concept-in-transformation.aspx

F5 Networks and WhiteHat Security Solutions (2008); Vulnerability Assessment Plus Web Application Firewall
(VA+WAF)

11. Acknowledgements
Thanks to all the WAF and IPS vendors for all their graciousness and patience. Things always seem to
take longer than you expect. The more scrutiny we place our solutions under, the better for everyone.
This is a nascent space with a lot of potential. Special thanks to:

Kasey Cross at Imperva
Ido Berger and Tom Spector at F5 Networks
Grant Murphy at Barracuda Networks
Renaud Bidou at DenyAll

Also I would like to thank NT OBJECTives and Mavituna Security for DAST scanner and filter support.
In addition I would like to thank Amir Khan for helping host the systems and Ahmed Masud for valuable
editorial commentary.

14


Appendix A: Definition of Terms and Acronyms
Blacklist - The list of items, that when matched by a pattern matching system that decides to authorize
an operation, will result in rejection of such an authorization.

Brute-force attack - An attack that simply tries all possible permutations of a given scenario to discover
and exploit any possible vulnerabilities.

Cross-site scripting - The ability to include executable code (typically Javascript code) in non Same-
origin contexts.

Origin - Defined as the tuple (domain name, application layer protocol, port number) for purpose of
resource access control. Resources (objects, methods, instances) can access other resources created
from the same origin. See Same-origin context

Same-origin context - A programmatic boundary in browser-side programming languages that restricts
access to methods, properties and objects across different domains (origins). See also: Origin

Deny-by-default - An authorization paradigm where conditions, input, output and processing actions
must be explicitly allowed (using a Whitelist). The rationale behind deny-by-default paradigm is to
significantly reduce the possibilities of unexpected or malicious behavior.

Denial of Service attack - Any attack that affects the availability and normal usage of a service.

DoS - Denial of Service: See also Denial of Service Attack

DDoS - Distributed Denial of Service. This variation uses multiple clients to perform attacks: See also
Denial of Service Attack

Fuzzing - See fuzz testing

Fuzz Testing - Fuzz testing or fuzzing is a software testing technique that applies random unexpected
data to inputs of a computer program. The program is then monitored for unanticipated behavior.

HTTP Response Splitting - An attacker's ability to send a single HTTP request that forces the web
server to form an output stream, which is then interpreted by the target as two HTTP responses instead
of one response. This type of vulnerability can be exploited to perform several web application based
attacks.

OS Command Injection - An injection attack that utilizes the lack of input validation on web applications
that execute out to command line applications. An attacker can use such a vulnerability to execute
arbitrary programs on the host operating system.

SQL injection attack - SQL injection attack utilizes the lack of input validation vulnerability of web data
that an application uses to build a SQL query string. An attacker can use such a vulnerability construct an
input in such a way as to execute arbitrary code on the back-end database.

Whitelist - See Deny-by-default

XSS - See Cross-site Scripting

15


Appendix B: Test sites and vulnerability data
Summary of raw results

16


Acunetix AspNet - http://testaspnet.vulnweb.com

17


Acunetix TestPHP - http://testphp.vulnweb.com

18


Cenzic Crackme - http://crackme.cenzic.com

19


HP zerowebappsecurity - http://zero.webappsecurity.com

20


IBM Testfire - http://demo.testfire.net

21


NTO Web Scan Test - http://www.webscantest.com

22

Analyzing the Effectivess of Web Application Firewalls

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

En vedette

En vedette (8)

Similaire à Analyzing the Effectivess of Web Application Firewalls

Similaire à Analyzing the Effectivess of Web Application Firewalls (20)

Dernier

Dernier (20)

Analyzing the Effectivess of Web Application Firewalls