nullcon 2010 Penetration Testing versus Source Code Review - by Nikhil Wagholikar http://nullcon.net http://null.co.in

1. Penetration Testing versus Source Code Review
1. Penetration Testing vs. Source Code Review
Within the domain of application security assessments, there are multiple approaches
that can be considered. The most common ones are black-box penetration testing,
application design review and source code reviews. Of course, more arcane ones such as
reverse engineering or binary analysis can also be considered in specific situations. But
for the moment, let’s focus on the two most common approaches to application
security assessments – penetration testing and source code review.
Currently, there is a debate raging within the information security community as to
which approach is better, with some participants going so far as to say that penetration
testing is dead! (ref).
With the experience of having performed hundreds of penetration tests and dozens of
source code reviews, I have come to the conclusion that each technique has its own
place. It is also extremely worthwhile for the developer and security communities to be
aware of the advantages and disadvantages of both the techniques. In fact, in my
judgment a well-timed combination of both approaches can be extremely beneficial in
identifying and fixing security issues at the earliest.
Let’s look at some of the issues that we have discovered in penetration tests, which
would be near to impossible to discover in a source code review.
Penetration Testing
Risk-based testing
Nowadays, even penetration tests are becoming risk-based. This means, that we spend
time learning about the business of the client and the risks to the business through the
Internet based infrastructure – primarily web applications – in order to focus our
penetration testing work in line with these risks. A source code review can never
simulate the penetration testing approach, which requires building business-specific test
cases, and then testing for those.
For example, during the penetration test of a local search engine website, the business
risk is not so much from a cross-site scripting, as it would be from someone running
repeated search requests, downloading the HTML results, and then parsing the results
to recreate the database – essentially an online data mining attack.
Similarly, while pen-testing an ERP system, the emphasis would not be so much on
unpatched vulnerabilities, as it would be on executing attacks that would simulate a
financial fraud. For instance, can we violate the existing access controls to not only
accept vendor quotations, but also to approve a selected vendor, under the guise of
Confidential Page 1

2. Penetration Testing versus Source Code Review
another user? So privilege escalation attacks would be of far greater consequence than
regular SQL injection and cross-site scripting attacks.
Search engine indexing faulty
During a recent penetration test, we found a search functionality on the web
application, and upon entering interesting words such as “password” and
“administrator”, some very interesting documents were revealed. Upon further
searching, we discovered that the terms “.jpg”, and “.bmp” revealed scanned copies of
customers’ personal information such as driving licenses and passport copies. Again, it is
difficult to imagine a source code review discovering these sorts of issues.
Weak Authentication Mechanisms
While static source code analysis tools would be able to discover weak authentication
schemes (pages, which allow access without authentication), specific issues such as
weak CAPTCHA implementations, or if the web application allows short and weak
passwords to be selected, could be obtained from a penetration test more easily than
from an automated source code review. A manual source code review would possibly be
able to discover these flaws. Similarly, the ability to enumerate users due to different
error messages (one when the username is incorrect, and another one when the
password is incorrect), is also easily discovered in a penetration test.
In another instance, we discovered that the credit card numbers were being shown up
in plaintext. An automated source code review failed to show up these issues.
{Other points yet to work on:
Configuration issues with the web server and the operating system
Manipulation of business rules, such as for web-based ERPs in terms of
transaction processing controls being violated
Source code not being available, of course means penetration testing is the only
option
Where it is a regulatory requirement – such as for Internet Banking}
Source Code Reviews
Code review is systematic examination of computer source code intended to find and fix
mistakes and security vulnerabilities overlooked in the initial development phase,
improving both the overall quality of software and the developers' skills.
Largely source code developers encounter two typical errors/mistakes/issues/hurdles:
Confidential Page 2

3. Penetration Testing versus Source Code Review
Implementation errors: These quality-style defects in code are fairly atomic, and
typically stand alone when identified, and remediation is applied. They are caused by
bad or "loose" programming practices. Examples include buffer overflows, which result
from mismanagement of memory, and race conditions, which result from call timing
mismatches.
Design errors: These include the failure to utilize or adequately implement security
related functions. This includes authentication, encryption, and the use of insecure
external code types, and validation of data input and application output.
Source code reviews can be done in couple of ways – manual or automated. Even
manual reviews cannot look through each line of code, but are supposed to identify key
functions and then review those. An automated source code review looks through the
call-tree using proprietary algorithms, and offers the following distinct advantages over
a penetration test:
Unlike PT, it’s more of a white box approach to testing the application. This gives
the in-sight of the application security issues.
Identify weak encryption algorithms or weak implementation of strong
encryption algorithms, such as by choosing a shorter key length or insecure
storage of the key
Almost the complete set of cross-site scripting and SQL injection possible
vulnerabilities. A penetration test in contrast ends up finding fewer of these, as
some of them are missed out by both manual as well as automated pen-testing
methods
Getting down to the root-function that is vulnerable. A pen-test will typically
identify the web page or form that is vulnerable, but a source code review will
drill down to the common function or module, which when fixed, would end up
resolving multiple security issues, thus providing for a tremendous savings in
time and cost
With source code reviews being embedded into IDEs, developers can identify
and fix security issues almost immediately. This tremendously reduces the cost
of fixing application security issues.
SCR can also check whether secure coding practices and other development
practices have been followed or not
It could happen that in black box PT, the pen tester might miss out certain
security vulnerabilities in the application. This possibility is very less in SCR, since
the whole source is in-front.
{Other points yet to be added here in support of source code reviews}
Confidential Page 3