1. University of Ottawa | University of Luxembourg
University of Ottawa | University of Luxembourg
1
Nazanin Bayati - Metamorphic Testing for Web System Security
13 September 2023
Journal First – IEEE Transaction on Software Engineering
Presented by: Nazanin Bayati
13 September 2023
University of Ottawa University of Luxembourg
Nazanin Bayati
University of Ottawa
Fabrizio Pastore
University of Luxembourg
Lionel Briand
University of Ottawa
University of Luxembourg
Arda Goknil
SINTEF Digital, Norway
Metamorphic Testing for Web System Security

2. 2
Security vulnerabilities are subtle
Discovered when testing with many inputs
Specifying expected results is infeasible

3. University of Ottawa | University of Luxembourg
University of Ottawa | University of Luxembourg
3
Nazanin Bayati - Metamorphic Testing for Web System Security
13 September 2023
Metamorphic Testing alleviates the Oracle Problem
• Metamorphic Testing (MT) is based on the idea that
• it may be simpler to reason about relations between outputs of multiple test executions,
called Metamorphic Relations (MRs), than to specify the output of the system for a given
input
• In MT, system properties are captured as MRs that
• specify how to automatically transform an initial set of test inputs (source inputs) into
follow-up test inputs
• specify the relation between the outputs obtained from source and follow-up inputs
• A failure is observed when such relations are violated.

4. University of Ottawa | University of Luxembourg
University of Ottawa | University of Luxembourg
4
Nazanin Bayati - Metamorphic Testing for Web System Security
13 September 2023
Metamorphic Security Testing
• Source input: a sequence of valid interactions with the system
{login(Admin), RequestURL(settings_page)}
• Follow-up input: generated by altering valid interactions as an attacker would do
{login(User1), RequestURL(settings_page)}
• Relations: capture properties that hold when the system is not vulnerable
if the user in the follow-up input cannot access the URL from her GUI then the output of the
source and follow-up inputs should be different

5. University of Ottawa | University of Luxembourg
University of Ottawa | University of Luxembourg
5
Nazanin Bayati - Metamorphic Testing for Web System Security
13 September 2023
MST-wi: Metamorphic Security Testing for Web Interfaces
Web System
Execute the Data
Collection Framework
Catalog of 76
Metamorphic Relations
Select or Specify the
Metamorphic Relations
Execute the
Metamorphic Testing
Framework
Test results
Translate Metamorphic
Relations to Java
List of
Metamorphic Relations
Executable
Metamorphic
Relations in Java
Source Inputs
1 2
3 4
Submit
form
logout
Log in
logout
Log in

6. University of Ottawa | University of Luxembourg
University of Ottawa | University of Luxembourg
6
Nazanin Bayati - Metamorphic Testing for Web System Security
13 September 2023
MST-wi – MR Example
• Security issue: Bypass Authorization Schema

7. University of Ottawa | University of Luxembourg
University of Ottawa | University of Luxembourg
7
Nazanin Bayati - Metamorphic Testing for Web System Security
13 September 2023
MST-wi – MR Example
• Security issue: Bypass Authorization Schema

8. University of Ottawa | University of Luxembourg
University of Ottawa | University of Luxembourg
8
Nazanin Bayati - Metamorphic Testing for Web System Security
13 September 2023
MST-wi – MR Example
• Security issue: Bypass Authorization Schema

9. University of Ottawa | University of Luxembourg
University of Ottawa | University of Luxembourg
9
Nazanin Bayati - Metamorphic Testing for Web System Security
13 September 2023
MST-wi – MR Example
• Security issue: Bypass Authorization Schema

10. University of Ottawa | University of Luxembourg
University of Ottawa | University of Luxembourg
10
Nazanin Bayati - Metamorphic Testing for Web System Security
13 September 2023
MST-wi – MR Example
• Security issue: Bypass Authorization Schema

11. University of Ottawa | University of Luxembourg
University of Ottawa | University of Luxembourg
11
Nazanin Bayati - Metamorphic Testing for Web System Security
13 September 2023
MST-wi – MR Example
• Security issue: Bypass Authorization Schema

12. University of Ottawa | University of Luxembourg
University of Ottawa | University of Luxembourg
12
Nazanin Bayati - Metamorphic Testing for Web System Security
13 September 2023
MST-wi – MR Example
• Security issue: Bypass Authorization Schema
Our metamorphic testing algorithm executes
each MR multiple times, to ensure that every
possible combination of source and follow-up
inputs is exercised

13. University of Ottawa | University of Luxembourg
University of Ottawa | University of Luxembourg
13
Nazanin Bayati - Metamorphic Testing for Web System Security
13 September 2023
MST-wi – Research Questions
• RQ1. What testing activities can be automated thanks to oracle automation provided by MST-wi?
• RQ2. What vulnerability types can MST-wi detect?
• RQ3. What testability guidelines can we define to enable effective test automation with MST-wi?
• RQ4. How does MST-wi compare to state-of-the-art SAST and DAST tools?
• RQ5. Can we identify patterns for writing MST-wi relations?
• RQ6. Is MST-wi effective?
• RQ7. Is MST-wi efficient?

14. University of Ottawa | University of Luxembourg
University of Ottawa | University of Luxembourg
14
Nazanin Bayati - Metamorphic Testing for Web System Security
13 September 2023
MST-wi – What vulnerability types can MST-wi detect?
• We investigated the feasibility of implementing MRs that discover the vulnerability types described in the
MITRE Common Weakness Enumeration (CWE) database
• Considered three subsets:
• CWE view for common security architectural tactics
• CWE Top 25 most dangerous software errors
• OWASP Top 10 Web security risks
• To implement an MR, for each weakness, we first inspect its description, its demonstrative examples, the
description of concrete vulnerabilities (CVE) and common attack patterns (CAPEC) associated with the
weakness.
• This process led to a catalog of 76 MRs.

15. University of Ottawa | University of Luxembourg
University of Ottawa | University of Luxembourg
15
Nazanin Bayati - Metamorphic Testing for Web System Security
13 September 2023
MST-wi – What vulnerability types can MST-wi detect?
Security Design Principle Vulnerability types Addressed by MST-wi Rank
Audit 6 1(16%) 10th
Authenticate Actors 28 12 (43%) 4th
Authorize Actors 60 34 (57%) 3rd
Cross Cutting 9 3 (33%) 6th
Encrypt Data 38 8 (21%) 8th
Identify Actors 12 3 (25%) 7th
Limit Access 8 3 (38%) 5th
Limit Exposure 6 0 (0%) 11th
Lock Computer 1 0 (0%) 11th
Manage User Session 6 4 (67%) 2nd
Validate Inputs 39 31 (79%) 1st
Verify Message Integrity 19 2 (20%) 9th
Total 223 101 (45%)
Summary of the CWE architectural security design principles and weaknesses
addressed by MST-wi.

16. University of Ottawa | University of Luxembourg
University of Ottawa | University of Luxembourg
16
Nazanin Bayati - Metamorphic Testing for Web System Security
13 September 2023
MST-wi – How does MST-wi compare to state-of-the-art SAST and DAST tools?
• We compared the vulnerability types detected by MST-wi, with the vulnerability types detected by state-
of-the-art SAST and DAST tool reported in a recent empirical study

17. University of Ottawa | University of Luxembourg
University of Ottawa | University of Luxembourg
17
Nazanin Bayati - Metamorphic Testing for Web System Security
13 September 2023
MST-wi – How does MST-wi compare to state-of-the-art SAST and DAST tools?
Security Design
Principle
Weaknesses Addresses by Weaknesses Addressed by MST but not
by
Weaknesses bot addresses by MST but
addresses by
MST Zap DA2 Sonar SA2 Zap DA2 Sonar SA2 Zap DA2 Sonar SA2
Audit 1 0 0 0 3 1 1 1 0 0 0 0 2
Authenticate Actors 12 0 2 1 9 12 11 11 7 0 1 0 4
Authorize Actors 34 2 0 1 13 32 34 34 25 0 0 1 4
Cross Cutting 3 0 0 2 0 3 3 2 3 0 0 1 0
Encrypt Data 8 2 5 8 10 8 8 7 4 2 5 7 6
Identify Actors 3 1 1 1 7 3 3 3 1 1 1 1 5
Limit Access 3 0 1 1 5 3 3 2 0 0 1 0 2
Limit Exposure 0 1 0 0 1 0 0 0 0 1 0 0 1
Lock Computer 0 0 0 0 0 0 0 0 0 0 0 0 0
Manage User Session 4 0 0 0 2 4 4 4 2 0 0 0 0
Validate Inputs 31 10 7 2 14 24 25 30 19 3 1 1 2
Verify Message Integrity 2 1 0 0 3 2 2 2 1 1 0 0 2
Total 101 17 16 16 67 92 94 96 62 8 9 11 28
84
The set of weaknesses targeted by MST-wi
is larger than what can be targeted by applying
all four competing approaches together.

18. University of Ottawa | University of Luxembourg
University of Ottawa | University of Luxembourg
18
Nazanin Bayati - Metamorphic Testing for Web System Security
13 September 2023
MST-wi – Is MST-wi effective?
Applied MST-wi to test well-known Web systems:
• Jenkins v 2.121
• Joomla v. 3.8.7.
Assessed MST-wi capability to detect known vulnerabilities:
• 11 for Jenkins, 3 for Joomla.
• One of them discovered by MST-wi (CVE-2018-17857)
Considered two setups:
• Derive source inputs with crawler only
• Consider additional manually implemented functional test cases
Metrics:
• Sensitivity: proportion of vulnerabilities identified
• Specificity: proportion of inputs not leading to false alarms

19. University of Ottawa | University of Luxembourg
University of Ottawa | University of Luxembourg
19
Nazanin Bayati - Metamorphic Testing for Web System Security
13 September 2023
MST-wi – Is MST-wi effective?
• The high specificity indicates that only a negligible fraction of follow-up inputs leads to false alarms
• Since sensitivity reflects the fault detection rate (i.e., the proportion of vulnerabilities discovered),
we conclude that our approach is highly effective
• We can discover more than 60% of vulnerabilities in a completely automated manner, using only the crawler
• And up to 85% using both crawler and manual inputs

20. University of Ottawa | University of Luxembourg
University of Ottawa | University of Luxembourg
20
Nazanin Bayati - Metamorphic Testing for Web System Security
13 September 2023
https://github.com/MetamorphicSecurityTesting/MST

21. University of Ottawa | University of Luxembourg
University of Ottawa | University of Luxembourg
21
Nazanin Bayati - Metamorphic Testing for Web System Security
13 September 2023
Metamorphic Testing for Web
System Security
Presented by: Nazanin Bayati
13 September 2023
N. Bayati Chaleshtari, F. Pastore, A. Goknil, and L. Briand, "Metamorphic Testing for Web System Security",
IEEE Transactions on Software Engineering, 2023, https://ieeexplore.ieee.org/document/10089522
n.bayati@uottawa.ca
University of Ottawa University of Luxembourg

22. University of Ottawa | University of Luxembourg
University of Ottawa | University of Luxembourg
23
Nazanin Bayati - Metamorphic Testing for Web System Security
13 September 2023
MST-wi – How does MST-wi compare to state-of-the-art SAST and DAST tools?
Security Design
Principle
Weaknesses Addresses by Weaknesses Addressed by MST but not
by
Weaknesses bot addresses by MST but
addresses by
MST Zap DA2 Sonar SA2 Zap DA2 Sonar SA2 Zap DA2 Sonar SA2
Audit 1 0 0 0 3 1 1 1 0 0 0 0 2
Authenticate Actors 12 0 2 1 9 12 11 11 7 0 1 0 4
Authorize Actors 34 2 0 1 13 32 34 34 25 0 0 1 4
Cross Cutting 3 0 0 2 0 3 3 2 3 0 0 1 0
Encrypt Data 8 2 5 8 10 8 8 7 4 2 5 7 6
Identify Actors 3 1 1 1 7 3 3 3 1 1 1 1 5
Limit Access 3 0 1 1 5 3 3 2 0 0 1 0 2
Limit Exposure 0 1 0 0 1 0 0 0 0 1 0 0 1
Lock Computer 0 0 0 0 0 0 0 0 0 0 0 0 0
Manage User Session 4 0 0 0 2 4 4 4 2 0 0 0 0
Validate Inputs 31 10 7 2 14 24 25 30 19 3 1 1 2
Verify Message Integrity 2 1 0 0 3 2 2 2 1 1 0 0 2
Total 101 17 16 16 67 92 94 96 62 8 9 11 28
56
MST can detect 56 weaknesses that any other approach cannot address.

23. University of Ottawa | University of Luxembourg
University of Ottawa | University of Luxembourg
24
Nazanin Bayati - Metamorphic Testing for Web System Security
13 September 2023
MST-wi – How does MST-wi compare to state-of-the-art SAST and DAST tools?
Security Design
Principle
Weaknesses Addresses by Weaknesses Addressed by MST but not
by
Weaknesses not addresses by MST but
addresses by
MST Zap DA2 Sonar SA2 Zap DA2 Sonar SA2 Zap DA2 Sonar SA2
Audit 1 0 0 0 3 1 1 1 0 0 0 0 2
Authenticate Actors 12 0 2 1 9 12 11 11 7 0 1 0 4
Authorize Actors 34 2 0 1 13 32 34 34 25 0 0 1 4
Cross Cutting 3 0 0 2 0 3 3 2 3 0 0 1 0
Encrypt Data 8 2 5 8 10 8 8 7 4 2 5 7 6
Identify Actors 3 1 1 1 7 3 3 3 1 1 1 1 5
Limit Access 3 0 1 1 5 3 3 2 0 0 1 0 2
Limit Exposure 0 1 0 0 1 0 0 0 0 1 0 0 1
Lock Computer 0 0 0 0 0 0 0 0 0 0 0 0 0
Manage User Session 4 0 0 0 2 4 4 4 2 0 0 0 0
Validate Inputs 31 10 7 2 14 24 25 30 19 3 1 1 2
Verify Message Integrity 2 1 0 0 3 2 2 2 1 1 0 0 2
Total 101 17 16 16 67 92 94 96 62 8 9 11 28
39
Combined together, DAST/SAST approaches can address only 39 weaknesses that
MST-wi does not address.

24. University of Ottawa | University of Luxembourg
University of Ottawa | University of Luxembourg
25
Nazanin Bayati - Metamorphic Testing for Web System Security
13 September 2023
MST-wi – How does MST-wi compare to state-of-the-art SAST and DAST tools?
Security Design
Principle
Weaknesses Addresses by Weaknesses Addressed by MST but not
by
Weaknesses not addresses by MST but
addresses by
MST Zap DA2 Sonar SA2 Zap DA2 Sonar SA2 Zap DA2 Sonar SA2
Audit 1 0 0 0 3 1 1 1 0 0 0 0 2
Authenticate Actors 12 0 2 1 9 12 11 11 7 0 1 0 4
Authorize Actors 34 2 0 1 13 32 34 34 25 0 0 1 4
Cross Cutting 3 0 0 2 0 3 3 2 3 0 0 1 0
Encrypt Data 8 2 5 8 10 8 8 7 4 2 5 7 6
Identify Actors 3 1 1 1 7 3 3 3 1 1 1 1 5
Limit Access 3 0 1 1 5 3 3 2 0 0 1 0 2
Limit Exposure 0 1 0 0 1 0 0 0 0 1 0 0 1
Lock Computer 0 0 0 0 0 0 0 0 0 0 0 0 0
Manage User Session 4 0 0 0 2 4 4 4 2 0 0 0 0
Validate Inputs 31 10 7 2 14 24 25 30 19 3 1 1 2
Verify Message Integrity 2 1 0 0 3 2 2 2 1 1 0 0 2
Total 101 17 16 16 67 92 94 96 62 8 9 11 28
39
Combined together, DAST/SAST approaches can address only 39 weaknesses that
MST-wi does not address.
• The weaknesses that MST-wi cannot address are mostly those
(i) that can only be discovered using program analysis,
(ii) that are not related to user-system interactions, or
(iii) that concern non-Web-based systems.

25. University of Ottawa | University of Luxembourg
University of Ottawa | University of Luxembourg
26
Nazanin Bayati - Metamorphic Testing for Web System Security
13 September 2023
MST-wi – How does MST-wi compare to state-of-the-art SAST and DAST tools?
Security Design
Principle
Weaknesses Addresses by Weaknesses Addressed by MST but not
by
Weaknesses bot addresses by MST but
addresses by
MST Zap DA2 Sonar SA2 Zap DA2 Sonar SA2 Zap DA2 Sonar SA2
Audit 1 0 0 0 3 1 1 1 0 0 0 0 2
Authenticate Actors 12 0 2 1 9 12 11 11 7 0 1 0 4
Authorize Actors 34 2 0 1 13 32 34 34 25 0 0 1 4
Cross Cutting 3 0 0 2 0 3 3 2 3 0 0 1 0
Encrypt Data 8 2 5 8 10 8 8 7 4 2 5 7 6
Identify Actors 3 1 1 1 7 3 3 3 1 1 1 1 5
Limit Access 3 0 1 1 5 3 3 2 0 0 1 0 2
Limit Exposure 0 1 0 0 1 0 0 0 0 1 0 0 1
Lock Computer 0 0 0 0 0 0 0 0 0 0 0 0 0
Manage User Session 4 0 0 0 2 4 4 4 2 0 0 0 0
Validate Inputs 31 10 7 2 14 24 25 30 19 3 1 1 2
Verify Message Integrity 2 1 0 0 3 2 2 2 1 1 0 0 2
Total 101 17 16 16 67 92 94 96 62 8 9 11 28
Combining MST-wi with SA2 seems to be a particularly effective
combination as it enables detecting 129 weaknesses (i.e., 101 + 28),
which is 92% of the 140 weaknesses that can be detected by any approach.

26. University of Ottawa | University of Luxembourg
University of Ottawa | University of Luxembourg
29
Nazanin Bayati - Metamorphic Testing for Web System Security
13 September 2023
MST-wi: Metamorphic Security Testing for Web Interfaces
Web System
Execute the Data
Collection
Framework
List of Predefined
Metamorphic Relations
Select and Specify
the MRs
Execute the
Metamorphic Testing
Framework
Test results
Transform MRs
to Java
List of MRs
Executable
MRs
S(x,y)
Source Inputs

27. University of Ottawa | University of Luxembourg
University of Ottawa | University of Luxembourg
30
Nazanin Bayati - Metamorphic Testing for Web System Security
13 September 2023
MST-wi – Is MST-wi effective?
• The high specificity indicates that only a negligible fraction of follow-up inputs leads to false alarms
• Since sensitivity reflects the fault detection rate (i.e., the proportion of vulnerabilities discovered),
we conclude that our approach is highly effective
• We can discover more than 60% of vulnerabilities in a completely automated manner, using only the crawler
• And up to 85% using both crawler and manual inputs