This document discusses using fuzzing to generate tests for RISC-V compliance testing. It proposes extending an LLVM-based fuzzer with custom mutators and coverage metrics tailored for RISC-V. Experimental results found bugs in several RISC-V simulators, demonstrating the effectiveness of fuzzing for negative compliance testing. The approach generates platform-independent assembly tests and filters invalid tests. It leverages an open-source RISC-V virtual prototype for test execution.

1. 12. November 2020 1
Closing the RISC-V Compliance Gap via Fuzzing
Vladimir Herdt1,2, Daniel Große2,3, Rolf Drechsler1,2
VerSys
01IW19001 16ME0127
1Group of Computer Architecture, University of Bremen, Germany
2Cyber-Physical Systems, DFKI GmbH, Bremen, Germany
3Institute for Complex Systems, Johannes Kepler University Linz, Austria
vherdt@uni-bremen.de
University of Bremen - Computer Architecture

2. 12. November 2020 2
Overview
• RISC-V Compliance Testing
• Our Contribution
 RISC-V Compliance Testing via Fuzzing-based Negative Testing
 Experimental Results
• Wrap-Up

3. 12. November 2020 3
What is Compliance Testing?
• Show that processor core / ISS adheres to ISA specification
• Compliance Testing ≠ (Formal) Verification
• Compliance checks:
 Available registers, their widths, access combinations, correct sign
extension, etc.
 Available instructions and basic sanity checks (e.g. ADD performs an
addition not a subtraction)
 No additional behavior accidentally added
• Compliance ensures: Compatibility with RISC-V SW ecosystem

4. 12. November 2020 4
The Challenge of RISC-V Compliance
“RISC-V is an open-source standard ISA with exceptional modularity and
extensibility. Anyone can build an implementation and there are no license fees
[...]. Implementers are free to add custom extensions to boost capabilities and
performance, while at the same time don’t have to include features that aren’t
needed.” Allen Baum, chair of the RISC-V Compliance Task Group
An increasingly large number of customized RISC-V cores is being implemented
Unconstrained flexibility greatly increases risk of introducing incompatibilities
source: https://semiengineering.com/toward-risc-v-compliance/, by Brian Bailey, 2019
Compliance Testing becomes very important and challenging

5. 12. November 2020 5
A Path Towards RISC-V Compliance Testing
• Not clear how to best approach the
problem
• Current official „solution“:
 Provide compliance test-suite (for
each RISC-V ISA part)
 Problem: significant manual effort
& difficult to get high coverage
& negative testing neglected
• Compliance testing format
 Test-cases provided in assembly
 Platform independent
Test
Outputs
Core/Sim.
Under Test
Reference
Outputs
Reference
Simulator
Compliance
Test-Suite
need to run
only once
per test-case
signature
check against
ref. output
Official RISC-V Compliance Testing
=
?
https://github.com/riscv/riscv-compliance
assembly
format

6. 12. November 2020 6
Our RISC-V Compliance Testing Approaches
a) Specification-based test-suite generation (positive testing):
Increased RISC-V ISA coverage from 67% to 86%
V. Herdt, D. Große, and R. Drechsler. Towards specification and testing of RISC-V
ISA compliance. In DATE, 2020.
b) Mutation-based test-suite generation (positive testing):
Found gaps in the testing effort by symbolic execution techniques
V. Herdt, S. Tempel, D. Große, and R. Drechsler. Mutation-based compliance
testing for RISC-V. In ASP-DAC, 2021.
c) Fuzzing-based test-suite generation (negative testing):
Found bugs/mismatches in several RISC-V simulators
V. Herdt, D. Große, and R. Drechsler. Closing the RISC-V compliance gap: Looking
from the negative testing side. In DAC, 2020.

7. 12. November 2020 7
Fuzzing-based Compliance Testing
Fuzzer
+ custom
mutators
Input
(Instructions)
Simulator
+ custom
coverage
Test Template
(Assembly)
Test Template
(Binary)
Coverage
Compliance
Testsuite
Fuzzing-based
Generation
Input
(Instructions)
Filter
generate execute
report execution coverage
compile
pre-load
collect on
new coverage
combine
automated
and strong
negative testing
Leverage LLVM-based „libFuzzer“

8. 12. November 2020 8
Test Generation Filter
• Compliance tests are written in platform independent assembly
 „Memory layout“ defined by platform dependent macros
 Thus, cannot use hardcoded memory/jump addresses
• Filter drops „invalid“ tests:
 Unsupported instructions (e.g. WFI, CSR, MRET)
 Infinite loops
 Hardcoded jumps or memory accesses
• Conservative Static Analysis
• All illegal instructions are kept
 Their semantic is clearly specified
 Retain negative testing capability
ADD
BEQ
ILLEGAL
JMP
XOR
pass
pass

9. 12. November 2020 9
Extension 1: Custom Mutator
• Fuzzing is random, without domain specific knowledge
• Good idea/intelligent?
-> Explain the fuzzer what is the instruction format
-> Generate longer sequences of valid instructions
RISC-V instruction format, source: official RISC-V ISA spec.
instruction opcode

10. 12. November 2020 10
Custom Mutator Extension
1. Inject random instruction opcode
 Register and immediate values stay random
2. Inject random instruction sequence
 e.g. load large immediate: two subsequent instructions with same RD
but random value
fuzzer generated bytestream (interpreted as instruction list):
inject position
inject position
imm[11:0] RS1 RD
1. inject ADDI opcode
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx…
xxxxxxxxxxxxxxxxx000xxxxx0010011xxxxxxxxxxxxxxxxxxxx…
Custom mutator chosen
with same probability as
built-in fuzzer mutators

11. 12. November 2020 11
Extension 2: Custom Coverage
• Code (Branch) Coverage
 covers switch-case based on opcode in ISS
 tracing instrumentation performed by clang
• Enough for ISS verification?
−> stronger coverage metric tailored for ISS verification is useful
31 20 19 15 14 12 11 7 6 0
I_imm[11:0] rs1 000 rd op=0010011
ADDI (Add Immediate): R[rd] = R[rs1] + I_imm

12. 12. November 2020 12
Coverage Metric Extension
• Functional Coverage
 R1: RD=0, RD≠0
 R2: RD=RS1, RD≠RS1
 R3: similar to R2 but for three registers
 vRx: register x (RD, RS1, RS2) value within range
{REG_MIN, -1, 0, 1, REG_MAX}
 vImm: immediate value within range
{IMM_MIN, -1, 0, 1, IMM_MAX}
• Trace instrumentation
 Add trace function before/after instruction execution in ISS
 Provide: register values and instruction (including decoded opcode)

13. 12. November 2020 13
Open Source RISC-V Virtual Prototype (VP)
• 32 & 64 bit core (RV32/64GC+SUN)
 Instruction Set Simulator (ISS)
• Implemented in C++ with SystemC and TLM-2.0
 approx. 25k LOC (w/o comments, blanks)
• Open Source (MIT License) via GitHub
 http://www.systemc-verification.org/riscv-vp
 Features: Eclipse-based debugging, Linux support, SiFive
HiFive1 platform, etc.
Foundation for the
fuzzing process

14. 12. November 2020 14
Experimental Evaluation
• Linux system with an Intel Core i5-7200U
processor
• Test Generation:
 30 minutes runtime
 45K executions per second
 1GB memory consumption
 13.5K tests generated
• Evaluation Results:
 Found bugs in several RISC-V simulator
RISC-V ISA SPIKE VP SAIL GRIFT
RV32I 7 5 crash 124
RV32IMC 9 32 crash 1047
RV32GC 9 / / 141
Number of mismatches against riscvOVPsim
(the reference simulator for RISC-V compliance testing)
riscvOVPsim

15. 12. November 2020 15
Wrap-Up
• Approach for RISC-V Compliance Testing
 Focus on Negative Testing
• Leverage Coverage-guided Fuzzing for Test Generation
 Tailored for RISC-V ISA
 Custom Mutation Procedures and Coverage Metrics
 Filter to Drop Platform Dependent Tests
• Promising Experimental Results
 Found several bugs in RISC-V simulators
• Find more/related information:
http://www.systemc-verification.org/risc-v