An updated version of our Learning on Deep Learning presentation delivered on June 18, 2020 for an IBM Middleware User Group series.

1. Learning about Deep Learning
Applications for OpenJDK verification
@ShelleyMLambert and Longyu Zhang
AdoptOpenJDK, Eclipse OpenJ9, Eclipse OMR Verification

2. Intro & Motivation
• Early days, thought-starter
• Explore and experiment, determine feasibility
• Revive projects in waiting
• Make test better
*from Professor Andrew Ng
AQA
Test
Data
“Virtuous Circle for AI”* as applied to verification
• Create/gather tests
• Run tests
• Gather data from runs
• Make test better

3. What is Deep Learning?
Deep learning is a subset of ML
algorithms distinguished by:
• Loosely based on structure and
function of the brain, use
artificial neural networks (ANN)
• Multiple layers of processing
units, “neurons”, output of a
layer is input to another layer
• Modes of learning, supervised
(regression, classification) or
unsupervised (pattern analysis)

4. Remember Your Math
Computationally Heavy
• Non-linear functions applied at
each layer (sigmoid, tanh, ReLU,
etc)
• Forward/Backward propagation,
derivatives, gradient descent
• Weight adjustment (the action to
improve)
• Human level error
• Training set error
• Dev set error
• Next step is known, when in
doubt add more data
bias
variance

5. The Process
Select problem
Determine features
Gather examples
Learn
Adjust weights (update
model/ heuristic)
Apply deep learning?

6. Guidance for Problem Selection
• Anything that a human can do with a second of
thought, can be automated with AI*
• Where are you data-rich?
• Parked ideas, waiting for the right
tool/approach
• Outputs that help drive next actions
*from Professor Andrew Ng

7. Data-hungry DL
what we can feed it?
• Code reviews
• Static analysis
• Pull requests
• Code coverage values
• Test output
– Verbose console, result status, exceptions, trace info,
benchmark results, GC/JIT logs, cores, instrumented data
• Github issues / cores
• Job schedules, execution times
• Machine config info / status
static
dynamic
peripheral

8. Audience Participation
Categorize [infra, test, jdk]infra

9. Audience Participation
Categorize [infra, test, jdk]test

10. Audience Participation
Categorize [infra, test, jdk]test

11. zlinuxzlinux
QA is Swimming in Data
Test output:
• Vast amounts of data per day:
– 6 impls (openj9/15, hotspot/18, ibm/22, sap/1, corretto/3, upstream/3)
– sum([15,18,22,1,3,3]) = 62 impl_spec value
– 250,000+ unique tests
– 6 versions (8, 9, 10, 11, 12, 13, 14, 15, 16, Valhalla)
– ~36 variants (unique inputs / commandline options)
• Impls_specsTotal x numTests x versions
– 62x250000x6=93,000,000
– With variants -> 93,000,000x36=3,348,000,000 tests run
OpenJ9 Hotspot SAP IBM
8 11 14 15 16 +
osxosx aix win xlinux plinux zlinux
openjdk functional perf system external
Corretto Upstream R.H.
aarch64 riscv
Valhalla
Conservative
estimate, excludes
PR & Docker
image testing

12. Activities (and Questions) Related to Test
Plan Implement Automate Execute
Triage Exclude
Report
What? How?
How
often?
How
easy?
How
few?
How
fast?
What failed?
Why? What
next?
Decompose into a set of services by test activity, services to help answer the questions
and take next actions.

13. A lot of questions.
A lot of data.
What next?

14. Past Prototypes Revisited
ResultAnalytics
Data
Services
UI Layer
Cores
raw refined
custom dashboardother clients
TestGeneration BenchEngineCoreAnalytics TestSelection
BugPrediction
InputOptions ResultSummary
ResultCompare
github
repos
Jenkins
servers
TRSS

15. Core Analytics Service
• visualize & analyze data from cores
• predict crashes based on data mined from core files

16. Bug Prediction Service
• Scores per file based on ‘recent’ changes due to defects (github PRs/issues),
predict based on change & defect history, other features?

17. Input Options Service
• Input Options Service – grabs input options defined in tests
at start of a build, names and stores them (unique sentences
of options), for sharing with other builds/tests, can reply
whether inputs are valid for particular platforms

18. Deep Learning Service
ResultAnalytics
Data
Services
UI Layer
Cores
raw refined
custom dashboardother clients
TestGeneration BenchEngineCoreAnalytics TestSelection
BugPrediction
InputOptions ResultSummary
ResultCompare
github
repos
Jenkins
servers
DL service

19. Areas of Interest
• Test generation (write tests for me)
• Find or predict defects
– fuzz testing to verify compilers, to find security vulnerabilities
– bug prediction
• Triage failures
– Categorization, which component is root cause of failure?
• Next action post-failure
– Binary classifier, is ‘real’ defect or not
• Analyze performance
– Predict if changes will improve performance or not, by how much?
• Optimize machine usage, lab reqs
– Optimize automation, scheduling
– Predict test execution time, predict if a test run will fail
• Replace myself with automation

20. Model Building
Version
Variants used
Things we know
(input layer)
Failure expression
Platform
Impl
Machine ‘age’
Failure age
PR list
Find/predict bugs
Things we want to know
(output layer)
Bug prediction scores
Triage failures
Predict perf
Optimize usage
Write tests
Next best action

21. DL Projects in Progress

22. DeepSmith (CAS project*)
• Automatically generating test programs with deep learning
technology to verify compilers & find security vulnerabilities.
*with Professor Hugh Leather from University of Edinburgh
Scraped 400 GB
Java Program
from GitHub
Train DL model with LSTM
(Long Short-Term Memory)
to automatically generate
more Java Tests
Compare outputs
to verify
compilers & find
vulnerabilities
A/B Testing with different
JDKs/JVMs on Jenkins
JDK 8 with OpenJ9
JDK 11 with OpenJ9
JDK 11 with Hotspot
JDK 11 with Corretto
A/B Testing with various
JIT settings on Jenkins
JIT enabled
JIT disabled

23. • Analyzing test outputs with deep learning to classify test
result types: success or failure (compiler crash, build
timeout, build failure, program crash, wrong output).
*consulting with IBM Machine Learning Hub
Archive test
outputs and
results from
Jenkins
Store test
data into
database
Pre-process
data (use tf-idf
to generate
vocabulary)
Train DL model to classify
test results (weighted
model, dropout layer,
early stopping)
Evaluate DL model
with metrics
(precision, recall)
Continuous improvement with
more data and models
Apply developed
DL model to
analyze test
outputs
Test Output Analysis*

24. • Utilize deep learning model to recommend possible GitHub
issues related to test failures.
Collect Issues
from GitHub
repos
Pre-process
issue contents
Train DL model
to classify
multiple issues
Evaluate DL model
with TRSS/Jenkins
output
Continuous improvement with
more data and models
Deploy DL model in
TRSS to recommend
related issues
Recommend Related GitHub Issues

25. Demo

26. Plans Forward
• Build skills
• Groundwork
• Tools enhancements
• Model/deploy, observe & measure
• Collaborate & innovate
– researchers/universities
• Leverage useful models in open projects
– Incorporate into pipelines/workflows

27. References
• Papers from Hugh Leather
– Compiler Fuzzing through Deep Learning
– End to End Deep Learning of Optimization Heuristics
– Synthesizing Benchmarks for Predictive Modeling
• Videos & course work from Prof Andrew Ng
– Artificial Intelligence is the New Electricity
– Coursera: Deep Learning Specialization courses
• Bug prediction paper: *BugCache for Inspections: Hit or Miss?