Contenu connexe
Similaire à JavaOne 2012 CON 3961 Innovative Testing Techniques Using Bytecode Instrumentation (20)
JavaOne 2012 CON 3961 Innovative Testing Techniques Using Bytecode Instrumentation
- 1. Paul Thwaite – Java 8 Test Lead
Wednesday 3 October 2012
Testing with Bytecode Instrumentation (BCI)
JavaOne 2012 - CON3961
© 2012 IBM Corporation
- 2. Important Disclaimers
THE INFORMATION CONTAINED IN THIS PRESENTATION IS PROVIDED FOR INFORMATIONAL PURPOSES ONLY.
WHILST EFFORTS WERE MADE TO VERIFY THE COMPLETENESS AND ACCURACY OF THE INFORMATION
CONTAINED IN THIS PRESENTATION, IT IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED.
ALL PERFORMANCE DATA INCLUDED IN THIS PRESENTATION HAVE BEEN GATHERED IN A CONTROLLED
ENVIRONMENT. YOUR OWN TEST RESULTS MAY VARY BASED ON HARDWARE, SOFTWARE OR INFRASTRUCTURE
DIFFERENCES.
ALL DATA INCLUDED IN THIS PRESENTATION ARE MEANT TO BE USED ONLY AS A GUIDE.
IN ADDITION, THE INFORMATION CONTAINED IN THIS PRESENTATION IS BASED ON IBM’S CURRENT PRODUCT
PLANS AND STRATEGY, WHICH ARE SUBJECT TO CHANGE BY IBM, WITHOUT NOTICE.
IBM AND ITS AFFILIATED COMPANIES SHALL NOT BE RESPONSIBLE FOR ANY DAMAGES ARISING OUT OF THE USE
OF, OR OTHERWISE RELATED TO, THIS PRESENTATION OR ANY OTHER DOCUMENTATION.
NOTHING CONTAINED IN THIS PRESENTATION IS INTENDED TO, OR SHALL HAVE THE EFFECT OF:
- CREATING ANY WARRANT OR REPRESENTATION FROM IBM, ITS AFFILIATED COMPANIES OR ITS OR THEIR
SUPPLIERS AND/OR LICENSORS
2 © 2012 IBM Corporation
- 3. Introduction to the speaker – Paul Thwaite
■ 11 years experience developing and deploying Java SDKs
■ Recent work focus:
– Quality Assurance lead
• Leading cross-continent teams
• System Test on IBM SDKs
– Test lead for Java 8
• Focus on testing in OpenJDK
• OpenJDK contributor
■ My contact information:
– paul_thwaite@uk.ibm.com
– linkedin.com/profile/view?id=26327969
3 © 2012 IBM Corporation
- 4. What you will learn from this talk
■ Why me and what I do
■ What is BCI?
■ Typical problems encountered during testing
■ Two demos showing how BCI can help with testing
■ Introduction to Java agents
■ How BCI works
■ Limitations of BCI
■ Overview of the different BCI approaches
4 © 2012 IBM Corporation
- 5. Why me and what I do
■ Test IBM's Java SDK
– I break Java
■ Focus on system testing in QA
– Multi-threaded load testing
– Third-party applications
■ System test the Java SDK means writing Java
■ Can the VM handle any application or Java code?
5 © 2012 IBM Corporation
- 7. What is BCI?
■ Bytecode Instrumentation (BCI) is modifying the bytes of a class at runtime
■ Classes can be modified as the JVM loads them or at any point afterwards
■ Application and SDK classes can be changed (with few restrictions)
■ Abstraction frameworks exist to make BCI easier
Disk Runtime
Snippets
Class JVM
Code
Class
BCI Code
Code
Code
7 © 2012 IBM Corporation
- 8. ASM
■ ASM is an all purpose Java bytecode manipulation framework
■ It can be used to modify existing classes or dynamically generate classes, directly in binary
form
■ Many products such as AspectJ and JRuby use ASM
■ The examples in this presentation use ASM to abstract out the complexity of transforming
classes
8 © 2012 IBM Corporation
- 10. Typical problems encountered during testing
■ Test exception handling
■ Test for memory exhaustion
■ Test secure paths through code
■ Test specific code paths in very large systems such as WebSphere Application Server
10 © 2012 IBM Corporation
- 11. Problem 1: Testing all exception paths
try {
Socket aSocket = new Socket("www.example.com", 1234);
// more code...
} catch (UnknownHostException e) {
//Handle correctly
} catch (IOException e) {
//Handle correctly
} catch (SecurityException e) {
//Handle correctly
}
11 © 2012 IBM Corporation
- 13. Problem 1: Testing all exception paths
try {
Socket aSocket = new Socket("www.example.com", 1234);
// more code...
} catch (UnknownHostException e) {
//Handle correctly
} catch (IOException e) {
//Handle correctly
} catch (SecurityException e) {
//Handle correctly
}
■ The test running normally
13 © 2012 IBM Corporation
- 14. Problem 1: Testing all exception paths
try {
Socket aSocket = new Socket("www.example.com", 1234);
// more code...
} catch (UnknownHostException e) {
//Handle correctly
} catch (IOException e) {
//Handle correctly
} catch (SecurityException e) {
//Handle correctly
}
■ How do you force the exception?
– Pull the network cable out of the test machine
– Update /etc/hosts to force the exception
■ Not practical
– Manual, test-specific, other tests may run on the same machine
14 © 2012 IBM Corporation
- 15. Problem 1: Testing all exception paths
try {
Socket aSocket = new Socket("www.example.com", 1234);
// more code...
} catch (UnknownHostException e) {
//Handle correctly
} catch (IOException e) {
//Handle correctly
} catch (SecurityException e) {
//Handle correctly
}
■ How do you force the exception?
– Intercept the connection
– IOException is a “catch all” and very hard to create
– You would need to understand the implementation in order to test properly
■ Not practical
15 © 2012 IBM Corporation
- 16. Problem 1: Testing all exception paths
try {
Socket aSocket = new Socket("www.example.com", 1234);
// more code...
} catch (UnknownHostException e) {
//Handle correctly
} catch (IOException e) {
//Handle correctly
} catch (SecurityException e) {
//Handle correctly
}
■ How do you force the exception?
– A SecurityManager would need to be used
– Complicated to setup
– Not practical especially if testing a complex system
– Likely would need two versions of the test
16 © 2012 IBM Corporation
- 17. Demo using BCI to help solve
these problems
17 © 2012 IBM Corporation
- 18. Problem 1: Demo of UnknownHostException
■ What does the test do?
– Uses java.net.Socket to create a connection to three hosts
– Prints to stdout whether the connection was successful or not
– BCI is used to trigger the UnknownHostException on www.oracle.com
■ Test 1 – the test running normally
java -cp testcase tests.NetworkTest
■ Test 2 – the test running with BCI
java -Xbootclasspath/a:lib/asm-all-4.0.jar:lib/networkTrans.jar
-javaagent:lib/networkTrans.jar=www.oracle.com
-cp testcase tests.NetworkTest
Pass in the host on which to force the UnknownHostException
18 © 2012 IBM Corporation
- 19. Problem 1: Demo of UnknownHostException
■ What does the test do?
– Uses java.net.Socket to create a connection to three hosts
– Prints to stdout whether the connection was successful or not
– BCI is used to trigger the UnknownHostException on www.oracle.com
■ Test 1 – the test running normally
Connected to host: www.example.com
Connected to host: www.oracle.com
Connected to host: www.ibm.com
■ Test 2 – the test running with BCI
Connected to host: www.example.com
Couldn't find host: www.oracle.com UnknownHostException thrown
Connected to host: www.ibm.com
19 © 2012 IBM Corporation
- 20. Problem 1: Breakdown of the Java command
Append the ASM Append a Java
library to the agent library to
bootclasspath the bootclasspath
java -Xbootclasspath/a:lib/asm-all-
4.0.jar:lib/networkTrans.jar
-javaagent:lib/networkTrans.jar=www.oracle.com -cp
testcase tests.NetworkTest
Enable the Java agent Pass in options to
the agent
Run the test as normal -
the test does not change
Note: The bootclasspath is used because the Socket class is being modified.
20 © 2012 IBM Corporation
- 21. Problem 2: Testing Resource Exhaustion
■ A large system, such as WebSphere Application Server, should be able to handle an
OutOfMemoryError
■ An OutOfMemoryError can occur at any time
– It can also happen in different places within the system
■ Hard and complicated to test without side effects
– Changing the application server's environment is not a good idea
– Using a reduced heap will not guarantee all cases are covered
21 © 2012 IBM Corporation
- 22. Problem 2: Demo of OutOfMemoryError – Java command
■ What does the test do?
– Fills up the heap with Intergers until an OutOfMemoryError exception is thrown
– It will take a very long time for the OutOfMemoryError exception to be thrown
– BCI is used to set the trigger to just three Integers without having to change the test
■ Test 1 – the test running normally
java -cp testcase tests.AllocateTest
Option to trigger
OOME at just 3
Integers
■ Test 2 – the test running with BCI
java -javaagent:lib/resourceTrans.jar=3
-cp testcase:lib/asm-all-4.0.jar tests.AllocateTest
22 © 2012 IBM Corporation
- 23. Problem 2: Demo of OutOfMemoryError – stdout
■ What does the test do?
– Fills up the heap with Intergers until an OutOfMemoryError exception is thrown
– It will take a very long time for the OutOfMemoryError exception to be thrown
– BCI is used to set the trigger to just three Integers without having to change the test
■ Test 1 – the test running normally
Adding new value 0
Adding new value 1
Adding new value 2...
...Adding new value 450...
■ Test 2 – the test running with BCI
Adding new value 0
Adding new value 1
Adding new value 2
Adding new value 3
Exception in thread "main" java.lang.OutOfMemoryError at
transformer.ResourceTransformer.resourceHelper(ResourceTransformer.java:79)
at tests.AllocateTest.increment(AllocateTest.java)
at tests.AllocateTest.main(AllocateTest.java:12)
23 © 2012 IBM Corporation
- 24. Problem 2: Breakdown of the Java command
Enable the Java agent Pass in options to
the agent
java -javaagent:lib/resourceTrans.jar=3 -cp
testcase:lib/asm-all-4.0.jar tests.AllocateTest
Add the ASM library to Run the test as normal -
the classpath the test does not change
24 © 2012 IBM Corporation
- 25. Recap – why BCI?
■ Some parts of Java applications are hard to test
■ BCI provides a solution to this problem
■ With BCI, you can:
– Replace your method bodies at runtime
– Replace your entire class
– All without having to change your code
■ BCI is performed by inserting new bytecodes into your application at runtime
■ Using the UnknownHostException example, there are two ways to perform BCI
– Change the testcase to use a mock version of java.net.Socket which calls a different
class i.e. MySocketClass
– Modify the java.net.Socket class
■ Whichever you choose, you will be playing with bytecodes
25 © 2012 IBM Corporation
- 26. Recap – why BCI?
■ Some parts of Java applications are hard to test
■ BCI provides a solution to this problem
■ With BCI, you can:
– Replace your method bodies at runtime
– Replace your entire class
– All without having to change your code
■ BCI is performed by inserting new bytecodes into your application at runtime
■ Using the UnknownHostException example, there are two ways to perform BCI
– Change the testcase to use a mock version of java.net.Socket which calls a different
class i.e. MySocketClass
– Modify the java.net.Socket class
■ Whichever you choose, you will be playing with bytecodes
■ But don't fear – there are tools available which can help you!
26 © 2012 IBM Corporation
- 28. Java agent overview
■ A Java agent provides instrumentation capabilities to a Java application
■ Java agents provide the means to perform bytecode transformation
■ An agent must have a premain method
import java.lang.instrument.Instrumentation;
public class MyJavaAgent {
public static void premain(String agentArgument,
Instrumentation instrumentation)
{}
}
Manifest-Version: 1.0
Premain-Class: MyJavaAgent
■ It is packaged inside a JAR file
■ The JAR file manifest defines the agent class
■ The agent has 'no modelling restrictions' i.e. it can do anything any other java code can do
– Start threads, listen on the network etc
■ The Instrumention object is used to perform class transformations
28 © 2012 IBM Corporation
- 30. Using BCI to modify java.net.Socket
Socket
visitMethod(info)
Socket(InetAddress addr) isTargetMethod?
ASM
visitCode()
Socket(String host,int port)
insert new code
30 © 2012 IBM Corporation
- 31. Using BCI to modify java.net.Socket
Socket
visitMethod(info)
Socket(InetAddress addr) isTargetMethod?
ASM
visitCode()
Socket(String host,int port)
insert new code
31 © 2012 IBM Corporation
- 32. Using BCI to modify java.net.Socket
Socket
visitMethod(info)
Socket(InetAddress addr) isTargetMethod?
ASM
visitCode()
Socket(String host,int port)
insert new code
32 © 2012 IBM Corporation
- 33. Using BCI to modify java.net.Socket
Socket
visitMethod(info)
Socket(InetAddress addr) isTargetMethod?
ASM
visitCode()
Socket(String host,int port)
insert new code
33 © 2012 IBM Corporation
- 34. Using BCI to modify java.net.Socket
Socket
visitMethod(info)
Socket(InetAddress addr) isTargetMethod?
ASM
visitCode()
Socket(String host,int port)
insert new code
34 © 2012 IBM Corporation
- 35. Using BCI to modify java.net.Socket
Socket
visitMethod(info)
Socket(InetAddress addr) isTargetMethod?
ASM
visitCode()
Socket(String host,int port)
insert new code
35 © 2012 IBM Corporation
- 36. Using BCI to modify java.net.Socket
Socket
visitMethod(info)
Socket(InetAddress addr) isTargetMethod?
ASM
visitCode()
Socket(String host,int port)
insert new code
if(hostname.equals(host)) throw new
UnknownHostException();
36 © 2012 IBM Corporation
- 37. if (hostname.equals(host)) throw new
UnknownHostException();
ASM logic
mv.visitVarInsn(Opcodes.ALOAD, 1);
mv.visitMethodInsn(INVOKESPECIAL,
"java/net/UnknownHostException", "<init>", "()V");
mv.visitCode();
37 © 2012 IBM Corporation
- 38. if (hostname.equals(host)) throw new
UnknownHostException();
Bytecodes
0: aload_1
1: ldc #2; // String host
3: invokevirtual #3; //Method
java/lang/String.equals:(Ljava/lang/Object;)Z
6: ifeq 17
9: new #4; //class java/net/UnknownHostException
12: dup
13: invokespecial #5; //Method
java/net/UnknownHostException."<init>":()V
16: athrow
17: return
38 © 2012 IBM Corporation
- 39. Limitations of BCI at runtime
■ Restrictions when retransforming classes at runtime:
– Fields cannot be added or removed
– Methods cannot be added or removed
– Method signatures cannot be modified
■ Effectively only method bodies can be modified
■ Most classes can be modified (including most SDK classes)
39 © 2012 IBM Corporation
- 41. Replace a class using a pre-compiled version of your new class
■ Pre-compile a new version of a class
■ When the class to be loaded next is the one you want to replace
■ You can return the bytes for your new class and discard the old one
■ No knowledge of bytecode is required
■ Original source must be available and unchanged
■ Need to match the correct compiler version
41 © 2012 IBM Corporation
- 42. Modify bytecodes with a BCI framework
■ Use a framework
– Provides an abstraction layer
– Easier to learn (provides symbolic constants, helper APIs)
– Adds an additional dependency
■ ASM
– Widely used
– Uder active development
– Friendly license
■ ASM gives you a good symbolic framework for writing bytecodes
■ Knowledge of the JVM architecture is required
42 © 2012 IBM Corporation
- 43. Use javap to create bytecodes
■ Write some java code that does what you want
■ Compile your class to byte-code
■ Use a de-compiler to examine the class file
■ JVM ships with javap
■ verbose option (-v) decompiles method bytecodes
void setupConnections();
Code:
0: ldc #2 // String www.example.com
2: sipush 1234
5: invokestatic #3 // Method MySocketFactory.newSocket:
(Ljava/lang/String;I)Ljava/net/Socket;
8: astore_1
9: return
43 © 2012 IBM Corporation
- 44. ASMifier – Create ASM logic
■ ASMifier
– Part of standard ASM distribution
– Turn a Java method into byte code logic for insertion into an ASM class
– An Eclipse plugin is available
mv.visitLdcInsn("www.example.com");
mv.visitIntInsn(SIPUSH, 1234);
mv.visitMethodInsn(INVOKESTATIC, "transformer/SocketHelper$MySocketFactory",
"newSocket", "(Ljava/lang/String;I)Ljava/net/Socket;");
mv.visitVarInsn(ASTORE, 1);
mv.visitInsn(RETURN);
44 © 2012 IBM Corporation
- 45. What can go wrong? (Dealing with class verify problems)
■ If the JVM is unhappy with your modified class, it may throw a java.lang.VerifyError
■ 'stack shape inconsistent'/'stack underflow'
– Attempting to operate on the wrong type of variable/stack is empty
– Often due to failure to push/pop all the correct variables
■ 'Inconsistent stackmap frames'
– Stackmaps are method meta-data used by the JVM
– A map of all the variables on the stack at the target of any branch/goto instruction
– BCI cause targets to move, invalidating the stack maps
– ASM will re-calculate them for you
■ 'stack overflow'
– Maximum stack size specified in method meta data
– Any operations added via BCI may increase required stack
– ASM will calculate that for you too
45 © 2012 IBM Corporation
- 46. What can go wrong? (Runtime Problems)
■ No compile time checking
■ Access control
– Calling a private method is valid byte-code, fails at runtime
■ Calling a non-existent method
– Incorrect argument types
– Incorrect argument ordering
– Typos
46 © 2012 IBM Corporation
- 47. Conclusion
■ BCI is very powerful
■ BCI is used in many products
■ JVM knowledge is required but...
■ There are many tools available to make it easier
■ BCI allows testing of otherwise impossible or tricky situations
■ Slides will be available on Slideshare after the conference
■ Examples are available on request
– paul_thwaite@uk.ibm.com
47 © 2012 IBM Corporation
- 48. References
■ Get Products and Technologies:
– IBM Java Runtimes and SDKs:
• https://www.ibm.com/developerworks/java/jdk/
– IBM Monitoring and Diagnostic Tools for Java:
• https://www.ibm.com/developerworks/java/jdk/tools/
■ Learn:
– IBM Java InfoCenter:
• http://publib.boulder.ibm.com/infocenter/java7sdk/v7r0/index.jsp
■ Discuss:
– IBM Java Runtimes and SDKs Forum:
• http://www.ibm.com/developerworks/forums/forum.jspa?forumID=367&start=0
48 © 2012 IBM Corporation
- 49. References
■ Eclipse Bytecode Outline plugin (allows you to use asmifier in Eclipse)
– http://andrei.gmxhome.de/bytecode/index.html
■ ASM
– http://asm.ow2.org/index.html
■ Java VM Spec
– http://docs.oracle.com/javase/specs/jvms/se7/html/index.html
■ java.lang.instrument package documentation
– http://docs.oracle.com/javase/7/docs/api/index.html?java/lang/instrument/package-summary.html
49 © 2012 IBM Corporation
- 50. Copyright and Trademarks
© IBM Corporation 2012. All Rights Reserved.
IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business
Machines Corp., and registered in many jurisdictions worldwide.
Other product and service names might be trademarks of IBM or other companies.
A current list of IBM trademarks is available on the Web – see the IBM “Copyright and trademark
information” page at URL: www.ibm.com/legal/copytrade.shtml
50 © 2012 IBM Corporation