Managing Change and Achieving Regression Isolation with Dynamic Groovy Edges
1. 1
Managing Change and Achieving
Regression Isolation
Dynamic Groovy Edges
May 21st, 2012
Prepared for:
Triangle Java User’s Group (TriJUG)
2. 2
Who Am I? Well…
Java Architect with ICF Ironworks
Adjunct Professor
Started with HTML and Lotus Notes in 1992
• In the interim there was C, C++, VB, Lotus Script, PERL, LabVIEW,
etc.
Not so much an Early Adopter as much as a Fast Follower of Java
Technologies
• Learned Java 1.1 in 1997, J2EE in 1999
Alphabet Soup (MCSE, ICAAD, ICASA, SCJP, SCJD, PMP, CSM)
LinkedIn: http://www.linkedin.com/in/iamjimmyray
Blog: http://jimmyraywv.blogspot.com/ Avoiding Tech-sand
4. 4
JEE Application Architecture
Arranged into layers
When there are N layers:
• There are N-1 layer junctions
• There are (N-1) X 2 layer edges
In this diagram there are three
layers, two layer junctions, and four
layer edges.
We
control
edges
5. 5
The Issue with Layers
Often written by someone else
• You often have little control or influence over the implementation of the
layer to which you must integrate
• You control your side (edge) of the interface junction
Notorious change zones
• Changes occur at different paces
6. 6
Keeping Up the Pace
Pace Layering Approach to Application Design
• http://www.doozer.com/developer-exchange/entry/accelerating-innovation-
by-adopting-a-pace-layered-application-strategy
Complex systems can be decomposed into multiple layers, where the layers change at different
rates. The “fast layers” learn, absorb shocks and get attention; the “slow layers” remember,
constrain and have power. One of the implications of this model is that information architects can
do what they have always done–slow, deep, rich work; while tagging can spin madly on the
surface. If it is worth keeping, it will seep down into the lower layers. – Donna Maurer
7. 7
When Layers Change
In Java web applications, regression analysis and testing are
usually tied to the boundaries of the application or deployment
artifacts: JARs, WARs, EARS etc.
• Upstream and downstream dependencies also come into play
• Not to be confused with functional testing of changed components
If you redeploy the entire WAR or EAR, you could find yourself
testing that entire application boundary.
• As well as its integration junctions (critical patches not withstanding)
8. 8
Detecting Changes
How do we detect changes to web services as soon as possible?
• XMLUnit/JUnit Tests
• Easily written in Groovy
• Hudson scheduled Ant tasks
DIFFERENCE CODE:19:CHILD_NODELIST_LENGTH_ID:Comparing 2 nodes with different
numbers of children
Control(prod) Node: <xs:sequence xmlns:xs="http://www.w3.org/2001/XMLSchema">
<xs:element minOccurs="0" name="address" type="tns:address"/>
<xs:element minOccurs="0" name="employeeId" type="xs:string"/>
<xs:element minOccurs="0" name="hireDate" type="xs:dateTime"/>
</xs:sequence>
Target(dev) Node: <xs:sequence xmlns:xs="http://www.w3.org/2001/XMLSchema">
<xs:element minOccurs="0" name="address" type="tns:address"/>
<xs:element minOccurs="0" name="department" type="tns:department"/>
<xs:element minOccurs="0" name="employeeId" type="xs:string"/>
<xs:element minOccurs="0" name="hireDate" type="xs:dateTime"/>
</xs:sequence>
10. 10
CI Using Groovy XmlUnit Tests
Wrap Groovy script with Ant
Execute in Hudson build
Use Gradle with the Groovy Plugin
• Run your Groovy test within Gradle build
11. 11
Managing Change
How do we better manage the change between the layers?
• Understand that we cannot stop change
• Reduce regression testing needs by shrinking regression boundaries
• Prepare our teams for the changes with more dynamic reaction
techniques
• Start with Groovy beans
• Or…Start with Java beans and move to Groovy as needed
• Detect changes as early as possible, and before they impact
downstream systems
12. 12
Groovy Edges
Build your edges with Groovy (or other dynamic
JVM language) objects
• Groovy can be written as a script and parsed at run-
time and compiled into Java byte-code and then
loaded by the Groovy class loader.
• Groovy scripts can contain Groovy and Java syntax
in the script file (*.groovy).
• Groovy can use the same Java APIs that are used
by traditional Java code.
• Groovy adds features that make it easier to work
with XML (primary language of choice for most
integrations)
• Markup Builder
• XML Slurper
13. 13
Groovy Markup Builder
Allows developers to use “builder patterns” to construct XML
def writer = new StringWriter();
def xml = new MarkupBuilder(writer);
xml.'ser:writeEmployee'('xmlns:ser':"http://service.rjug.ironworks.com/"){
employee{
dateOfBirth(emsRequest.getEmployee().getDateOfBirth())
firstName(emsRequest.getEmployee().getFirstName())
lastName(emsRequest.getEmployee().getLastName())
middleName(emsRequest.getEmployee().getMiddleName())
employeeId(emsRequest.getEmployee().getEmployeeId())
hireDate(emsRequest.getEmployee().getHireDate())
address{
addressLine1(emsRequest.getEmployee().getAddress().getAddressLine1())
…
14. 14
A Word About Hard-Coding (or not)
When is hard-coding not really?
• Our internal beliefs have shifted slightly.
How about when it is in Groovy?
• Especially when that Groovy is treated as content and not part of a
rigid deployment architecture?
• Still have to work within the boundaries (constraints) of the JVM
15. 15
Groovy XML Slurper
Groovy construct for efficient partial XML processing.
EmployeeManagerServiceResponse emsResponse = new EmployeeManagerServiceResponse();
GPathResult response;
try{
response = new XmlSlurper().parseText(rawResponse.getWriter().toString());
emsResponse.getEmployee().setFirstName(response.'return'.firstName.text());
emsResponse.getEmployee().setLastName(response.'return'.lastName.text());
…
17. 17
A Word About Groovy Slurping XML Payloads
No longer tied to JAXB
• This is uncomfortable for some traditional Java developers
Can use Groovy closures on collections (even nested collections)
Easy to enable/disable getter calls, especially if Groovy is stored
outside of rigid deployment structure (CMS, Apache, etc.)
• This is scary to some
• Still requires Configuration Management rigor
• It’s about risk tolerance
• It could be that this approach is better suited for non-production
environments within certain organizations
18. 18
Groovy XML Slurper Trade-offs
Use the Groovy XML Slurper when you do not have to parse the
entire XML
• Slurper uses paths expressions as apposed to constructing a DOM
• Path expression might be executed multiple times
• Could be slower in larger, more complex XML
There are also differences when editing XML
• The XML Parser allows for edits inline
• The XML Slurper would require another “slurp” after changes are
applied.
The XML Slurper is well suited for simple web service clients
• Allows direct access via getters to “return” object graph
19. 19
Groovy Edge Component – Web Service Client
For SOA interfaces implemented with web services, web service
clients are primarily the target edge component for dynamic
updating.
Using Groovy to write these web services clients makes it easier to
update clients.
• Features can be added or removed and quickly “redeployed” via
content changes, and not necessarily Java deployments
Spring will reload the Groovy web service client when it changes
This provides a great solution for prototyping in an Agile
environment.
For MVC containers, the controller is the edge component that is
up for “Groovy-enizing”, “Groovy-ization”, etc.
22. 22
Loading Properties
<!-- Read service related properties from the classpath -->
<bean id="services.propertyConfigurer"
class="org.springframework.beans.factory.config.PropertyPlacehold
erConfigurer">
<property name="location">
<value>http://localhost/properties/service.properties</value>
</property>
</bean>
23. 23
Demo – Web Services Provider
Simple JAX-WS Bottom-up web service
Environment:
• Apache2.2
• Tomcat 6.0.35
• Java 1.6 JRE
• JAX-WS RI 2.2
• JAXB 2.2.3
• SoapUI 4.0.0
• Spring Data Layer – MongoDB Data Repository
• MongoDB NoSQL Database (Upcoming TriJUG Event: 07/16/12 Deep
Mistry Mongo DB)
24. 24
Demo – Web Services Consumer
Simple Spring WS Client
Environment:
• Apache2.2
• Tomcat 6.0.35
• Java 1.6 JRE
• Spring Framework 3.0.x
• Spring WS
• Groovy Web Services Framework
25. 25
This Is Not Bleeding Edge
Yet, it is still new to a lot of Spring users
January 2009: http://www.ibm.com/developerworks/java/library/j-
groovierspring2/index.html
At the NVSS NFJS conference in Reston in April 2012, there were
still sessions on Spring and Groovy integration with dynamic beans
• NFJS Due in Raleigh, NC – August 24-26
• http://www.nofluffjuststuff.com/home/main
26. 26
Spring Bean Configuration Issues
Spring needs to be able to locate Groovy script content in order to
load configurations.
ERROR - ContextLoader.initWebApplicationContext(220) | Context
initialization failed
org.springframework.beans.factory.BeanCreationException: Error creating
bean with name 'employeeManagerService' defined in file
[C:Toolsapache-tomcat-6.0.16w
ebappsDynamicGroovyTutorialWEB-INFclassescontextservice_ws.xml]:
Cannot resolve reference to bean 'employeeManagerServiceImpl' while
setting bean property
'groovyImpl';
27. 27
Spring Bean Config With Content Proxy
Use a Content Proxy
• The Content Proxy is deployed into a separate WAR and addressable
via a URL (perhaps a RESTful web service) that would retrieve the
dynamic Spring Bean configuration scripts
• If the scripts are not found in script repository, the Content Proxy can load
the scripts from a backup site or even from inside the application boundaries
– Default Location.
• The Content Proxy is a layer of indirection to better manage the link
between your Spring application and the content management system that
you may be using.
• Content Proxies are also very helpful when scripts (or other content) are
loaded from a CMS through an API that would not link directly to the Spring
Bean configurator.
• Provides a layer of indirection and intelligence for Cache Forward
Architectures.
28. 28
Avoiding the Race Condition
Using a ServletContextListener allows you to specify how long the
application will try to seek the ContentProxy (or other URLs) in
other WARs that still may be spinning-up, before it lets the rest of
the application load.
• Reduces the “race-condition” introduced by URLs across multiple
WARs
30. 30
Inline Groovy – You can, but why would you?
package com.ironworks.framework.scripting;
import org.springframework.core.io.ResourceLoader;
import org.springframework.scripting.ScriptSource;
import org.springframework.scripting.support.*;
public class URLScriptFactoryPostProcessor extends ScriptFactoryPostProcessor{
@Override
protected ScriptSource convertToScriptSource(String beanName,
String scriptSourceLocator, ResourceLoader resourceLoader) {
if (scriptSourceLocator.startsWith(INLINE_SCRIPT_PREFIX)) {
return new StaticScriptSource(
scriptSourceLocator.substring(INLINE_SCRIPT_PREFIX.length()), beanName);
}
else if (scriptSourceLocator.startsWith("classpath")) {
return new
ResourceScriptSource(resourceLoader.getResource(scriptSourceLocator));
}
else{
return new URLScriptSource(resourceLoader.getResource(scriptSourceLocator));
}
}
}
31. 31
Spring Bean Delegating Construct
Use a Spring Bean Delegate construct as an alternative to
ServletContextListener and ContentProxy for handling potentially
unreliable or dynamic URLs during application startup
Deploy JAXB version of services, but leave them open to
replacement by Groovy, without needing another deployment
• Still able to use Dynamic Groovy Edges, if JAXB version of service
client requires updating
<bean id="iw_service.getData" class="com.ironworks.vdc.framework.service.DelegatingService">
<property name="delegates">
<list>
<ref bean="iw_service.getDataGroovy" />
<ref bean="iw_service.getDataJaxb" />
</list>
</property>
</bean>
32. 32
Spring Bean Delegate Construct Code
public final class DelegatingService<REQUEST extends ServiceRequest, RESPONSE extends
ServiceResponse>
implements Service<REQUEST, RESPONSE> {
private List<Service<REQUEST, RESPONSE>> delegates =
new ArrayList<Service<REQUEST, RESPONSE>>();
@Override
public RESPONSE execute(final REQUEST request) {
for (Service<REQUEST, RESPONSE> service : delegates) {
if (!(service instanceof AvailabilityStatus)
|| ((AvailabilityStatus) service).isAvailable()) {
return service.execute(request);
}
} throw new UnsupportedOperationException(
"No service implementation available to execute request");
}
public void setDelegates(final List<Service<REQUEST, RESPONSE>> delegates) {
this.delegates = delegates;
}
}
33. 33
AvailabilityStatus Interface
public interface AvailabilityStatus {
/**
* Indicate whether this service instance is available.
*
* @return true if the service is available, false otherwise.
*/
boolean isAvailable();
}
34. 34
Spring Delegate Construct Code Explanation
DelegatingService is substituted into the class attribute of the bean
definition
Possible “delegates” are added to the property list as objects
DelegatingService looks for viable services in the list of “delegates”
• Could be services that implement the AvailabilityStatus interface, and
are available (isAvailable())
• Beans that implement AvailabilityStatus, but are not available our by-passed
– This allows Groovy substitutes to be enabled/disabled quickly
• Could be services that do not implement AvailabilityStatus
35. 35
Some Tradeoffs for Groovy Approach
Groovy Web Clients
• Pros
• Easier to program with Getters and Setters approach to accessing and
mutating XML objects for request and response
• Markup Builder
• XML Slurper
• Cons
• Lose auto-generation of object graphs via JAXB
• Lose tighter integration to WSDL/XSD and makes it harder to track down
changes.
• It’s not for every change…more complex WSDLs, possibly with nested
object architectures, would not be good candidates for Groovy re-write
36. 36
What About All Those Classes?
Almost every technical decision comes with a trade off…
When using dynamic Groovy with the approach we have outlined,
obviously we are generating more “non-setup”, Groovy classes in
the JVM.
• Every time Spring detects a changed script, Groovy will parse it and
load another class.
• Depending on your JVM, GC settings and JVM memory settings, you
could experience OOM issues with PermGen
37. 37
Does Your JVM Have a PermGen?
IBM: No PermGen, classes exist in the native C-Heap
Oracle/WebLogic JRockit: No PermGen, classes exist in the
native C-Heap
Oracle/Sun Hotspot: Yes, PermGen in Java Heap
• Prior to Java 5, the Hotspot JVM assumed that classes were forever
• Then came CMS (Concurrent Mark Sweep) class unloading settings
38. 38
What Can We Do With PermGen?
Use GC settings for the CMS Collector
• Java 1.6: -XX:+CMSClassUnloadingEnabled
• Java 1.5: -XX:+CMSPermGenSweepingEnabled
Both require CMS GC enabled
• This may not be suitable for all applications
Increase size (not really desirable)
• Switch to 64 bit, if you have not already
Increase JVM bounce frequency (not really desirable)
• Not really a good long term strategy
39. 39
Don’t Use PermGen.
Some of our projects use JRockit
• While we do not see the same sensitivity to PermGen in our dynamic
environment, occasionally we still see issues with class memory areas
• Mostly in non-production environments where we tend to be more
dynamic
Problem: JRockit does not exist beyond Java 7
• Oracle has announced that it is being merged with Hotspot in Java 7
40. 40
Use Java 7?
In Java 7, PermGen is supposed to go away over time.
• This implies better class unloading and memory management
• JRockit merges with Hotspot
• When?
• http://javaeesupportpatterns.blogspot.com/2011/10/java-7-features-
permgen-removal.html
• http://blogs.oracle.com/java/entry/java_7_questions_answers
41. 41
Groovy Spring MVC Controllers
Spring MVC controllers are another example of edge components
that could benefit from more dynamic deployment and less rigid
deployment structures
• Remember: Just because it is marked as Groovy (*.groovy) does not
preclude you from writing some Java
<lang:groovy id="claimsController" script-source="${groovyClaimsController}" refresh-check-
delay="1000" >
<lang:property name="view" value="claim_summary"/>
<lang:property name="detailView" value="claim_detail"/>
<lang:property name="unsolicitedView" value="claim_unsolicited"/>
<lang:property name="errorView" value="claim_error"/>
<lang:property name="serviceDownView" value="claim_service_down"/>
</lang:groovy>>
43. 43
Summary
We cannot control changes in layers beyond our influence or
control
We can better prepare for change by using dynamic technologies
in the edges of interface junctions.
• Groovy blurs the once clear line between hard coding and properties
configurations.
• By using Spring to dynamically load Groovy scripts, stored outside of
web applications, web applications can narrow regression boundaries
by reducing the need to deploy EARs and WARs.
• The Content Proxy provides a layer of indirection that smoothes the
process for locating dynamic scripts an providing digestible data for the
Spring Bean Configuration process.
• Testing WSDL and XSD artifacts with XMLUnit provides early warning
of web service changes and failures
44. 44
Special Thanks To:
Laith Al-Samir – ICF Ironworks
• Spring WS application and Groovy WS Client
Mike Killeen – ICF Ironworks
• Groovy/XMLUnit/JUnit WSDL/XSD tests
Vince Jorrand – ICF Ironworks
• Spring Bean Delegates Construct