Apache Camel - Stéphane Kay - April 2011

Apache Camel - Stéphane Kay - April 2011
Apache Camel Stephane Kay Ville de Lausanne
Scope  IS Integration  EIP  Routing  Mediation  Camel Container  Unit testing  Hands-on  Real Use Case JMS / Active MQ JBI / NMR OSGI / ServiceMix Scala
Table of contents IS Integration heuristics I. ◦ ◦ Common issues Paradigms Camel’s bleat II. ◦ ◦ Concepts responding to integration needs Getting started What’s under the humps? III. ◦ ◦ Camel architecture Distribution Camel in Action IV. ◦ ◦ Persistence, transactions, concurrency, failover Aggregation A Real Use-Case V. ◦ Description, issues & achievement
I. IS Integration Heuristics
I. IS Integration heuristics Just do it !  Just coding simple things might be just not so simple … public class FileCopier { public static void main(String args[]) throws Exception { File inboxDirectory = new File("data/inbox"); File outboxDirectory = new File("data/outbox"); outboxDirectory.mkdir(); File[] files = inboxDirectory.listFiles(); for (File source : files) { if (source.isFile()) { File dest = new File( outboxDirectory.getPath() + File.separator + source.getName()); copyFile(source, dest); } } } private static void copyFile(File source, File dest) throws IOException { OutputStream out = new FileOutputStream(dest); byte[] buffer = new byte[(int) source.length()]; FileInputStream in = new FileInputStream(source); in.read(buffer); try { out.write(buffer); } finally { out.close(); in.close(); } } }
I. IS Integration heuristics A Framework?  The bad : ◦ Learning effort consumes time. ◦ Adds anarchical dependencies. ◦ Risk of coming with a new need / not implemented  The good : ◦ Gain value after a training period ◦ Leverage your code to a higher abstraction ◦ Potentially a future standard ?  Indicator : community commits & posts curve.  The plus-value ◦ Not obvious
I. IS Integration heuristics Separation of concerns  IS Integration involves 1. Endpoints  A common endpoint interface abstraction 2. Data Processing  Using Message / exchange abstraction and conversion tool 3. Routing  A common vocabulary / pattern as DSL foundation.
I. IS Integration | endpoint Message Endpoint pattern   Routes should refer to a common abstract Message Endpoint rather than directly using their technical interface A good Endpoint abstraction is a URI description
II. Camel’s bleat | data Uniformed Exchange  Message’s container during routing
I. IS Integration | data Data Transformation  Data Format Transformation ◦ Message body is transformed from one format to another.  CSV -> XML  Data Type Transformation ◦ Java Type of message body is transformed/casted into another.  java.lang.String -> java.jms.TextMessage  java.io.File -> java.lang.String
I. IS Integration | routing Use case : Filtering Route ◦ How can a component avoid receiving uninteresting messages ? ◦ …coming from a different system
I. IS Integration | routing EIP?  A Catalog of Design patterns for common IS integration issues  http://camel.apache.org/enterprise-integration-patterns.html
I. IS Integration | routing EIPs Samples…  Filter ◦  Splitter ◦  How can a component avoid receiving uninteresting messages? How can we process a message if it contains multiple elements, each of which may have to be processed in a different way? Router ◦ How can you decouple individual processing steps so that messages can be passed to different filters depending on a set of conditions?  Aggregator ◦ How do we combine the results of individual, but related messages so that they can be processed as a whole?
I. IS Integration heuristics Putting all needs together  Route definition attributes : ◦ Endpoints  Producer / Consumer(s) ◦ Processor(s)  Uniformed exchange processing interface ◦ Predicate(s)  A route language should provide those key abstractions Endpoint A = endpoint(“activemq:queue:quote”); Endpoint B = endpoint(“mq:quote”); Predicate isWidget = xpath(“/quote/product = „widget‟”); from(A).filter(isWidget).to(B);
I. IS Integration heuristics I. Review  IS Integration paradigms ◦ Endpoint abstraction -> URIs? ◦ Data Tx -> Functions // Java beans ? ◦ Routing -> EIPs
II. Camel’s bleat Apache Camel is a Powerful Open Source Integration Framework based on known Enterprise Integration Patterns *  Mission Statement : Making integration easier and more accessible to developers
II. Camel’s bleat | endpoint Endpoint URI  A camel endpoint URI consists of three parts : ftp://riders.com/orders?username=rider&password=secret Scheme Context path Options
II. Camel’s bleat | routing Java DSL : Builder pattern  The Builder Pattern (GOF) is central to the Camel Java DSL public class FileCopierWithCamel { public static void main(String args[]) throws Exception { // create CamelContext CamelContext context = new DefaultCamelContext(); // add our route to the CamelContext context.addRoutes(new RouteBuilder() { public void configure() { from("file:data/inbox?noop=true").to("file:data/outbox"); } }); // start the route and let it do its work context.start(); Thread.sleep(10000); // stop the CamelContext context.stop(); } }
II. Camel’s bleat | data Data Format transformations  Marshal / unmarshal from("activemq:QueueWithJavaObjects”) .marshal().jaxb() .to("mq:QueueWithXmlMessages"); from("activemq:queue:MY_QUEUE").unmarshal().zip().proce ss(new UnZippedMessageProcessor()); http://camel.apache.org/data-format.html
II. Camel’s bleat | data Message Translator EIP  Aka Adapter (GOF)  Using a Processor from("direct:start").process(new Processor() { public void process(Exchange exchange) { exchange.getIn().setBody(in.getBody(String.class) + " World!"); } }).to("mock:result");  Using Bean binding from("direct:start") .bean(MyBean.class, "hello(String,String)") .to("mock:result");  Using transform() DSL from("direct:start").transform(body().regexReplaceAll(“n”,”<br/>”)) .to("mock:result");
II. Camel’s bleat | first ride Getting started : IDE Eclipse  Maven integration (m2eclipse)  ◦ http://m2eclipse.sonatype.org/sites/m2e ◦ Check your eclipse.ini : locate your JDK -vm C:/dev/java/jdk1.6.0_18/bin  Optional : Fuse Camel plugin for Eclipse ◦ http://repo.fusesource.com/beta/eclipse/u pdate/
II. Camel’s bleat | first ride Getting started : Maven  Basic Maven configuration
II. Camel’s bleat | first ride First Ride Copy contents from one folder to another, control final name / dir  JUnit  ◦ ◦ ◦ ◦  Startup context Route definition : File component Processor / Exchange Simple Language (File Component) Delegate JUnit test to Main class
II. Camel’s bleat | first ride Second Ride  Add some features to JUnit Class ◦ ◦ ◦ ◦  Log Add failure if file already exist Predicate : Simple Language Error handling Router EIP (Choice)
II. Camel’s bleat | spring Spring XML : content-based router <camelContext> <route> <from uri="activemq:NewOrders"/> <choice> <when> <xpath>/order/product = 'widget'</xpath> <to uri="activemq:Orders.Widgets"/> </when> <otherwise> <to uri="activemq:Orders.Gadgets"/> </otherwise> </choice> </route> </camelContext>
II. Camel’s bleat II. Review Endpoint / Component URIs  Java DSL  How to start / process exchanges  Log, error handling / routing 
III. What’s under the humps?  High-level architecture ◦ Container & Services Toolkit provided out-of-the box  Distribution 
III. What’s under the humps? High-level view  Container responsibilities : ◦ Create and activate the necessary Endpoint instances using the available Component implementations
III. What’s under the humps? Camel Context Services  Components ◦  Endpoints ◦  Contains the loaded data formats. Registry ◦  Contains the loaded type converters. Camel has a mechanism that allows you to manually or automatically convert from one type to another. Data formats ◦  Contains the routes that have been added. Type converters ◦  Contains the endpoints that have been created. Routes ◦  Loads components on the fly either by autodiscovery on the classpath or when a new bundle is activated in an OSGi container. Contains a registry that allows you to look up beans. By default, this will be a JNDI registry. If you’re using Camel from Spring, this will be the Spring ApplicationContext. It can also be an OSGi registry if you use Camel in an OSGi container. Languages ◦ Contains the loaded languages. Camel defines 3 different DSLs (Java, XML/Spring, and Scala) and allows you to use many different languages to create expressions.
III. What’s under the humps? Out of the box : EIPs  50+ Routing Patterns based on EIPs  http://camel.apache.org/enterprise-integration-patterns.html
III. What’s under the humps? Out of the box : Components 70+ Endpoint impl. aka « Components »  All available via URI/scheme paradigm  http://camel.apache.org/components.html 
III. What’s under the humps? Out of the box : Data Format  18 Data Formats - uniformed usage : from(“file://input").marshal().zip().to("activemq:queue:MY_QUEUE");  http://camel.apache.org/data-format.html
III. What’s under the humps? Out of the box : Languages  Predicates & Expressions  http://camel.apache.org/languages.html
Camel distribution http://camel.apache.org/download.html Components sources  Examples  Maven Central (current v2.7)  ◦ Snapshot Repo <repository> <id>apache.snapshots</id> <name>Apache Development Snapshot Repository</name> <url>https://repository.apache.org/content/repositories/snapshots/</url> <releases> <enabled>false</enabled> </releases> <snapshots> <enabled>true</enabled> </snapshots> </repository>
III. Review Camel is an integration framework  Based on Enterprise Integration Patterns  Routing and mediation  Easy to use DSL to define routes  No heavy specification  No container dependency  Payload agnostic  Connectivity to a great wealth of transports 
IV. Camel in Action Test support  Transactions  Exceptions  Concurrency  Persistence  Aggregation  Runtime 
IV. Camel in Action| test Camel Test Support  Provides context/registry support  MockEndpoint assertions resultEndpoint.expectedBodiesReceived(expectedBody); template.sendBodyAndHeader(expectedBody, "foo", "bar"); resultEndpoint.assertIsSatisfied();
IV. Camel in Action| test Using test support CamelTestSupport Junit  @EndpointInject  @Produce  Filter Pattern 
IV. Camel in Action | tx Transactions  Component level ◦ Polling strategy public interface PollingConsumerPollStrategy { boolean begin(Consumer consumer, Endpoint endpoint); void commit(Consumer consumer, Endpoint endpoint, int polledMessages); boolean rollback(Consumer consumer, Endpoint endpoint, int retryCounter, Exception cause) throws Exception; }  Routing level ◦ Transactional Client EIP  http://camel.apache.org/transactional-client.html ◦ Transacted DSL from("jetty:http://localhost/myservice/order").transacted() .to("bean:validateOrder").to("jms:queue:order");
IV. Camel in Action | tx Propagation  Define transaction behaviour <bean id="PROPAGATION_REQUIRED" class="org.apache.camel.spring.spi.SpringTransactionPoli cy"> <property name="transactionManager" ref="txManager"/> <property name="propagationBehaviorName" value="PROPAGATION_REQUIRED"/> </bean> <bean id="PROPAGATION_REQUIRES_NEW" class="org.apache.camel.spring.spi.SpringTransactionPolicy"> <property name="transactionManager" ref="txManager"/> <property name="propagationBehaviorName" value="PROPAGATION_REQUIRES_NEW"/> </bean>  Use transacted DSL from("direct:mixed") .transacted("PROPAGATION_REQUIRED“) .setBody(constant("Tiger in Action")) .beanRef("bookService") .setBody(constant("Elephant in Action")) .beanRef("bookService“).to("direct:mixed2");
IV. Camel in Action | errors Exceptions  onException clause new RouteBuilder() { public void configure() { onException(IOException.class).maximumRedeliveries(3); from("direct:start") .process("processor1") .process("processor2") // <--- throws a ConnectException .to("mock:theEnd"); } ◦ Note : By default, a producer will not redeliver  Handle onException(ValidationException.class) .handled(true).transform(body(constant("INVALID ORDER"))); from("jetty:http://localhost/myservice/order").transacted() .to("bean:validateOrder").to("jms:queue:order");
IV. Camel in Action | errors Dead Letter EIP The Error Handler propagates error back to the client.  Dead Letter Channel will always moved failed exchanges to this queue.   useOriginalMessage is used for routing the original input message instead of the current message that potentially is modified during routing. from("jms:queue:order:input").to("bean:validateOrder") .to("bean:transformOrder").to("bean:handleOrder"); errorHandler(deadLetterChannel("jms:queue:dead") .useOriginalMessage().mamimumRedeliveries(5) .redeliverDelay(5000);
IV. Camel in Action | toolkit Concurency  Dealing with data flows means mastering concurency ◦ Seda Component : from("seda:bar?concurrentConsumers=5").to("file://output"); ◦ Thread DSL  How do we avoid the messages that does not expect a reply to block ? from("jms:queue:order") // do some sanity check validation .to("bean:validateOrder") .to("mock:validate").threads(20) // do some CPU heavy processing of the message (we simulate and delay) .unmarshal(mySecureDataFormat).delay(1000) .to("bean:handleOrder").to("mock:order");
IV. Camel in Action | toolkit Sync to Async  toAsync Processor from("direct:start").to("mock:a").toAsync("direct:bar", 5).to("mock:result"); from("direct:bar").to("mock:b").transform(constant("Bye World"));  How does this work? void process(Exchange exchange, AsyncCallback callback) throws Exception; ◦ With AsyncCallback interface : void onTaskCompleted(Exchange exchange);  Http Async Producer public class JettyHttpProducer extends DefaultProducer implements AsyncProcessor ◦ Then invoke the callback when reply is ready on the Exchange. ◦ Routing with Spring DSL <route>  <from uri="direct:start"/>  <to uri="log:+++ request +++"/>  <to uri="jetty://http://0.0.0.0:9123/myapp" async="true" poolSize="10"/>  <to uri="log:+++ reply +++"/>  <to ref="result"/> </route>
IV. Camel in Action | toolkit Persistence   SQL, JDBC and JPA components provided out-of the box File idempotent consumer with pluggable repository from("file://inbox?idempotent=true").to("..."); <bean id="fileStore" class="org.apache.camel.processor.idempotent.FileIdempotentRepository">  <property name="fileStore" value="target/fileidempotent/.filestore.dat"/>  <property name="maxFileStoreSize" value="512000"/>  <property name="cacheSize" value="250"/> </bean> <camelContext xmlns="http://camel.apache.org/schema/spring"> <route> <from uri="file://target/fileidempotent/?idempotent=true&idempotentRepository= #fileStore&move=done/${file:name}"/> <to uri="mock:result"/> </route>  Stateful EIP : Aggregator
IV. Camel in Action | toolkit Aggregator EIP  Combine a number of messages together into a single message. ◦ http://camel.apache.org/aggregator2.html  Completion predicates from("direct:start“). .aggregate(header("id"), new BodyInAggregatingStrategy()).completionTimeout(3000) .to("mock:aggregated"); from("direct:start“).aggregate(header("id"), new BodyInAggregatingStrategy()).completionSize(3) .to("mock:aggregated"); from("direct:start“).aggregate(header("id"), new BodyInAggregatingStrategy()) .eagerCheckCompletion().completionPredicate(body().isEqualTo("END")) .to("mock:aggregated"); from("direct:start“).aggregate(header("id"), new BodyInAggregatingStrategy()). completionSize(header("mySize")) .to("mock:aggregated");
IV. Camel in Action | toolkit Aggregator Strategy & Fail Over  Strategy public class NumberAggregationStrategy implements AggregationStrategy { public Exchange aggregate(Exchange oldExchange, Exchange newExchange) { if (oldExchange == null) { return newExchange; } Integer num1 = oldExchange.getIn().getBody(Integer.class); Integer num2 = newExchange.getIn().getBody(Integer.class); Integer num3 = (num1 != null ? num1 : 0) + (num2 != null ? num2 : 0); oldExchange.getIn().setBody(num3); return oldExchange; } }  http://camel.apache.org/aggregate-example.html
IV. Camel in Action | runtime Running Camel  Known Deployment Options ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦ Standalone Java Application Java Web Start Spring Application Web Application Google App Engine J2EE Application JBI OSGi
IV. Riding Camel | runtime Running inside App Server Servlet component camel-example-servlet-tomcat  CXF Component camel-example-cxf-tomcat  Camel Web console actually « Web Component » 
IV. Review Camel is not a server  Camel is lightweight and embeddable  Camel supports Enterprise Integration paradigms from the JEE/Spring world.  Camel is not an ESB though it implements EIPs.  Camel doesn’t need but supports any messaging infrastructure (JMS / MQ)  Camel supports messaging at JBI abstraction level 
V. A real Use-Case  Multiple source FTP import to a centralized repository ◦ Collect files (0-5 x 2-20 Megs) from N different FTP endpoints, every hours on a 24/7 basis ◦ End-user configuration ◦ Logs : technical + business ◦ Monitor failures FTP SAN OPCON
Constraints         Big binary files : maybe JMS would not fit Implementation should fit some common corporate approach to entreprise intergation. Currently only SAN @ client. Do not need App Server yet. Client is technical-oriented : would like hands-on. Keep intact current logs / traces, but must be possible to connect on JDBC. Would be interested in indexing meta –infos within Alfresco (JSR-170). Central schedule/alerting infrastructure must be connected. Nagios could be an option at mid-term.
Real issues  Multiple source endpoints pattern : ◦ not found (!!!). FTP component is an active poller; no easy way to stop polling after having consuming all files.  No easy way to put into configuration the context path of Endpoint URIs 
Other applications… Camel as an ETL : Excel sources  PeopleSoft to ActiveDirectory  Bidirectional Federal <-> City Entreprises DB official data  Reconcilliation of mutations streams 
Conclusions  Open-Source ◦ Help yourself through debug mode on sources  Maven-based development ◦ « Real » modularity  Real Standards ◦ EIPs ◦ Maven, Spring, Java6. ◦ JBI (Glassfish)  Gateway to the future ◦ OSGI (Fuse ESB)  Camel. What else?
More information needed?  Camel website: ◦ http://camel.apache.org  Camel article: ◦ http://architects.dzone.com/articles/apachecamel-integration  FuseSource website: ◦ http://fusesource.com  Camel in Action book: ◦ http://manning.com/ibsen  Camel in Action Source code: ◦ http://code.google.com/p/camelinaction/
Questions

Check these out next

Apache Camel - Stéphane Kay - April 2011

Recommandé

Contenu connexe

Présentations pour vous(20)

Similaire à Apache Camel - Stéphane Kay - April 2011(20)

Plus de JUG Lausanne(20)

Dernier(20)

Apache Camel - Stéphane Kay - April 2011