Integrating Postgres with ActiveMQ and Camel

Justin Reock
Chief Architect
Rogue Wave Software
Justin has over 20 years’ experience
working in various software roles and is
an outspoken free software
evangelist, delivering enterprise
solutions and community education on
databases, integration work,
architecture, and technical leadership.
He is currently the Chief Architect at
Rogue Wave Software.

• I’m going to show you three slides
• They all have something in common….
• See if you can figure out what it is!

BillingFulfillment
CRME-Commerce

 None of these things exist in real life!

• Technically, MOM is any platform that sends and receives
messages between distributed applications
• This allows for asynchronous processing, and normalization of
data exchanges
• Clients connect to a messaging provider, and send and receive
messages via that provider

• Data normalization makes it a lot easier for disparate applications to
trade information
• Just like in web services, heterogeneous systems can agree on a data
“contract” and send and receive data in that format
• All the application needs is a client compatible with the messaging
provider
• And since messages are allowed to queue up on the message provider,
it also makes it very easy to perform asynchronous processing
• Most message providers offer additional benefits, like guaranteed
delivery, options for traffic shaping, high availability, and scaling

• ActiveMQ is a Java-based multi-purpose messaging middleware solution,
called a message broker, designed for high throughput and scale
• It is useful in a number of situations, including aggregating large amounts of
data in an efficient and safe manner, federating disparate or heterogeneous
applications across an enterprise, and/or moving data between
geographically separated data centers or applications
• ActiveMQ supports a number of connection protocols, allowing it to provide
connection points between applications coded in different languages, such as
Java, C++, .NET, PHP, etc
• Customers commonly migrate to ActiveMQ from commercial middleware
such as WebSphereMQ, Oracle AQ, TIBCO EMS, and/or MSMQ

• ActiveMQ is a mature and well-adopted platform, with thousands of companies
using it, and backed by a strong community
• It is by far the most flexible OSS message broker available, offering integration
points for dozens of application frameworks and languages
• ActiveMQ ships with Apache Camel, one of the most revolutionary integration
frameworks available, and shares developers and committers with the Apache
Camel product
• ActiveMQ can be tuned for massive scale, supporting large data sets, such as
those generated by the CERN large hadron collider
• Its configuration can support numerous traffic and implementation models
• ActiveMQ has a powerful statistics and metrics engine, exposing crucial details
about the health of the broker and the amount and size of message traffic and
client status

• ActiveMQ acts as integration middleware, sitting between
various enterprise applications, data sources, and APIs, and
“federating” data between the systems

• ActiveMQ can also be used as a reliable message buffer
• Applications can send messages into ActiveMQ and continue
working, knowing that ActiveMQ will guarantee delivery of
the message
• Or, ActiveMQ can be used to broadcast messages to several
different systems
• ActiveMQ supports transactions between the broker and
the client, so that exceptions can be caught and managed
• Using these constructs, you can easily create asynchronous
message patterns by sending or receiving messages from
ActiveMQ

Message grouping
Scheduled messages
Security framework
Metrics and
statistics
Network of Brokers
Inactivity monitoring
Transactions
Fault tolerance
Traffic Shaping
Plugin stack
Virtual destinations
Temporary destinations
Transformation
Load balancing
Client failover
Property encryption
Destination policies
Dispatch policies
Producer flow control
Message swapping
And of course… Camel!

• So, data routing and transformation in the Enterprise lends itself to a
lot of what you would call “boilerplate” code
• Boilerplate text, in publishing, is text that can be used and reused in
many places
• Boilerplate code is the same concept
• How many times have you written code to open up a web server,
push a file to an ftp site or drop a message into a message broker?
• Have you ever felt that you’re just writing different versions of the
same thing over and over again?
• You’re not alone
• Enter the concept of an Enterprise Integration Pattern…

• Camel is an open source MOM language developed by Apache and based
on Enterprise Integration Patterns
• Book by Gregor Hohpe and Bobby Woolf
• Centers around 60+ common patterns found in
Enterprise integration projects
• Defines these common patterns in a vendor
neutral way
• Camel aims to provide a language which
implements these patterns, in a style modeled
after the UNIX pipeline

• Content Based Router
– Where a message is routed to disparate endpoints
based on message criteria

• Content Enricher
– In which a basic message enters the pattern, takes data from
another source, and then adds it to the message

• Recipient List
– A message is broadcast to n channels based on message criteria. When this message is
broadcast to all channels regardless, it becomes a Multicast pattern

• Scatter-Gather
– In which messages are broadcast out to disparate endpoints and results are
aggregated into a single message

• Camel is maintained by the Apache Camel community
• It is a top-level project hosted at http://camel.apache.org
• The project’s chief maintainer is Claus Ibsen, author of Camel in Action
• As of August of 2017, the project has had almost 400 distinct committers, spanning over
30,000 commits
• It is written primarily in Java, and also contains some Scala code
• The project is licensed under the ALv2 (Apache License v2.0), which is considered a permissive
license
• So, Camel has been folded in to many other projects, including the Mule ESB and the
Switchyard project
• Camel can be run standalone, packaged into a WAR, or deployed as an OSGI bundle
• A POJO syntax as well as a Spring DSL XML syntax are made available for developers

• ActiveMQ needs to be able to persist messages in
something called a persistence journal
• This way, if the broker crashes, it can replay unsent
messages when it comes back up
• And Camel is just plain hungry for endpoints, and
databases are no exception!
Postgres works great with ActiveMQ and Camel!

• ActiveMQ needs a
persistence journal to
manage High Availability
• The default is a NoSQL
database called KahaDB
• The default is fast, but
difficult to replicate leading
to a SPOF architecture

When should I use…
Postgres/JDBC? KahaDB?
• I need true no-SPOF HA
• Throughput is less important
• I need to easily introspect
messages in the persistence
store
• I want familiar backup and
replication
• I don’t need true no-SPOF
HA
• Throughput is the most
important aspect
• I can lose messages or don’t
need to introspect them
• I can rely on storage backup

Quick reality check…
KahaDB
Postgres w/
Pooling
Postgres w/o
Pooling

• You can initialize Postgres as an ActiveMQ Persistence
Journal in a few simple steps:
– Configure Postgres with an ActiveMQ User and
Database
– Copy Postgres JDBC driver into ActiveMQ Classpath
– Configure ActiveMQ Persistence Adapter and Pooling
– Disable automatic schema creation going forward

Several components are used to connect…
ActiveMQ Code
Third-party
or
PGPooling
Postgres Code

From a fresh install…

pg_hba.conf
Adjust your trust settings accordingly for your environment…

• ActiveMQ’s default classpath is in the /lib directory of
the installation root

Create a Bean reference in /conf/activemq.xml
<bean id="postgres-ds"
class="org.postgresql.ds.PGPoolingDataSource"
destroy-method="close">
<property name="serverName" value="localhost"/>
<property name="databaseName" value="activemq"/>
<property name="portNumber" value="0"/>
<property name="user" value="activemq"/>
<property name="password" value="activemq"/>
<property name="dataSourceName" value="postgres"/>
<property name="initialConnections" value="1"/>
<property name="maxConnections" value="10"/>
</bean>

Set the <persistenceAdapter> to JDBC and reference the bean
<persistenceAdapter>
<jdbcPersistenceAdapter dataSource="#postgres-ds"
createTablesOnStartup=”true” />
</persistenceAdapter>
 Remember that createTablesOnStartup setting…..

Fire up the broker and check the persistence…
 Looks good, lets check out the schema …..

Lets see all tables…

Our ACK Table…

Our Lock table…

And our Messages table…

• To avoid an ugly (but benign) warning condition…
Turn off create tables on startup….

• Demonstrate Postgres as a persistence store for
ActiveMQ
• Persist some messages into the broker
• Introspect the messages
• Crash the broker
• Restart the broker and watch as messages replay
• Bonus: Demonstrate the HA failover functionality
I thought he’d never get there….

• Recall that Camel is a normalized routing engine meant to shuttle data
around
• Camel takes a component driven approach where components are
published for various endpoints
• One of those endpoints is the JDBC endpoint, meant to interact with JDBC-
compliant database drivers
• So, Camel can either read data from a database to enrich a message, or
produce data into a database from an external process
• Camel’s marshalling and transformation capabilities make it ideal for
building API abstraction in front of a Postgres database
Camel moves data around an enterprise…

• Camel-JDBC – Most basic integration, uses explicit SQL to
execute queries against a JDBC-compliant driver
• Camel-JPA – For using Object Relational Mapping
solutions like OpenJPA and Hibernate
• Camel-SQL – Variant of camel-jdbc which uses Spring IoC
to provide query details
• As well as NoSQL-specific components like MongoDB,
Cassandra, and Hadoop
Camel provides a lot of options…

• Camel uses a repeatable, normalized concatenation of
“Processor” and ”Message” objects in a group called an
Exchange
• There is an ”In” message, a “Processor”, and an “Out”
message
Processor
In Out

• And, you can chain these Exchanges together – just like piping
commands through UNIX – and form a Camel Route
• The “Out” message of a previous Exchange becomes the “In”
message of a new Exchange:
P
In Out
P
In Out
P
In Out

• Now here’s the magical part – a Processor can be anything you want
• Components in Camel work by creating Processors that perform various business
functions
• The components can be chained together to build Integration Patterns
• For instance, you can have an “FTP” processor that takes the In message and initiates an
FTP session
• Or you can have an “SMTP” processor that takes the In message and sends an email
with it
• And as we’ll see in a few minutes, there are several components designed specifically for
interacting with JDBC-compliant database drivers
• This is in addition to Camel’s many built-in patterns, such as LoadBalance, Multicast,
DataSets, etc.
• And there are a lot of components: http://camel.apache.org/components.html

• We will develop a simple RESTful API for writing and
reading to a table in Postgres
• GET requests to our API will output data in JSON
• POST requests will write the post body to the database
• We will do this using very little code, demonstrating
Camel’s propensity for rapid development of integration
logic

• To initialize our datasource, we will use a bean very similar to the one that we
used for ActiveMQ, since Camel is also powered by Spring:
<bean id="postgres-ds"
class="org.postgresql.ds.PGPoolingDataSource"
destroy-method="close">
<property name="serverName" value="localhost"/>
<property name="databaseName" value="cameljdbc"/>
<property name="portNumber" value=”5432"/>
<property name="user" value=”camel"/>
<property name="password" value="camel"/>
<property name="dataSourceName" value="postgres"/>
<property name="initialConnections" value="1"/>
<property name="maxConnections" value="10"/>
</bean>

<camelContext xmlns="http://camel.apache.org/schema/blueprint" id="JDBCDemos">
<dataFormats>
<json id="jack" library="Jackson"/>
</dataFormats>
<route>
<description>PostgresConf 2019 :: JDBC Marshalling</description>
<from uri="jetty:http://0.0.0.0:8080/jdbcWeb"/>
<when>
<simple>${headers.CamelHttpMethod} == "GET"</simple>
<setBody>
<constant>SELECT * FROM post_data</constant>
</setBody>
<to uri="jdbc:postgres-ds"/>
<marshal ref="jack"/>
</when>
<when>
<simple>${headers.CamelHttpMethod} == "POST"</simple>
<setBody>
<simple>
INSERT INTO post_data VALUES ('${date:now:yyyy-MM-dd'T'HH:mm:ss.SSSZ}','${body}’)
</simple>
</setBody>
<to uri="jdbc:postgres-ds"/>
</when>
</route>
</camelContext>
Yep, that’s all the code we need….

• This one is a lot easier to just demonstrate
• We will write the code together, and I will deploy out to
ServiceMix
• Camel can be deployed in lots of different ways, but
ServiceMix is easy for prototyping
• Bonus: Add a parameter to allow searching for specific
data
Let’s watch it work!

LinkedIn – Only Justin Reock in the world apparently!
Twitter – @jreock - But I do get a little political on
there….
Blog - http://blog.klocwork.com/author/justin-reock/
Email – justin.reock@roguewave.com
Feel Free to Reach Out – I Get Lonely…

Integrating Postgres with ActiveMQ and Camel

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

Similaire à Integrating Postgres with ActiveMQ and Camel

Similaire à Integrating Postgres with ActiveMQ and Camel (20)

Plus de Justin Reock

Plus de Justin Reock (8)

Dernier

Dernier (20)

Integrating Postgres with ActiveMQ and Camel

Notes de l'éditeur