Talk presented at the 2015 pgday.it Conference. It talk about MOM and how PostgreSQL send messages to Rabbitmq.

1. PGDay.IT 2015 – 23 Ottobre 2015 - Prato 1 di 33
Rabbitmq & Postgresql
Lucio Grenzi
l.grenzi@gmail.com

2. PGDay.IT 2015 – 23 Ottobre 2015 - Prato 2 di 33
Who is this guy?
Delphi developer since 1999
IT Consultant
Front end web developer
Postgresql addicted
Nonantolando.blogspot.com
lucio.grenzi
lucio grenzi

3. PGDay.IT 2015 – 23 Ottobre 2015 - Prato 3 di 33
AgendaAgenda
Messaging system
Rabbitmq introduction
Postgresql and pg_amqp
PostgreSQL Listen exchange

4. PGDay.IT 2015 – 23 Ottobre 2015 - Prato 4 di 33
Message-oriented middlewareMessage-oriented middleware
Message­oriented middleware (MOM) is software or hardware
infrastructure supporting sending and receiving messages
between distributed systems. MOM allows application modules
to be distributed over heterogeneous platforms and reduces
the complexity of developing applications that span multiple
operating systems and network protocols
­ https://en.wikipedia.org/wiki/Message­oriented_middleware ­

5. PGDay.IT 2015 – 23 Ottobre 2015 - Prato 5 di 33
Why Should I use MOM?Why Should I use MOM?
Need to scale
Monitoring data feeds
Need things done in order
Use the cloud
Decoupling applications by separating sending and receiving
data
Messaging is asynchronous

6. PGDay.IT 2015 – 23 Ottobre 2015 - Prato 6 di 33
Different protocolsDifferent protocols
Historically, message queuing has used proprietary, closed
protocols
By the time three standards have emerged which are used in
open source message queue implementations:
Advanced Message Queuing Protocol (AMQP)
Streaming Text Oriented Messaging Protocol (STOMP)
MQ Telemetry Transport (MQTT)

7. PGDay.IT 2015 – 23 Ottobre 2015 - Prato 7 di 33
STOMPSTOMP
Text based
Like AMQP, STOMP provides a message header with
properties, and a frame body
Does not deal in queues and topics
The broker must map onto something that it understands
internally such as a topic, queue, or exchange
Destinations are not mandated in the specification, different
brokers may support different flavours of destination

8. PGDay.IT 2015 – 23 Ottobre 2015 - Prato 8 di 33
STOMP BrokerSTOMP Broker
Apache ActiveMQ
HornetQ
Net::STOMP::Client (an open source client implementation in
Perl)
Open Message Queue (OpenMQ)
RabbitMQ (message broker, has support for STOMP)
Ruby server, also known as stompserver
http://stomp.github.io/implementations.html

9. PGDay.IT 2015 – 23 Ottobre 2015 - Prato 9 di 33
AMQPAMQP
Designed for interoperability between different vendors
wide range of features related to messaging
Lot of fine­grained control possible with such a rich feature set

10. PGDay.IT 2015 – 23 Ottobre 2015 - Prato 10 di 33
Protocol implementationsProtocol implementations
Amqp :SwiftMQ, Apache Qpid, Apache MQ,
Apache Apollo, RabbitMQ
Stomp: Apache MQ, Apache Apollo, RabbitMQ,
HornetQ, Stampy
(http://stomp.github.io/implementations.html#STOM
P_Servers)
MQTT: Apache MQ, Apache Apollo, RabbitMQ,
mosquitto, WebSphereMQ, HiveMQ, IBM Message
Sight
(https://github.com/mqtt/mqtt.github.io/wiki/server­
support)

11. PGDay.IT 2015 – 23 Ottobre 2015 - Prato 11 di 33
MQTTMQTT
Originally developed out of IBM’s pervasive computing team
Provides publish­and­subscribe messaging
Publish = data to topics
Subscribe = get data from topics
Designed for resource­constrained devices and low bandwidth,
high latency networks (ideal for mobile and “Internet of Things”)
MQTT’s strengths are simplicity (just five API methods)

12. PGDay.IT 2015 – 23 Ottobre 2015 - Prato 12 di 33
MQTT BrokerMQTT Broker
ActiveMQ,
Apollo
JoramMQ
Mosquitto
RabbitMQ
http://www.scalagent.com/IMG/pdf/Benchmark_MQTT_servers
­v1­1.pdf

13. PGDay.IT 2015 – 23 Ottobre 2015 - Prato 13 di 33
RabbitmqRabbitmq
open source message broker software (MPL)
Actively developed and with commercial support by Pivotal
written in Erlang
Client libraries to interface with the broker are available for all
major programming languages.
Gateways for HTTP, Streaming Text Oriented Messaging
Protocol (STOMP), and MQTT protocols

14. PGDay.IT 2015 – 23 Ottobre 2015 - Prato 14 di 33
Amqp native clientsAmqp native clients
.NET PHP Java Python
C /C++ Node.js Go Erlang
Javascript Ruby Perl Scala
Delphi Groovy Cobol Clojure

15. PGDay.IT 2015 – 23 Ottobre 2015 - Prato 15 di 33
Amqp packetAmqp packet
Headers
Properties
Bytea [] data
Headers: defined by the Amqp specification
Properties: contain arbitrary, application – specific,
information

16. PGDay.IT 2015 – 23 Ottobre 2015 - Prato 16 di 33
Amqp message propertiesAmqp message properties
Content-type MIME content type
Content-encoding MIME content
encoding
headers Message header field
table
Delivery-mode Non-persistent (1) or
persistent (2)
priority Message priority, 0 to
9
Correlation-id Application
correlation identifier
Reply-to Address to reply to
expiration Message expiration
specification
Message-id Application message
identifier
timestamp Message timestamp
type Message type name
User-id Creating user id
App-id Creating application
id
reserved Reserved, must be
empty

17. PGDay.IT 2015 – 23 Ottobre 2015 - Prato 17 di 33
Producer - ConsumerProducer - Consumer
Producer: a program that sends messages
Queus: a “buffer” where messages are stored
Consumer: a program that waits to receive messages

18. PGDay.IT 2015 – 23 Ottobre 2015 - Prato 18 di 33
Work queuesWork queues
Work Queue: it is used to distribute tasks among multiple
workers.
Tasks will be shared between workers
Useful in web applications where it's impossible to handle a
complex task during a short HTTP request window

19. PGDay.IT 2015 – 23 Ottobre 2015 - Prato 19 di 33
Publish/SubscribePublish/Subscribe
Deliver a message to multiple consumers (messages are going
to be broadcast to all the receivers.)
The exchange must know exactly what to do with a message it
receives (rules for that are defined by the exchange type)

20. PGDay.IT 2015 – 23 Ottobre 2015 - Prato 20 di 33
Exchange typesExchange types
Direct: the binding key must match the routing key exactly – no
wildcard support
Fanout: it broadcasts all the messages it receives to all the
queues it knows
Topic: same as Direct, but wildcards are allowed in the binding
key
Headers: no routing key, string matching in the message
headers property

21. PGDay.IT 2015 – 23 Ottobre 2015 - Prato 21 di 33
pg_amqppg_amqp
Amqp client for Postgresql
Allow Postgresql to communicate with other components in the
architecture
Stable
Suggested version 9.x, but it can works too with older versions
http://pgxn.org/dist/pg_amqp/
https://github.com/omniti­labs/pg_amqp

22. PGDay.IT 2015 – 23 Ottobre 2015 - Prato 22 di 33
How to install pg_ampq (v0.3.0)How to install pg_ampq (v0.3.0)
Wget http://api.pgxn.org/dist/pg_amqp/0.3.0/pg_amqp­0.3.0.zip
Cd pg_ampq­0.3.0
Make
Make install
env PG_CONFIG=/path/to/pg_config make && make install
shared_preload_libraries = 'pg_amqp.so' postgresql.conf→
CREATE EXTENSION amqp; (Postgresql >= 9.1)
psql ­d mydb ­f /path/to/pgsql/share/contrib/amqp.sql
(Postgresql < 9.1)

23. PGDay.IT 2015 – 23 Ottobre 2015 - Prato 23 di 33
pg_amqp (v 0.4.0)pg_amqp (v 0.4.0)
Only supported as a PostgreSQL Extension
PG versions prior to 9.1 and means to install it without the
extension system are no longer officially supported.
Added support for upgrading extension installation from 0.3.0
to 0.4.0
Add support for amqp message properties (delivery_mode,
content_type, reply_to and correlation_id)

24. PGDay.IT 2015 – 23 Ottobre 2015 - Prato 24 di 33
SetupSetup
Insert AMQP broker information into the`amqp.broker` table
Insert into ampq.broker(host,port,vhost,username,password)
values (localhost,'5432',null,'myuser','mypassword')

25. PGDay.IT 2015 – 23 Ottobre 2015 - Prato 25 di 33
How it worksHow it works
A process starts and connects to PostgreSQL and runs:
SELECT amqp.publish(broker_id, 'amqp.direct', 'routing­key',
'message');
snprintf(sql, sizeof(sql), "
SELECT host, port, vhost, username, password "
" FROM amqp.broker "
" WHERE broker_id = %d "
" ORDER BY host DESC, port", broker_id);

26. PGDay.IT 2015 – 23 Ottobre 2015 - Prato 26 di 33
DisconnectDisconnect
In case of needing to disconnect from a specific broker
select amqp.disconnect(broker_id);
do nothing if it is already disconnected

27. PGDay.IT 2015 – 23 Ottobre 2015 - Prato 27 di 33
PostgreSQL LISTEN ExchangePostgreSQL LISTEN Exchange
A RabbitMQ exchange type that translates PostgreSQL
NOTIFY messages to AMQP messages and publishes them to
bound queues
Require Postgresql >= 9.0
Rabbitmq plugin so unobtrusive to Postgresql environment
https://github.com/aweber/pgsql­listen­exchange

28. PGDay.IT 2015 – 23 Ottobre 2015 - Prato 28 di 33
Installation & configurationInstallation & configuration
Unzip file into your RabbitMQ plugins directory
rabbitmq­plugins enable pgsql_listen_exchange
To connect to PostgreSQL using the default
pgsql://postgres@localhost:5432/postgres
Edit rabbitmq.properties
[{pgsql_listen_exchange,
[
{host, "localhost"},
{port, 5432},
{dbname, "postgres"},
{user, "postgres"},
{password, ""}
]}
].

29. PGDay.IT 2015 – 23 Ottobre 2015 - Prato 29 di 33
Send a notifcationSend a notifcation
Psql (9.4.1)
postgres=# NOTIFY channel_name, 'This is a test';

30. PGDay.IT 2015 – 23 Ottobre 2015 - Prato 30 di 33
Receive a notificationReceive a notification
Psql (9.4.1)
Postgresl=# listen channel_name

31. PGDay.IT 2015 – 23 Ottobre 2015 - Prato 31 di 33
ResourcesResources
http://www.rabbitmq.com
http://blogs.vmware.com/vfabric/2013/02/choosing­your­m
http://interlinked.org/tutorials/postgresql.html

32. PGDay.IT 2015 – 23 Ottobre 2015 - Prato 32 di 33
Questions?Questions?

33. PGDay.IT 2015 – 23 Ottobre 2015 - Prato 33 di 33