Building a national Data Repository System Integration Architecture Overview

Hi my name is Alan MacKenzie, I’m the head of product management at a UK software engineering consultancy called Digirati I have been working essentially full time on RDSS for 2 years as the chief architect through the full lifecycle of the project. From the discovery phase of the project to Jisc closing with their first customer.
A architectural pattern for the integration of applications.
1 - Well defined atomic packets of data to be transported 2 - Sender doesn't block waiting for a response. Fire and usually forget 3 – Not loosely coupled - Communication from one application to another is via a middleman called a channel. Depending on how relaxed an individual is when discussing the integration they may use the word topic or queue instead of channel.
1 – There’s a slew of technologies around publish-subscribe, including from IBM, Amazon and Orcale. My advice is just to ignore all the marketing. 2- Remote API: RESTful webservice, SOAP
UML sequence diagram Read top to bottom Thick lines on top of the dotted lines are processing by the actor Simple – Not included things like transactional behavior. Publisher = Repository system Subscriber = Preservation system
I’ll be talking about both big picture strategic concerns as well as tactical benefits. As we talking about the benefits it will also give you a more detailed grasp of what this integration pattern is.
Integration is critical to RDSS. Thinking back years ago the vision was for researchers to only key in information once and systems to use auto-complete whenever possible. A core part of the value proposition to Jisc customers is they do not need to do the software engineering work themselves. Easily hit 10 different components integrated with the system, not too far away from that already. This project would not have been feasible if it was done point to point. With 10 different components there’s 90 integration touch points. It’s worth noting that the architecture also vastly improves the speed of delivery, vendor engineering teams only need to talk to Jisc with a focus on the API specification owned by a core team. There is no communication between different vendor teams.
Real photograph of a whiteboard from the project. Along the x axis we have time On the y axis we have risk priority Worth remembering RDSS is was an external customer facing product development project, not an internal facing IT project. We could be outmanoeuvred by our competitors or our customers could be disinterested when it comes to parting with cold hard cash. With this in mind it is best to optimise for flexibility early so you can easily make changes to the system to match what will sell in the market. The industry jargon for a product that a product has good product to market fit when it satisfies a strong demand. With this in mind optimising for the “-abilities” such as operability, scalability and maintainability become more of a priority later in the project. Eventually the priority of flexibility versus the “-abilities” flips.
So flexibility is very important, fortunately publish-subscribe messaging is a high level pattern and it comes with an entire toolbox of tried and tested patterns that can combined to give us the flexibility we need. This is a collection of icons representing most of the patterns from Enterprise Integration Patterns, the canonical reference for building message based integrations. It was written by Gregor Hohpe and Bobby Woolf and published in 2003. You can download the first two chapters for free from the website, I’ve put a link in the references of this presentation. Publish-subscribe with messaging is very popular with commercial enterprise organisations but from what I have seen from the pilot organisations not widely deployed in academia – at least the UK.
Publish-subscribe with messaging is boring and very well understood, though perhaps not widely deployed in academia. We want drama free delivery, the product can be innovative but the technology need not be. Bringing new products to market is risky, we want to stack the deck in our favour.
Final strategic benefit: Competition and portability Over the last couple of years I have spent a significant amount of time talking to higher education institutions about the frankly poor service they receive from many suppliers. An application that uses our standard message API can be easily swapped out for another compliant application in the same category in a couple of days work. This should hopefully prompt healthy competition rather than vendors retreating behind a moat.
1 - How easy the system is to operate We log every message sent and reject any invalid messages. This makes it easy to debug when something goes wrong as we have a high degree of visibility. 2 – There is almost zero scope for vendors to design and implement solutions that are not in line with the architecture. Maybe that sounds a bit mean spirited but with 50 to 65 developers working parallel split between 7 organisations and 3 or 4 timezones keeping everyone aligned becomes a full time job. 3 - When the message adaptors or gateways integrated into the applications are done well and in conjunction with the right channel the transportation of messages comes with strong guarantees on delivery and systems can publish or consume messages in a transactional manner. The asynchronous nature of messaging allows for systems to easy synchronise or “catch up” after a period of downtime and control the workload applied to them.
Messages that support reasonably straightforward for now. New version due out in the next few weeks. We use semver as a versioning system so you can infer the impact of a release from the version id. 1 - Spec states that every message comes with a header and body. Header handles the software engineering part and the body the domain. Wont talk further about the domain as this is more Dom’s area. Header includes items like A unique id for the message A timestamp for when the message was generated The name of the application that generated the message History 2 - Spec specified the transportation mechanism or the channel which all applications must use to send and receive messages, which is amazon kinesis. Cloud hosted and operated by AWS. Marginally cheaper to operate ourselves at a considerably cost in time. Worth noting that many vendors opted to building an adaptor from their native APIs to kinesis rather than baking the integration into their core application. Whatever works best for your particular application. 3 - Spec also specifies some required behaviour of applications when they send or receive messages. For example how to appropriated behave when a
This was the original architecture for topology but we have optimised this during delivery to be simpler and more secure. I know the diagram looks complicated but it’s actually quite simple. Middleware 2 micro services for routing, each could be rewritten in a day or two. First one is the message broker which simply duplicates any message from an input channel to all output channels Second one is a content based router which intelligently routes messages to the appropriate customer channel. Not represented here but 1 adaptor microservice to work around contractual issues, modifying incoming messages from a vendor application
Would highlight the final link, which is a 50 page report authored by yours truly comparing the various options for the integration architecture and proposing how the project should operate.
Questions?

Building a national Data Repository System Integration Architecture Overview

Recommandé

Recommandé

Contenu connexe

Similaire à Building a national Data Repository System Integration Architecture Overview

Similaire à Building a national Data Repository System Integration Architecture Overview (20)

Plus de Jisc RDM

Plus de Jisc RDM (20)

Dernier

Dernier (20)

Building a national Data Repository System Integration Architecture Overview

Notes de l'éditeur