Case of study about using Python, ZCA, Pyramid, Angular 4 and Deis Paas with Kubernets in AWS

1. From Legacy
To a Microservices Architecture
Python, ZCA, Pyramid, Angular 4 & Deis

2. About me
Ruda Porto Filgueiras
Software Architect at Briefy
http://briefy.co
From Brazil, based in Berlin
Twitter @rudaporto
ruda@briefy.co

3. The Company

4. Briefy Team

5. Creating visual content at scale
INPUT & IMPORT
We define a briefing together with
our Customers. After this we
receive a listing with all items to
shoot Hotels, Flats, Restaurants, etc.
Receive Orders
ASSIGNMENTS
We connect Creatives assigning
them to Orders in the location are
available. Creative schedules the
shoot with the third-party contact and
execute it.
Assign Creative
VALIDATE & PACKAGING
Creative upload assets and we
validate and do QA to make sure we
are following the briefing and specific
requirements. We deliver the content
to Customer: SFTP, GDrive, API.
QA & Delivery

6. Visual content matter

7. Embrace the change

8. Overview

9. A solution based on three pillars
API CENTRIC
To serve distinct use cases using the
same codebase, Briefy built its
solution around REST APIs.
Web, Mobile and customer integration
were the planned consumers.
MICROSERVICES
Single responsibility and distinct life
cycles for parts of the solution
pointed Briefy to use a micro
services architecture.
EVENTS BASED
Most of the communication between
the services could be done
asynchronously, so an event-based
— with messaging — solution was
implemented.

10. Code quality and reuse as principles
TESTING
Codebase testing with
Continuous Integration and
Continuous deployment
CODE QUALITY
Style guide and
documentation for
the entire codebase
DEVOPS
Use existing toolset to
minimise devops work.
CODE REUSE
Shared packages/libraries
Build upon existing OSS
codebase

11. Tech stack
INFRA
Storage: GDrive & S3
Messages: SQS
Cluster:
Kubernetes 1.7.8
DEIS 2.18
API
Python 3.6.3
Zope Component Architecture
Pyramid 1.9 / Cornice
SQLAlchemy 1.2
Postgresql 9.6
DEV TOOLS
Repo: GitHub
CI: TravisCI
Logging: Logstash
Reporting: Sentry /
NewRelic / Slack
WEB FRONTEND
Angular 4
Typescript
Generic views
Generic Listings
Generic CTAs

12. Infrastructure

13. AWS
THREE ENVIRONMENTS
Live: eu-central-1
Staging: us-east-1
Development: eu-west-1
SERVICES
Message routed using SQS
S3 for static data & assets
Kubernets Cluster on EC2
Dies Workflow

14. Kops
AUTOMATE
Kubernets cluster creation with a single
command line, including bastion and nat
High availability, multiple SDN options
MAINTAIN
Cluster upgrade and change
Export format templates to cloud formation and
terraform

15. Kubernets
KUBERNETES OVER EC2
One master, n minions
DEIS ON TOP OF KUBERNETES
Workflow solution similar to Heroku PaaS
EC2 WITH AUTO SCALING
T2 and M4 instances.

16. Deis Workflow
DOCKER
Automatic build and save images
SIMPLE
git push deis master
TWELVE-FACTOR
And also support legacy if necessary

17. Devops
MONITORING WITH NEWRELIC
Servers and applications
REPORTS ON SLACK / EMAIL
Deploy, errors and alerts posted on
tech channel
AUTOMATION WITH ANSIBLE
AWS and Cloudflare integration

18. Code

19. Shared codebase
BRIEFY.COMMON
Events / Queue
Base Models / Mixins
Logging
Workflow
BRIEFY.WS
REST API blueprints
Validation
Error handling
BRIEFY.GDRIVE
Common GDrive actions
Create, Copy, Metadata,
List, Download
BRIEFY.S3
Common S3 actions
Create, Copy, Metadata,
List, Download

20. briefy.common
QUEUE WORKER
Queue worker to consume
SQS messages and
dispatch
WORKFLOW
Workflow engine to create
solid and secure business
processes
EVENTS
Event firing to internal
component architecture
and to SQS
MODELS
Base mixins and sqlalchemy
models with versioning, local
roles support and polymorphic
types
CACHE
Cache object serialization
on redis with dogpile.cache

21. briefy.ws
FILTER
Sort and filter listings
automatically based on
model fields
RESOURCES
Base classes to exposure
models as API endpoints:
REST, workflow transitions,
versioning
EVENTS
Object lifecycle events
firing
SCHEMA
Automatic colander schema
generation for validation
and deserialization
SECURITY
Resource factory to
dynamically compute ACLs
based on model workflow and
local roles

22. Productivity tools
CHECKSTYLE
Linting rules and code style
guide
COOKIECUTTERS
Templates to create new
packages and services
DOCKER IMAGES
Python 3.6.3
Plone 5.0.8
Nginx / Varnish

23. Services

24. Core services
MS.LAURE
Automatic quality
assurance and renaming of
assets
LEICA
Workflow and scheduling
solution.
API-based
CHOREOGRAPHER
Events handling and
routing.
ROLLEIFLEX
Authorisation and user
management service.
ALEXANDRIA
Digital Assets management
solution
API-based

25. Support services
CMS
Content management
backend for Briefy website.
Using Plone CMS.
THUMBOR
Image server handling
automatic cropping, resizing
and image transforms
BRIEFY.SLACK
Service handling Slack
notifications.
BRIEFY.MAIL
Service handling Mail
notifications using Mandrill
as the provider
HOTELS CRM
CRM solution to deal with
Hotels acquisition for the
City Packages initiative.
LEADS
Service handling leads and
quotes coming from
briefy.co website.
SITEMAP
Service that generates a
sitemap.xml for CMS and static
content.
SEO helper
API GATEWAY
Gateway and router
handling public accesses
to Briefy API

26. Leica: from briefing to assets

27. The core of Briefy
WORKFLOW & PERMISSIONS
Every model have its own workflow
HUNDREDS OF EVENTS
Workflow transition, task execution, object creation
and update.
DATA MODELS
Customer, Project, Order, Assignment, Creative
and User profiles
Being the scheduling and workflow solution for visual content
creation, Leica is the core of Briefy operations.

28. Elements of Leica
BRIEFY.FRONTEND
Web application developed with
Angular4, connects to Leica API.
This app has 3 distinct users:
Customer, Creative and Briefy team
member, with distinct permissions.
User Interface
BRIEFY.LEICA
Domain models, permissions,
business logic and API endpoints.
Codebase is well documented and
90% code coverage.
API
LEICA TASKS
Scheduled tasks execute workflow
transitions, generate reports and
post KPIs.
Share the same codebase as
briefy.leica
Tasks

29. Domain Models
CUSTOMER
Company requesting visual
content creation.
PROJECT
A grouping of orders from the
same customer.
Usually attached to one
contract (or agreement)
ORDER
A content creation request.
When it is not confirmed, it is
called Lead.
Customer oriented.
ASSIGNMENT
A content creation
execution schedule.
Creative oriented.
JOB POOL
Groups available Orders to
be offered to a group of
pre-selected Creatives.
CREATIVE
A visual content creator -
Photographer,
videographer — profile.
CUSTOMER USER
Profile of a user attached to
a Customer.
BRIEFY USER
Profile of a Briefy team
member user.

30. Managing visual content creation at scale
cost is complex
Confidential
Managing
Visual
Content
Creation Is
Complex

31. From Events
To Actions

32. Every action, one event (1)
ARIANA
Assign a creative to an Order
ORDER.WORKFLOW.ASSIGN
2017-07-27 15:46:10.03267
120
Times

33. Every action, one event (2)
OPHELIE
Validates a Creative
PROFESSIONAL.WORKFLOW.VALIDATE
2017-07-31 15:55:11.436015
213
Times

34. Events trigger new actions
ASSIGN. IS SCHEDULED
Assignment is scheduled on
Leica, generating an event
EVENT IS ROUTED
Actions are registered to this
event, i.e.: Send email to Creative
EMAIL SENT TO CREATIVE
Using Mandrill, a message is sent
to the Creative
An event on Leica will be sent to Choreographer, there it will be routed to other micro services
to execute new actions.
This flow is asynchronous and all messages are sent using SQS queues.
BRIEFY.LEICA BRIEFY.CHOREOGRAPHER BRIEFY.MAIL

35. Thank you!
@rudaporto
ruda@briefy.co
+49 176 28607522
Rudá Porto Filgueiras