Relato da experiência com Pyramid, Python e micro-serviços na Briefy. Esta palestra foi apresentada em 11/11/2017 no encontro do GruPy-SP. Baseada na apresentação "From Legacy To a Microservices Architecture" ministrada por Rudá Porto na Plone Conference 2017

1. From Legacy
To a Microservices Architecture
Python, ZCA, Pyramid, Angular 4 & Deis

2. About me
Érico Andrei
CTO at Briefy
https://briefy.co
From Mooca, based in Berlin
Twitter @ericof
erico@briefy.co

4. Briefy

6. The Team

8. Overview

9. A solution based on three pillars
API CENTRIC
To serve distinct use cases using the
same codebase, Briefy built its
solution around REST APIs.
Web, Mobile and customer integration
were the planned consumers.
MICROSERVICES
Single responsibility and distinct life
cycles for parts of the solution
pointed Briefy to use a micro
services architecture.
EVENTS BASED
Most of the communication between
the services could be done
asynchronously, so an event-based
— with messaging — solution was
implemented.

10. Code quality and reuse as principles
TESTING
Codebase testing with
Continuous Integration and
Continuous deployment
CODE QUALITY
Style guide and
documentation for
the entire codebase
DEVOPS
Use existing toolset to
minimise devops work.
CODE REUSE
Shared packages/libraries
Build upon existing OSS
codebase

11. Tech stack
INFRA
Storage: GDrive & S3
Messages: SQS
Cluster:
Kubernetes 1.7.8
DEIS 2.18
API
Python 3.6.3
Zope Component Architecture
Pyramid 1.9 / Cornice
SQLAlchemy 1.2
Postgresql 9.6
DEV TOOLS
Repo: GitHub
CI: TravisCI
Logging: Logstash
Reporting: Sentry /
NewRelic / Slack
WEB FRONTEND
Angular 4
Typescript
Generic views
Generic Listings
Generic CTAs

12. Infrastructure

13. AWS
THREE ENVIRONMENTS
Live: eu-central-1
Staging: us-east-1
Development: eu-west-1
SERVICES
Message routed using SQS
S3 for static data & assets
Kubernets Cluster on EC2
Dies Workflow

14. Kubernetes
KUBERNETES OVER EC2
One master, n minions
DEIS ON TOP OF KUBERNETES
Workflow solution similar to Heroku PaaS
EC2 WITH AUTO SCALING
T2 and M4 instances.

15. Deis Workflow
DOCKER
Automatic build and save images
SIMPLE
git push deis master
TWELVE-FACTOR
And also support legacy if necessary

16. Code

17. Shared codebase
BRIEFY.COMMON
Events / Queue
Base Models / Mixins
Logging
Workflow
BRIEFY.WS
REST API blueprints
Validation
Error handling
BRIEFY.GDRIVE
Common GDrive actions
Create, Copy, Metadata,
List, Download
BRIEFY.S3
Common S3 actions
Create, Copy, Metadata,
List, Download

18. briefy.common
QUEUE WORKER
Queue worker to consume
SQS messages and
dispatch
WORKFLOW
Workflow engine to create
solid and secure business
processes
EVENTS
Event firing to internal
component architecture
and to SQS
MODELS
Base mixins and sqlalchemy
models with versioning, local
roles support and polymorphic
types
CACHE
Cache object serialization
on redis with dogpile.cache

19. briefy.ws
FILTER
Sort and filter listings
automatically based on
model fields
RESOURCES
Base classes to exposure
models as API endpoints:
REST, workflow transitions,
versioning
EVENTS
Object lifecycle events
firing
SCHEMA
Automatic colander schema
generation for validation
and deserialization
SECURITY
Resource factory to
dynamically compute ACLs
based on model workflow and
local roles

20. Productivity tools
CHECKSTYLE
Linting rules and code style
guide
COOKIECUTTERS
Templates to create new
packages and services
DOCKER IMAGES
Python 3.6.3
Plone 5.0.8
Nginx / Varnish

21. Services

22. Core services
MS.LAURE
Automatic quality
assurance and renaming of
assets
LEICA
Workflow and scheduling
solution.
API-based
CHOREOGRAPHER
Events handling and
routing.
ROLLEIFLEX
Authorisation and user
management service.
ALEXANDRIA
Digital Assets management
solution
API-based

23. Support services
CMS
Content management
backend for Briefy website.
Using Plone CMS.
THUMBOR
Image server handling
automatic cropping, resizing
and image transforms
BRIEFY.SLACK
Service handling Slack
notifications.
BRIEFY.MAIL
Service handling Mail
notifications using Mandrill
as the provider
HOTELS CRM
CRM solution to deal with
Hotels acquisition for the
City Packages initiative.
LEADS
Service handling leads and
quotes coming from
briefy.co website.
SITEMAP
Service that generates a
sitemap.xml for CMS and static
content.
SEO helper
API GATEWAY
Gateway and router
handling public accesses
to Briefy API

24. Leica: from briefing to assets

25. The core of Briefy
WORKFLOW & PERMISSIONS
Every model have its own workflow
HUNDREDS OF EVENTS
Workflow transition, task execution, object creation
and update.
DATA MODELS
Customer, Project, Order, Assignment, Creative and
User profiles
Being the scheduling and workflow solution for visual content
creation, Leica is the core of Briefy operations.

26. Elements of Leica
BRIEFY.FRONTEND
Web application developed with
Angular4, connects to Leica API.
This app has 3 distinct users:
Customer, Creative and Briefy team
member, with distinct permissions.
User Interface
BRIEFY.LEICA
Domain models, permissions,
business logic and API endpoints.
Codebase is well documented and
90% code coverage.
API
LEICA TASKS
Scheduled tasks execute workflow
transitions, generate reports and
post KPIs.
Share the same codebase as
briefy.leica
Tasks

27. Domain Models
CUSTOMER
Company requesting visual
content creation.
PROJECT
A grouping of orders from the
same customer.
Usually attached to one
contract (or agreement)
ORDER
A content creation request.
When it is not confirmed, it is
called Lead.
Customer oriented.
ASSIGNMENT
A content creation
execution schedule.
Creative oriented.
JOB POOL
Groups available Orders to
be offered to a group of
pre-selected Creatives.
CREATIVE
A visual content creator -
Photographer,
videographer — profile.
CUSTOMER USER
Profile of a user attached to
a Customer.
BRIEFY USER
Profile of a Briefy team
member user.

28. Managing visual content creation at scale
cost is complex
Confidential
Managing
Visual
Content
Creation Is
Complex

29. From Events
To Actions

30. Every action, one event (1)
ARIANA
Assign a creative to an Order
ORDER.WORKFLOW.ASSIGN
2017-07-27 15:46:10.03267
120
Times

31. Every action, one event (2)
OPHELIE
Validates a Creative
PROFESSIONAL.WORKFLOW.VALIDATE
2017-07-31 15:55:11.436015
213
Times

32. Events trigger new actions
ASSIGN. IS SCHEDULED
Assignment is scheduled on
Leica, generating an event
EVENT IS ROUTED
Actions are registered to this
event, i.e.: Send email to Creative
EMAIL SENT TO CREATIVE
Using Mandrill, a message is sent
to the Creative
An event on Leica will be sent to Choreographer, there it will be routed to other micro services
to execute new actions.
This flow is asynchronous and all messages are sent using SQS queues.
BRIEFY.LEICA BRIEFY.CHOREOGRAPHER BRIEFY.MAIL

33. Thank you!
erico@briefy.co
@ericof
https://briefy.co
Érico Andrei