Nello sviluppo di un servizio software, da sempre, una grande quantità di lavoro è richiesta per curare aspetti, sì necessari all’operatività di tale applicazione, ma non strettamente legati alle funzionalità offerte ai clienti. I principali vantaggi aziendali, sia economici che organizzativi, derivati dall’adozione di DeFacto, sono: - Capacità di distribuire l'intera fabbrica in un cluster Kubernetes, nuovo o esistente, in indipendenza dal fornitore cloud Governance della software factory tramite modelli standard e processi olistici altamente personalizzabili atti a supportare: - Creazione di risorse software e infrastrutturali - Generazione di build, bake e deploy pipelines standards - Tracciabilità e audit completi - Ottimizzazione del flusso di valore e riduzione del tempo medio di dispiegamento - Visibilità completa del WIP - Riduzione dei passaggi di consegna grazie ad un approccio shift-left e all'automazione di processo - Trasferimento di esigenze applicative dal dominio dello sviluppo al livello dell'infrastruttura - Politiche di comunicazione distribuita (Circuit breaks, retries, fault injections) Logging, Monitoring, Alerting, Distributed Tracing, Fault Tolerance - Orchestrazione del carico di lavoro - Capacità di integrare facilmente tecnologie di terze parti - Riduzione dei rischi legati al cofiguration drift grazie a un'infrastruttura immutabile e ad un approccio XaC (Everything as Code) - Abbattimento delle barriere di utilizzo grazie ad uno Smart Factory Assistant che permette di effettuare le principali operazioni semplicemente parlando alla Software Factory - Dimensionamento un cluster Kubernetes minimizzando il costo complessivo dell'infrastruttura grazie al componente di ottimizzazione sviluppato dal DIMES.

2. DeFacto – Project
DeFacto è un progetto di ricerca e sviluppo cofinanziato dalla
Regione Calabria nell’ambito del POR
CALABRIA FESR-FSE 2014-2020, ASSE I – Promozione della
Ricerca e dell’innovazione
Obiettivo specifico 1.2 “Rafforzamento del sistema innovativo
regionale e nazionale”
Azione 1.2.2 “Supporto alla realizzazione di progetti complessi
di attività di ricerca e sviluppo su poche aree tematiche di
rilievo e all’applicazione di soluzioni tecnologiche funzionali alla
realizzazione delle strategie di S3”.

3. DeFacto – The Partners
La Herzum S.r.l. Unipersonale è una società di
consulenza internazionale, leader nelle metodologie Agili
e DevOps e uno dei principali fornitori di servizi Atlassian
nel Mondo. È stata costituita nel 2006 con sede legale a
Genova e sede operativa a Rende. Dal 2012 anche la
sede legale è stata trasferita a Rende.
Si occupa, oltre che di consulenza informatica, anche
delle attività di Ricerca e Sviluppo del gruppo Herzum,
costituito nel marzo 2000 a Chicago. Ad oggi il gruppo
Herzum ha uffici in Italia, Regno Unito, Svizzera, India. Il
gruppo Herzum gestisce progetti e organizzazioni dalla
fase di avvio al livello Fortune 100.
Unical – DIMES (Dipartimento di Ingegneria Informatica,
Modellistica, Elettronica e Sistemistica) si è costituito nel 2013
per aggregazione di gran parte del personale afferente al
dipartimento di Elettronica, Informatica e Sistemistica ed al
dipartimento di Modellistica per l’Ingegneria. Al dipartimento
afferiscono 74 tra professori e ricercatori e 22 unità di
personale tecnico-amministrativo.
Il DIMES promuove e coordina le attività di ricerca e di didattica
nei settori dell’ingegneria informatica, della modellistica
nell’ambito della meccanica strutturale e dei processi chimici
innovativi, dell’elettronica, dei campi elettromagnetici, delle
telecomunicazioni, dell’automazione, della ricerca operativa e
dell’ottimizzazione. Il DIMES è uno dei 180 "Dipartimenti di
Eccellenza" selezionati dal MIUR su base nazionale per il
periodo 2018-2022.

4. DeFacto – The Goals
Herzum Srl Unipersonale, as Software Factory, and DIMES as a university research reality,
with the realization of this research and development project, intend to:
• define a Digital Ecosystem for a "FACTOry of the Future" software
• realize a prototype of a technological platform with Defacto's intelligent services
• make the industrial production line and the life cycle of software components more
efficient, intelligent and competitive
• design models and methods of mathematical optimization for the allocation of resources
in the digital ecosystem
• improve the competitiveness of Herzum in the "digital transformation" services sector
• make DeFacto operational in the IoT environment, creating a software prototype closely
integrated in the digital ecosystem.

5. DeFacto – Project Structure
• WP1 - Research and definition of a digital ecosystem, named DeFacto, as a model of a
Software Factory of the Future
• WP2 - Mathematical models for the optimal allocation of resources in the digital
ecosystem
• WP3 - Methods for the efficient resolution of resource allocation models in the digital
ecosystem
• WP4 - Implementation of a significant prototype of technological platform and smart
services of Defacto
• WP5 - Search and definition of a component that enables Defacto to operate in the IoT
context
• WP6 - Realization of a demonstration prototype of the IoT component in the Defacto
digital ecosystem
• WP7 - Dissemination and exploitation of project results

6. Amazon - May Deployment Stats
(production hosts & environments only)
• 11.6 seconds Mean time between deployments (typical
weekday)
• 1,079 Max # of deployments in a single hour
• 10,000 Mean # of hosts simultaneously receiving a
deployment
• 30,000 Max # of hosts simultaneously receiving a deployment
DeFacto Drivers
Why we need
DeFacto?
Being like Amazon is HARD
Every company wants to be like
Amazon

7. DevOps Adoptions Barriers
Eliminate DevOps Adoption Barriers
Reconcile DevOps agility with
Compliance and Governance needs
Improve Collaboration and Sharing
Reduce the Software Factory
operational complexity

8. How to facilitate the
Cultural Change?
Culture
Cultural Change and Larman’s Laws
The organization’s
culture is not going
to change overnight
1. Organizations are implicitly optimized to avoid changing the
status quo middle- and first-level manager and “specialist”
positions & power structures.
2. As a corollary to (1), any change initiative will be reduced to
redefining or overloading the new terminology to mean
basically the same as status quo.
3. As a corollary to (1), any change initiative will be derided as
“purist”, “theoretical”, “revolutionary”, "religion", and “needing
pragmatic customization for local concerns” — which deflects
from addressing weaknesses and manager/specialist status
quo.
4. As a corollary to (1), if after changing the change some
managers and single-specialists are still displaced, they become
“coaches/trainers” for the change, frequently reinforcing (2)
and (3).
5. Culture follows structure.
Craig Larman

9. Communication and Sharing as a leverage for the Cultural Change
Cross-team sharing is key to scaling DevOps success.
We discovered that the foundational practices — the practices
with the most significant impact across the entire DevOps
evolutionary journey — are dependent on sharing, one of the
key pillars of DevOps.
Puppet | State of DevOps Report 2018

10. Mattermost
Amplifiers

11. Mattermost Amplifiers
for Atlassian

12. Mattermost Amplifier for JIRA
Issue
Events
Workflow
Transitions
Capture
everything in
JIRA
You decide
what ends up
in Mattermost
Configure
Destination
Channels
Customize
Messages
/command Issue-X
Act in JIRA
with chat
commands
System
Events
User
Events

13. System
Events
Issue
Events
Workflow
Events
User
Events
Configure
Destinatio
n Channels
/command Issue-X
Act in JIRA
with chat
commands
Herzum-MattermostJAVAlibrary
Message Buffer
Template
Service
Notification Service
Event Listener
&
Postfunctions
Admin Screen
Push Ready to go Messages
Notification Job
Pop Ready to go Messages
use
Issue Tab Panel Project Tab Panel
use
MattermostRESTAPI
Reacts to
use
use
use
Mattermost Amplifier
for JIRA
use
use
use
Customize
Messages
Mattermost Amplifier for JIRA - Architecture

14. Amplifiers Example Screenshots

15. Holistic
Software Factory

16. The software factory we WANT
Production OrchestrationRelease & DeployDevelopment
Planning
Build
Coding
Scaling & HealingDeploy & RollbacksRelease
Continuous Improvement KPI Alerting
Monitoring
Infrastructure Management
PortfolioManagemet
ProjectManagement
ProductManagemt
Continuous
Integration
Testing
Continuous
Release
Source Control Managemet
Continuous
Deployment
Change Management
Monitoring
Reporting
Alerting
Incident Management
Knowledge Management
Communication Management
BI & Big Data
AnalysisAgile
Development
Branching & Merging Code Reviews Quality GatesTechnical Debt Analisys
IaC
IaaS Multi Cloud Containers
Workload
Orchestration
Service Mesh
Configuration
Management
Artifacts Management
Dep MgmtSecurityCompliance
DevSecOps based Service Lifecycle Management
Continuous Quality
Continuous Security
Storage Metadata Versioning
Promotio
n
Distribution
Holistic
• Cohesive set of tools
• Visible processes
Extensible
• Custom SOPs
• Easy to integrate with 3rd Party
technologies
Transparent
• Easy to use and operate
• Focus on real work

17. The software factory we GOT
Infrastructure Management
PortfolioManagemet
ProjectManagement
ProductManagemt
Continuous Integration
Testing
Continuous
Release
Source Control Managemet
Continuous
Deployment
Change Management
Monitoring
Reporting
Alerting
Incident Management
Knowledge Management
Communication Management
BI & Big Data Analysis
Agile Development
Branching & Merging Code Reviews Quality GatesTechnical Debt Analisys
IaC
IaaS Multi Cloud Containers
Workload
Orchestration
Service Mesh
Configuration
Management
Artifacts Management
Dep MgmtSecurityCompliance
DevSecOps based Service Lifecycle Management
Continuous Quality
Continuous Security
Storage Metadata Versioning Promotion Distribution

18. The software factory we GOT
Siloed
• Tools fragmentation
• Complex legacy integration
Complex
• Users need to learn factory
technologies to perform real
work
• SW and HW architecture
difficult to provision and
operate
Infrastructure Management
PortfolioManagemet
ProjectManagement
ProductManagemt
Continuous
Integration
Testing
Continuous
Release
Source Control Managemet
Continuous
Deployment
Change Management
Monitoring
Reporting
Alerting
Incident Management
Knowledge Management
Communication Management
BI & Big Data
AnalysisAgile
Development
Branching & Merging Code Reviews Quality GatesTechnical Debt Analisys
IaC
IaaS Multi Cloud Containers
Workload
Orchestration
Service Mesh
Configuration
Management
Artifacts Management
Dep MgmtSecurityCompliance
DevSecOps based Service Lifecycle Management
Continuous Quality
Continuous Security
Storage Metadata Versioning
Promotio
n
Distribution
Production OrchestrationRelease & DeployDevelopment
Planning
Build
Coding
Scaling & HealingDeploy & RollbacksRelease
Continuous Improvement KPI Alerting
Monitoring

19. Production OrchestrationRelease & DeployDevelopment
Planning
Build
Coding
Scaling & HealingDeploy & RollbacksRelease
Continuous Improvement KPI Alerting
Monitoring
Missing the forest looking at trees
Simple actions at Factory level require a lot of manual
work across the toolchain
Entities manually created per tool
Agile Boards
Workflows
IssueTypes
Notification
Darshboards
Customfields
Projects
Dashboards
Permissions
Notification
Pages
Spaces
Projects
Automations
Queues
Notification
KB
SLAs
Portal
Agents
Deployments
Tasks
Plans
Projects
Notification
Permissions
Pull Requests
Merge Checks
Branches
Repositories
Projects
Notification
Permissions
Entities relationships are logical
and kept outside of the tools
Intrinsically Complex because the
number and heterogeneity of the
tools

20. Production OrchestrationRelease & DeployDevelopment
Planning
Build
Coding
Scaling & HealingDeploy & RollbacksRelease
Continuous Improvement KPI Alerting
Monitoring
How to do better?
Simple actions at Factory level require a lot of manual
work across the toolchain
Entities manually created per tool
Agile Boards
Workflows
IssueTypes
Notification
Darshboards
Customfields
Projects
Dashboards
Permissions
Notification
Pages
Spaces
Projects
Automations
Queues
Notification
KB
SLAs
Portal
Agents
Deployments
Tasks
Plans
Projects
Notification
Permissions
Pull Requests
Merge Checks
Branches
Repositories
Projects
Notification
Permissions
Entities relationships are logical
and kept outside of the tools
Intrinsically Complex because the
number and heterogeneity of the
tools
Factory as a whole
K
Continuous
Delivery +
Deploymen
t
O
p
e
r
a
ti
o
n

21. Nucleus

22. The Foundation - Smart Infrastructure
Version Control
Code Reviews Build & Bake
Plan & Track Static Analysis
QAGates
Trigger
Analysis
Code Changes
Continuous
Quality
Artifact
s
Release
Operation
Smart Deploy
Artifacts
Security
Compliance
Multi Cloud Infrastructure as a Service
Virtual Containerized Infrastructure – Workload Orchestration
Logging – Monitoring – Alerting - Reporting
Distributed Tracing
Service Mesh
Chaos Engineering KubeMonkey
Agile Development Continuous Delivery + Deployment Operation
Service Requests – Incident Management
Knowledge management - WCM
Real Time Collaboration
Elastic Build Infrastructure
Self Provisioning of Infrastructure Resources
ChatOps
Centralized Governance – Central Console Factory Orchestration – Processes Definition and Governance
Herzum
DeFacto
Orchestra
Event
Driven
Fault
Tolerant
DeFactoBackboneRealTime-MessageStreaming
Platform
Microservices
Based
KubernetesNative
Herzum
OnRamp
Nucleus
Customize Toolset
Install & Upgrade Monitoring
Backup & Restore
Deploy Orchestra
Able to work with the major cloud
providers
Simplify the provisioning and
installation of Kubernetes cluster
Power the infrastructure with the
required smart components
Deploy a minimal set of factory
components (the Nucleus)
Leverage on the nucleus to deploy
the remaining tools

23. Orchestra

24. Orchestra Anatomy
Elastic Build Infrastructure
Self Provisioning of Infrastructure Resources
ChatOps
Centralized Governance – Central Console Factory Orchestration – Processes Definition and Governance
Herzum
DeFacto
Orchestra
Event
Driven
Fault
Tolerant
DeFactoBackboneRealTime-MessageStreaming
Platform
Microservices
Based
KubernetesNative
Agile Development Continuous Delivery + Deployment Operation
Smart Deploy
Virtual Containerized Infrastructure – Workload Orchestration
Distributed Tracing
Service Mesh
Chaos Engineering KubeMonkey
Service Requests – Incident Management
Knowledge management - WCM
Version Control
Code Reviews
Artifacts
Security
Compliance
Multi Cloud Infrastructure as a Service
Virtual Containerized Infrastructure – Workload Orchestration
Service Mesh
Build & Bake
Plan & Track Static Analysis
QAGates
Trigger
Analysis
Code Changes
Continuous
Quality
Artifact
s
Release
Operation
Smart Deploy
Logging – Monitoring – Alerting - Reporting
Distributed Tracing
Chaos Engineering KubeMonkey
Service Requests – Incident Management
Knowledge management - WCM
Real Time Collaboration
Herzum
OnRamp
Nucleus
CustomizeToolset
Install&UpgradeMonitoring
Backup&Restore
• Ability to react to every
software factory event
• Orchestrate the tools using
BPMN
• Scalable and fault tolerant
architecture
• Tracing, logging, auditing
out of the box
• Asset management
powered by JIRA platform

25. Real Time - Message Streaming Platform
DeFacto Addon
Event Based
Pluggable Applications Webhooks Applications
Not Event Based
Database Powered
Event
Listener
Application Specific Adapter
Service Layer
Adapter Client
DeFacto
Webhooks
Internal
Event
Listener
Application Specific Adapter
Service Layer
Transaction Log
Backbone Client Backbone Client
Content Data Capture
DEFACTOBackbone
Defacto Backbone and Application Integration
• Go beyond Point to Point out of the box
integration
• Allow both open and legacy applications
to be integrated
• Enable the information flow to freely
span across the tool boundaries
• Ability to react to, monitor and
governate every Factory event

26. Tool Adapter
JIRA
Service Layer
Orchestrator Engine
Messaging Engine
Search Engine
Reporting Engine
Notification Engine
Permissions Engine
Lifecycle
HumanTask
Engine
JIRA REST Layer
JIRA
DeFacto
Addon
Client
Kubernetes Pod
Rest Client
Rest Client
Persistence Engine
Client
Kubernetes Pod
Functional Service
Client
Topic 1
Topic N
Producer
Producer
Producer
Consumer
Consumer
Consumer
DeFacto Event Log
TicketingEL
Kubernetes Pod
Orchestra - Component Anatomy
• Possibility to connect different
tools depending on which
adapter is used
• Rely on JIRA platform for
persistence, search, reporting,
notification , permissions etc.

27. Alexa Skill
ChatOps Component
Real Time - Message Streaming Platform
ChatOps
Component
Serverless
Functions
Instant Messaging
Room
holistic Factory Component
Service Layer
holistic Factory
Client
Commands
Backbone Client
Tool Chain
Intents
NLP
Hooks
BOT
Virtual Infrastructure
Component
Tools
• Lower the technical skill needs
to operate the Factory
• Provide a natural language
interface
• Ability to execute factory
processes typing command into
a chat room
• Ability to execute factory
processes simply talking to
Spoke (Factory virtual assistant,
powered by Alexa)

28. Create a
microservice
application

29. Paved Roads
29
Paved Roads
JIRA Project
Application Issue
Service Issue
Scrum Boards
Kanban Boards
Workflows etc. Mattermost
Channel for Bitbucket
Bitbucket Project
Spinnaker Application
Mattermost
Channel for JIRA
Bitbucket Build Spec Repo
Bitbucket Deploy Spec Repo
Decorate code and push to Bitbucket
Bamboo Build Plan
Bamboo CDC Plan
Spinnaker Pipeline
Deploy
Skeletal
Service
Create
AWS Routes
Update Asset
Info in JIRA
Bitbucket Deploy Repo
Bitbucket Code Repo

30. Asset Tracking
All Assets tracked in
JIRA
Complete visibility into the
infrastructure

31. Monitoring and Reporting
All Assets tracked in
JIRA
Complete visibility into the
infrastructure
Monitoring Alerting and
Reporting
Out of the Box
Build, Bake, Deploy
Standard pipelines as Code

32. SAFe and Scrum Boards
All Assets tracked in
JIRA
Complete visibility into the
infrastructure
Monitoring Alerting and
Reporting
Out of the Box
Build, Bake, Deploy
Standard pipelines as Code
SAFe, Scrum and Kanban
boards

33. Continuous Quality and Security
All Assets tracked in
JIRA
Complete visibility into the
infrastructure
Monitoring Alerting and
Reporting
Out of the Box
Build, Bake, Deploy
Standard pipelines as Code
SAFe, Scrum and Kanban
boards
Test tracking ad
Automation, Technical Debt
and Vulnerability analysis

34. Logging Monitoring and Distributed Tracing
All Assets tracked in
JIRA
Complete visibility into the
infrastructure
Monitoring Alerting and
Reporting
Out of the Box
Build, Bake, Deploy
Standard pipelines as Code
SAFe, Scrum and Kanban
boards
Test tracking ad
Automation, Technical Debt
and Vulnerability analysis
Logging, Distributed
Tracing, Mesh Observability

35. Pluto

36. Pluto features
• Cross Platform Microservice
• Strictly Integrated in Defacto
• Able to operate on generic catalogs
in favor of vendor neutrality
• Minimize infrastructure costs
• Respect Kubernetes constraints
Minimize infrastructure costs for a Kubernetes Cluster

37. Pluto Anatomy
AWS catalog
…..
Azure catalog
PLUTO
OptimizationModelbasedonGoogleOR-tools
POD 1 POD N
JIRA Service
Assets catalog
Holistic Infrastructure
JIRA
DeFacto Addon
Set of VMs
• Minimizing the
infrastructure costs
• Node Labels
recommendation for
each kubernetes node
POD 2

38. Next Steps

39. What is next?
Distribute Nucleus as a web
application and integrate it with Rancher
Add Serverless as first citizen in DeFacto
Initially experimented with Funktion. RedHat not supporting
the project anymore lead us to adopt either kubeless or
OpenWhisk
Finalize Hashi Vault integration for secret management
Finalize Nomad integration for cross cluster orchestration

40. What is next?
Create a DeFacto Kubernetes Operator
Offer DeFacto components
as first citizens in Kubernetes
Create DeFacto sidecar with automatic injection
Safeguard for unmanaged changes to the infrastructure
Ability to identify “unmanaged services” and track them within
DeFacto
Finalize Entangle (more info coming soon…..)
Who is using my library?
Which clients are failing if they switch to the new version?

41. Questions

42. Elastic Build

43. Envoy Proxy SDS

44. SDS with Let’s Encrypt

45. Bamboo Branch Updater

46. Monitoring and Reporting
All Assets tracked in
JIRA
Complete visibility into the
infrastructure
Monitoring Alerting and
Reporting
Out of the Box
Build, Bake, Deploy
Standard pipelines as Code

47. Expand Migrate Contract
Client A
Client Z
Old Clients
Client 1
Client N
New Clients
Old
version
New
Version
Expanded
Interface
Client A
Client Z
Old Clients
Client 1
Client N
New Clients
Old
version
New
Version
Expanded
Interface
Client A

48. Expand Migrate Contract
Client A
Client Z
Old Clients
Client 1
Client N
New Clients
Old
version
New
Version
Expanded
Interface Client A
Client Z
Client 1
Client N
New Clients
New
Version
Expanded
Interface Client A
Client Z

49. dynamism
deployment
runtime change
each request
Release
Toggles
Experiment
Toggles
Ops
Toggles
Permission
Toggles

50. System
Events
Issue
Events
Workflow
Events
User
Events
Configure
Destinatio
n Channels
/command Issue-X
Act in JIRA
with chat
commands
Herzum-MattermostJAVAlibrary
Message Buffer
Template
Service
Notification Service
• Rate limiting support
• Recover from Mattermost unavailability /
crashes
• Guarantee ordered delivery of messages
Event Listener
&
Postfunctions
Admin Screen
• Configure Notification
Policies
• Customize Message
Temprate
Push Ready to go Messages
Notification Job
Pop Ready to go Messages
use
Issue Tab Panel
• Display Channels Information in
the view issue screen
• Create new or link existing
channels
Project Tab Panel
• Display Channels
Information for the
project
• Create new or link
existing channels
use
MattermostRESTAPI
Reacts to
use
use
use
Mattermost Amplifier
for JIRA
use
use
use
Customize
Messages
Mattermost Amplifier for JIRA - Architecture