Two parts:
1. The evolution of Joyent's SmartDataCenter cloud infrastructure management software from a largely monolithic app to a microservices architecture.
2. How container infrastructure enables microservices.
More details in http://www.meetup.com/cloudclub/events/220026896/
3. Our data center or yours
Joyent Public Cloud
Joyent provides Container
Infrastructure as a Service.
We run our customer’s mission critical
applications across thousands of
containers in the Joyent Public Cloud.
Joyent SmartDataCenter
Joyent SmartDataCenter is an open source
on-premise container run-time
environment used by some of the world’s
most recognizable companies to run their
mission critical applications.
4. Node.js enterprise support
As the corporate steward of Node.js and
one of the largest-scale production
users, Joyent is uniquely equipped to
deliver the highest level of enterprise
support for this dynamic runtime.
Best
Practices
Performance
Analysis
Core File
Analysis
Debugging
Support
Critical
Incident
Support
⚠
6. Merging trends
12 factor app
Two pizza rule
Begrudging acceptance
of some SOA patterns
Lightweight, OS-based
virtualization
Infrastructure as code
Infrastructure Applications
10. Our data center or yours
Joyent Public Cloud
Joyent provides Container
Infrastructure as a Service.
We run our customer’s mission critical
applications across thousands of
containers in the Joyent Public Cloud.
Joyent SmartDataCenter
Joyent SmartDataCenter is an open source
on-premise container run-time
environment used by some of the world’s
most recognizable companies to run their
mission critical applications.
13. SmartDataCenter 6.5
• Two monolithic Ruby pieces:
• Machine API
• Customer API
• Some edge pieces in Node.js
14. SmartDataCenter 7
Booter
AMQP
broker
Public
API
Customer
portal
ZFS-based multi-tenant filesystem
VirtualNIC
VirtualNIC
Virtual
SmartOS
(OS virt.)
...
VirtualNIC
VirtualNIC
Linux
Guest
(HW virt.)
...
VirtualNIC
VirtualNIC
Windows
Guest
(HW virt.)
...
VirtualNIC
VirtualNIC
Virtual OS
or Machine
...
SmartOS kernel
(network booted)
SmartOS kernel
(flash booted)
Provisioner
Instrumenter
Heartbeater
DHCP/TFTP
AMQP
AMQP agents
Public HTTP
Head-node
Compute node
Tens/hundreds per
head-node
. . .
SDC 7 core services
BinderDNS
Operator
portal
. . .
Firewall
15. SmartDataCenter 7 core services
Analytics
aggregator
Key/Value
Service
(Moray)
Firewall
API
(FWAPI)
Virtual
Machine
API
(VMAPI)
Directory
Service
(UFDS)
Designation
API
(DAPI)
Workflow
API
Network
API
(NAPI)
Compute-
Node API
(CNAPI)
Image
API
Alerts &
Monitoring
(Amon)
Packaging
API
(PAPI)
Service
API
(SAPI)
DHCP/
TFTP
AMQP
DNS
Booter
AMQP
broker
Binder
Public
API
Customer
portal
Public HTTP
Operator
portal
Operator
Services Manta
Other DCs
Note: Service
interdependencies not
shown for readability
Head-node
Other core services
may be provisioned on
compute nodes
SDC7 Core Services
17. By the numbers
The SmartDataCenter application uses 25 containers for its
services. That’s without HA enabled!
Memory Count
128MB 2
256MB 2
512MB 1
768MB 2
1024MB 10
2048MB 3
4096MB 2
8192MB 3
19. Discovery
Moving from monolithic to microservices architecture means
moving from tightly coupled to loosely federated system
necessitates service discovery
We built Binder, a node.js-based DNS + ZooKepper system. It
works, but this is a complicated space.
Thorny issues without clear solutions:
• Upgrades
• Rollbacks
• Maintenance windows
20. State
Stateless microservices are easy, but stateful services require
more thought. Decentralization of services is good, but
decentralized state is insane.
Answer: statefulness is itself a service made up of a set of
microservices:
• Moray: key value store
• JSON over TCP messaging using node-fast
• Manatee: automated fault monitoring for PostgreSQL
• Automatically spins up new Postgres instances as needed
to maintain DB quorum
• PostgreSQL: reliable, performant
21. Messaging
• The principles of AMQP are very attractive…
• …but in practice, implementation and operational issues have
made message brokers a single point of failure
• Still using AMQP for some broadcast messages
• Now mostly using point-to-point HTTP for performance and
reliability
22. Interfacing
• More interfaces = more problems interfacing
• JSON’s loose structure multiplies the risk of interfacing
problems
• We use JSON Schema to add rigor without sacrificing
agility
• Interface versioning is required for sanity
• We use Restify for HTTP routing with built-in versioning
23. Debugging
Systems break, distributed systems have more breakable
surface area. Our diagnostic tools are as critical as any other in
the box:
• DTrace profiling of node.js
• Bunyan node.js logging facility with DTrace support
• JS heap analysis from core files
• “core files are the ultimate REPL”
Also see @dapsays’ Node 0.12 debugging improvements and
Node Summit slides.
24.
25. SmartDataCenter is open
On Github: https://github.com/joyent/sdc
Test it in VMware on your laptop or install on dedicated
hardware. Try the Docker support preview!
Connect:
• Mail list: sdc-discuss@lists.smartdatacenter.org
• IRC: #smartos on Freenode
26. SDC is easy to install
Deploy SmartDataCenter in hours, not days (or minutes, not
months) and get to work building value faster
41. CPU scheduling
1. Under hardware virtualization, the guest OS is a
hard realtime app
2. Hard realtime apps freak out when they don’t get
their expected CPU cycles
3. Oversubscribed CPU + hardware virtualization is a
recipe for disaster
42. I/O latency
Virtualized I/O and conflicts among competing schedulers
increase I/O latency, sapping performance
45. The Joyent difference
• Bare metal container performance
Eliminate the hardware hypervisor tax
• Proven container security
Run containers securely on bare metal in multi-tenant environments
• Simplified container networking
Each container has its own IP(s) in a user-defined network (SDN)
• Simplified host management
One API endpoint (including Docker API) fronts the whole data center
46. Security: solved
General Challenge
Buggy or malicious code can escape Linux containers, execute
unexpected processes, and read or write files in harmful ways.
Joyent Solution
Joyent’s container runtime was built for security isolation first, and we have
nearly ten years of experience without incident. We are now extending this
container runtime to work with native Linux binaries and Docker.
Joyent Value
Multi-tenant deployments enable cloud-scale economics and density.
47. Networking: solved
General Challenge:
Connecting containers between hosts, or with sophisticated network
topologies, is difficult and performs poorly.
Joyent Solution:
Joyent’s software defined networking solution brings full network
connectivity to each container (including a unique IP and wire-speed access)
and VxLANs across containers.
Joyent Value:
Lower costs of administration with simplified network management and
by leveraging leveraging commodity hardware for networking.
48. Containers…
you mean Docker?
Preview access to our next generation Docker
container service is launching “soon.”
Joyent’s infrastructure containers form the basis
of our Docker service.
49. Are containers
limited to Microservices?
Containerization does not dictate application architecture.
Joyent’s container technology scales from single-process
Docker containers to infrastructure containers supporting the
full suite of services expected from a unix host.
Infrastructure containers can be used as a persistent,
durable, and faster alternative to VMs.
…And, we can run KVM inside infrastructure containers to
support full hardware virtualized environments necessary to
host Windows and other OSs.
52. Remember Joyent for
• Container tech that scales from fat apps that expect a full
host down to tiny single-process microservices
Better efficiency performance advantages for all application
• Bare metal performance
Eliminate the hardware hypervisor tax
• Proven container security
Run containers securely on bare metal in multi-tenant environments
• Simplified container networking
Each container has its own IP(s) in a user-defined network (SDN)
• Simplified host management
One API endpoint (including Docker API) fronts the whole data center