- Concur is a travel and expense management company with 6500+ employees and offices worldwide. They process over 70 million transactions and $50 billion in travel and expense spend annually.
- The presenter is a Principal Architect at Concur who has been working with Kubernetes since 2015. He discusses why Concur chose Kubernetes and CoreOS for container orchestration.
- Concur runs multiple Kubernetes clusters across different regions for high availability. A custom tool called kube2cnqr manages load balancing between clusters.
3. What is Concur?
6500+ employees
Offices:
Australia
China
Czech
Republic
France
Germany
Holland
India
Japan
Philippines
Singapore
Sweden
UK
US
HQ: Bellevue
70m transactions
50b+ USD
T&E spend
annually
€59m
New cloud
bookings
Q3 2015
Cloud DNA
99
Established
1993
> 60%
Fortune 500
4. 2011 2012 2013 2014 2015 2016 20182017
Expense transactions
Bookings growth
2013 2015
55
million
77+
million
Provisioned users
35+
million
2015
Registered mobile users
2013 2015
2.1
million
5+
million
Concur growth
5. A little about me
Principal Architect
• Concur since 1998 in various roles (DBA, Ops Manager
& Architect, etc.)
• Corking w/ k8s since mid 2015
• contributor to k8s and participant in federation & service
catalog SIGs
• Email danwilson on github, k8s slack & gmail
6. Why Kubernetes?
• Largest community support
• Gifted to Linux Foundation
• Protection from shifts in container technology
• Pluggable API
• Capable of isolation between namespaces
• Declarative syntax for the complete make up of services
• Built in logging, service discovery, etc.
• Networking design eliminates port conflicts of containers
7. Why CoreOS?
• Simple updates by channel
• Designed to run as a cluster
• Low overhead
• Best documentation for k8s
8. Why CoreOS?
• Simple updates by channel
• Designed to run as a cluster
• Low overhead
• Best documentation for k8s
10. High Availability
• All things fail, yes everything!
• K8s HA guide: http://kubernetes.io/docs/admin/high-availability/
• cluster etcd – 3, 5 or 7 nodes
• replicated + load balanced api servers
• many minions
15. K8s @ Concur
• kube2cnqr
• Golang docker container
• Watches the k8s API for updates to services and worker nodes
• Calls a Concur internal API to addremove load balancer entries as
needed
• Handles connecting multiple k8s clusters to a single load balancer
endpoint
• Extras
• External-IP set on service objects
• Annotation w/ DNS name set on service objects
• Handles multiple load balancer tiers
• Allow services to specify DNS name
16. Sample project
• https://github.com/concur/kubegowatcher
• A sample golang program that leverages kubernetes watch endpoints
using the client-go library.
• Designed to run as a container on k8s
• example of setting annotation on the service
• example of checking event delay
• Apache 2.0 license
• Add your own business logic
• Contribute back improvements to make it better
17. Location US EUROPE
Zone A B C A B C
Cluster v.1.2 us12a us12b us12c eur12a eur12b eur12c
K8s @ Concur – k8s upgrades
18. Location US EUROPE
Zone A B C A B C
Cluster v.1.2 us12a us12b us12c eur12a eur12b eur12c
Cluster v.1.3 us13a us13b us13c eur13a eur13b eur13c
K8s @ Concur – k8s upgrades
19. K8s @ Concur
Deployment Challenges
• Stale kubectl scripts
• Stale kubectl config
• Handling credentials for each ci system
• Many commands for each cluster
• Handling proxy
26. K8s @ Concur
Add features to kubernetes federation
• v1.6 – deploying to clusters based on
label #29887
Other potentials…
• Location specific overrides
• Global overrides
• Feature limitations (for example,
don’t allow pods/rc’s without a
deployment)
Docs: https://github.com/concur/skipper
28. Q & A
Email danwilson on github, k8s slack & gmail
Editor's Notes
Tripit
Hipmunk
Tripit
Hipmunk
Explain hypercar components and how they relate to building a kubernetes cluster.
CoreOS makes it easy by already having the pieces tested together (kernel, flannel, etcd, systemd, etc)
Plan for unknown unknowns
Locking thresholds or bad query plans are known unknowns