Tradeshift delivers electronic invoicing and social networking to the browser of businesses around the world. To do so, we employ a highly dynamic cloud-based infrastructure that scales in near real-time. In this talk we present Tradeshift's e-invoicing product, how we manage the dynamics of the infrastructure and how we ensure a continuous product development- and delivery process. Anders Nickelsen has a background in networks and distributed system and has a PhD in computer networks from Aalborg University. At Tradeshift, he works as a quality assurance engineer with special focus on continuity of infrastructure management and quality of software releases. Mikkel Hippe Brun is co-founder of Tradeshift and holds a Masters degree in Computer Science from DIKU. Mikkel has worked for many years with e-invoicing and large scale infrastructures for exchange of business documents. Mikkel is chair of the OASIS Business Document Exchange Technical Committee.

1. Head in the cloud and feet
on the ground
How cloud computing enables Tradeshift to deliver
continuous and global e-invoicing and more
Mikkel Hippe Brun Anders Nickelsen
mhb@tradeshift.com PhD, Quality Assurance Engineer
Twitter: @hippebrun ani@tradeshift.com
Cell: +45 3118 9102 Twitter: @anickelsen
Cell: +45 3177 6511
April 2011

3. Our industry
 Characteristic
 Practices have not changed much for 20 years
 Expensive infrastructure
 Traditional business models
 Costly for customers
 On-boarding of a customer is a project

4. The problem with E-invoicing
Mo6vated
to
move
to
e-­‐invoicing
Advanced
business
so.ware
(ERP)
Office
packages
Accoun6ng
Paper
+
email
/
PDF
systems

5. Connecting the long tail
Un6l
now:
No
good
solu6ons
for
this
segment
Advanced
business
so.ware
(ERP)
Office
packages
Accoun6ng
Paper
+
email
/
PDF
systems

6. Cloud computing allows us to
 Establish an extremely scalable
infrastructure
 Scales with our demand (up and down)
 Offer our services for a fraction of the cost
of our competitors
 No CapEX
 OpEx scales with customers (and gets cheaper with volume)
 Think differently
 Is there a better business model?
 How do we provide real value?
 Network
 Real time
 Sharing

7. Our advantage
 Blank slate
 No existing customers
 No legacy systems that must be maintained
 = No migration
 Agile
 Pirate culture
 Flat organization
 Product development owned by teams
 Emergence of Cloud technologies
 We could start cheap!

8. Easytrade (“Nemhandel”) Architecture
Dispatch
UBL/
3
XML
RASP:
HTTP,
SOAP,
4
Receive
WS-­‐Security,
WS-­‐
Reliable
Messaging,
SMTP,
PKI
Danish
1
Register
2
Lookup
Easytrade
UDDI

9. PEPPOL Architecture

10. What We Learned from Building These
Infrastructures..
 Cloud + Open Standards + Open Source
Software
 Can be used to realize extremely low-cost, secure,
reliable high-volume B2B infrastructure
 Lowering cost significantly & opening the
infrastructure
 Can bring small business into E-invoicing relationships
 Capturing the long tail of small
businesses
 Can be achieved by including them into a network and
making them discoverable

11. Operations
 Cost per Unit
Unit
 Today: ~1.2$ / Unit
 Sept. 11: ~ 0.12$ / Unit
 Sept. 12: ~0.04$ / Unit

12. The Cloud in
Tradeshift

13. The cloud in Tradeshift
 Software-as-a-Service
 Self-organizing infrastructure
 Platform-as-a-Service
 Infrastructure-as-a-Service
 Continuous deployment into self-
organizing infrastructure

14. Software-as-a-Service
 go.tradeshift.com
 Companies have isolated storage on a shared cloud-based
platform
 No installations at customer sites
 Public APIs
 Integration points to 3rd party software
and enterprise customers

15. Platform-as-a-Service
 Servers provided by Amazon EC2 + Rackspace
 Storage provided by Amazon S3 + Rackspace
 We are (almost) independent of platform provider
 Regular Ubuntu 10.04 installations (LTS)
 Virtual private servers with full access
 New instances can be spun up in a matter of minutes
 Different instance types:
 Memory, processing, HPC (CPU+network)
 Enable modular infrastructure (every service on small dedicated
instances)

16. Infrastructure-as-a-Service
 3-tier software architecture
 Frontend, Backend, Database
 State-less tiers
(support: session sharing service)
 Security groups for firewalling
 Tier-based load balancing
 Modular infrastructure allows for
efficient scaling and load balacing

17. Infrastructure-as-a-Service
 Infrastructure as code
 Programmable
 Testable
 Deployable
 Reliability issue – full recovery from:
 Repository (code+infrastructure description)
 Application state backup (database)
 ’Bare metal resources’
 Automated deployment and test
of servers for infrastructure
 Infrastructure connections programmed
into deployment process
 Entities register automatically with our
monitoring tool for complete overview

18. Infrastructure-as-a-Service
 Deployable infrastructure + cloud = rapid replication
 Infrastructure replicated into different environments
 Environments may be used short or long term
 Manual triggering of automated environment deployment
Produc6on
Sandbox
Staging
1
Staging
2
Staging
n

19. Infrastructure deployment tools
 ControlTier
 Server instrumentation

20. Infrastructure deployment tools
 Puppet
 Configuration management
 Rapid and reusable

21. Infrastructure deployment tools
 Zabbix
 Infrastructure monitoring

22. Self-organizing infrastructure
 Can be seen as a traditional control problem
 Zabbix monitors load on each tier (sensor)
 ControlTier automatically spawns new servers
when needed (actuator)
 Load-balancing / fail-over
 Elastic load balancer (Amazon service) distributes load to pool of
servers (Frontend)
 Own customized application-based request distribution schema
(backend, database, proxies)
 Automatically removes servers when not needed

23. Continuous deployment (integration/delivery)
 Integration and release are frightening
tasks
 Huge, complex, long lasting tasks that are difficult to
estimate in time and risk
 Our continuous processes
 Automated build on each commit (fast) (Hudson)
 Automated test of each build (fast)
 Tested in freshly spawned staging environments
 unit-tests, Selenium
 Automated test of deployment process after each build
 Automated delivery of each successful build
 Release early, release often

24. Continuous delivery into a self-organizing infrastructure
 Classes of changes
 Purely code (bugfixes)
 Direct into production
 Database schema changes (features)
 Into new production environment  redirection
 Environment changes (major features)
 Environment specification changed
 Into new production environment  redirection
 Monitoring of product KPIs
 Ex. If signup rate or invoice rate decrease, something is wrong with the
latest deploy
 Roll-back mechanisms for all scenarios
 Old references and environments are preserved
 Feature bits – everything is in the code, either on or off
 Segmented roll-out, A/B testing

25. References
 Tradeshift
 http://developer.tradeshift.com
 http://tradeshift.com
 Self-organizing infrastructure
 Amazon EC2: http://aws.amazon.com/ec2/
 Puppet: http://www.puppetlabs.com/
 ControlTier: http://controltier.org/
 Zabbix: http://www.zabbix.com/
 Continuous deployment
 Jenkins CI (Hudson CI): http://jenkins-ci.org/
 Selenium: http://seleniumhq.org/
 SauceLabs: http://saucelabs.com/

26. Thank you