Vincent Garrigues talks about the transformation in the SoundCloud regarding the way they work. As SoundCloud grew, its backend infrastructure got more complex with the addition of many microservices. Ensuring their apps to work with these microservices in production is key to SoundCloud's continuing success.

2. Continuous Integration at
SoundCloud
Istanbul Tech Talks 2015
Vincent Garrigues
@garriguv

3. Why invest in
a mobile CI
pipeline?

4. • move fast
• keep codebase healthy
• confidence
• ship reliable apps

5. • move fast
• keep codebase healthy
• confidence
• ship reliable apps

8. • started from scratch
• months in development
• millions of users

9. iOS Crash Complaints (avg per Week)
0
35
70
105
140
April May June July August
2014 SoundCloud community team

10. iOS Crash Complaints (avg per Week)
0
35
70
105
140
April 2014 July 2014 October 2014 January 2015
2015 SoundCloud community team

12. How do we
work?

13. • work on master
• little to no branches
• pairing
• feature flags

14. - (id<AnalyticsProviderInterface>)soundcloudInternalProvider
{
if ([FlipTheSwitch isDevEventgatewayEnabled]) {
return [self eventGatewayProvider];
} else {
return [self eventLoggerProvider];
}
}
Feature flagging
github.com/michaelengland/fliptheswitch

15. Feature flagging

16. Timeline

17. 2012 2013 2014 2015
📱
📱
📱 📱
📱
📱
📱
📱
📱
📱
📱
📱
📱
📱
📱
📱📱
📱 📱

18. started very
simple

20. ⛔

22. • unit tests
• α and β builds
• app store build

23. Overview of
the current
system

24. build
analysis
unit tests
acceptance
tests
AppStore
AdHoc
α and β
🚦 🚦

25. build
analysis
unit tests
acceptance
tests
AppStore
AdHoc
α and β
~ 5min 30sec ~ 7min ~ 3min
~ 15min
🚦 🚦

26. Let's have a
look at the
details

27. build
analysis
unit tests
acceptance
tests
AppStore
AdHoc
α and β
🚦 🚦

28. build
unit
tests
acceptance
tests build
linter
dependencies
local libraries
unit tests
i18n
push

29. build
local libraries
unit tests
i18n
push
unit
tests
acceptance
tests build
linter
dependencies

30. build
unit
tests
acceptance
tests build
linter
dependencies
~2000
local libraries
unit tests
i18n
push

31. build
unit
tests
acceptance
tests build
linter
dependencies
~2000
~5000
local libraries
unit tests
i18n
push

32. build
unit
tests
acceptance
tests build
linter
dependencies
~2000
~5000
local libraries
unit tests
i18n
push

33. build
unit
tests
acceptance
tests build
linter
dependencies
~5000
~2000
local libraries
unit tests
i18n
push

34. build
unit
tests
acceptance
tests build
linter
dependencies
~5000
~2000
local libraries
unit tests
i18n
push

35. build
analysis
unit tests
acceptance
tests
AppStore
AdHoc
α and β
🚦 🚦

36. • frank
• cucumber
http://www.testingwithfrank.com
https://cukes.info

38. • flexible platform & OS
• bandwidth and reachability
settings
• launch arguments

39. stubby4j
✅
⛔
datasets

40. mobile apiproxy
The proxy can change:
• response status code
• response body
• record and undo actions
(like, repost…)

41. ~360
acceptance
tests

42. ~360
acceptance
tests
~250 min

43. Sharding

44. distributed
cucumber

45. distributed cucumber

46. x20
VM

47. build
analysis
unit tests
acceptance
tests
AppStore
AdHoc
α and β
🚦 🚦
~ 5min 30sec ~ 7min ~ 3min
~ 15min

48. acceptance
tests
• iOS version
• iPhone (4S, 5, 6, 6+)
• iPad (retina and non retina)
• feature flag configurations

49. What's the value
of an acceptance
test?

50. avoid
regressions on
hard to test
flows/screens

53. keeping the
build green

56. pre-ci

57. https://www.youtube.com/watch?v=BhMSzC1crr0 SpaceX

58. Flaky tests

59. Flaky tests
• test driver is flaky
• backend is flaky
• app is unpredictable

60. Flaky tests
• identify
• isolate
• fix

61. flakyrazor

62. flakyrazor
1. Take failing test out of the
test pool
2. Run the test multiple times
(flaky or failing?)
3. Assign it to the author/
committer

63. flakyrazor
4. Assess test value
5. Act on test duration changes
6. Show statistics on why the
test failed

64. Taking care of
our build
machines

65. ❤
❤
❤
❤
❤
❤
❤ ❤

67. • we have images for all the
machines
• CLIs to provision the
machines remotely
• OS X server stores the
images and controls the
imaging process

68. What about
Android?

69. Thank you!
Istanbul Tech Talks 2015
Vincent Garrigues
@garriguv