One of the things standing in the way of effective and scalable engineering is shared infrastructure. Ed Hargin, Lead of DevOps and Kyrylo Novotarskyi, Software Engineer for Currencies and Banking, focus on the journey TransferWise took from shared to independent infrastructure management, including moving from legacy manual towards automatic instance provisioning, the evolution of service registration and discovery adoption, and the introduction of centralized distributed configuration storage, including secret management techniques, testing, troubleshooting and disaster recovery scenarios.

1. SERVICE DISCOVERY AND
CONFIGURATION MANAGEMENT AT
TRANSFERWISE

2. FIRST INTRODUCTIONS

3. INTROS

6. FAMOUS FOR

7. ESTONIAN FOUNDERS

8. EVANGELICAL CUSTOMERS

9. $1 BILLION IN TRANSFERS
PER MONTH

10. SAVING OUR CUSTOMERS $50M
PER MONTH

11. ENGINEERING CULTURE

12. WEAK CODE OWNERSHIP

13. EVERY PART OF THE CODE IS
OWNED

14. ANY TEAM CAN CHANGE ANY PART
OF THE CODE

15. ENGINEERING FOR CUSTOMER
IMPACT

16. ENGINEERS GET CLOSE TO
CUSTOMERS AND MAKE DECISIONS
BASED ON DATA

17. SOME OF OUR TEAMS SERVICES
NEED TO BE HIGHLY RESPONSIVE,
SYNCHRONOUS AND SCALE REALLY
WELL

18. OUR PLATFORM TEAM IS 8 PEOPLE
STRONG
WITH A HUGE BACKLOG

19. HOW DID WE HANDLE IT BEFORE?

20. AN ENGINEER ASKS FOR A
PRODUCTION SERVICE

21. AND WAITS…

22. WAITS SOME MORE…

23. ASKS AGAIN…

24. WAITS SOME MORE…

25. GETS A SERVICE

26. NEEDS TO CHANGE PRODUCTION
CONFIG

27. WAITS…

28. THIS NEEDS TO STOP

29. TOO MUCH CHATTER
TOO LONG WAITS
STRESS FOR EVERYONE

30. CUSTOMERS SUFFER

31. WE NEED:
AUTOMATIC ALLOCATION
PROVISIONING
LOGGING AND MONITORING
WITHOUT ENGINEERS

32. SERVICE DISCOVERY CAN HELP

33. RESEARCH AND TEX

36. WHICH DISCOVERY APPROACH TO
CHOOSE?

37. CLIENT SIDE DISCOVERY

38. SELF REGISTRATION

40. NETFLIX OSS IS OUR WAY TO GO

41. PROOF OF CONCEPT

43. MAKE OUR MONOLITH CONSUME 1
SERVICE IN PRODUCTION

45. UNDERSTANDING THE SCOPE

46. Problem: making friends between spring cloud and anything
else except spring boot can be tricky. There is no good
adoption mechanism

47. Solution: grails and spring are close. Let’s read the source,
find out what hides beyond spring boot’s netflix-specific
annotations, mimic the approach it was designed for

48. Lets create a shared bean in the discovery space (used by
our monolith parts that consumes the config and starts
communication with Eureka)

49. Step 2: Let’s start a Ribbon LoadBalancer in that bean,
connect it with Eureka client and let him start listening for
apps

50. Step 3: Most of our communication is through
restTemplate. How can we make restTemplate awesome
and @LoadBalanced? Interceptors! Let’s build one

51. Step 4: Let’s add the interceptor to the
needed RestTemplate
Step 5: What if discovery
completely fails? Fallbacks!

52. IS THIS A PERFECT SOLUTION?
OF COURSE NOT!

53. NEED TO MAINTAIN
TIED TO RESTTEMPLATES
NEEDS ENGINEERS TO THINK ABOUT
IT

54. NOT GOOD ENOUGH

55. BUILD CLIENTS AND MAKE EVERY
CASE WORK AS A BLACK BOX

56. IF YOUR SERVICE WANTS TO BE
DISCOVERED - MAKE YOUR CLIENT
PROVIDE THE TOOLSET

57. PROBLEMS

58. SOME SERVICES HAVE A EUREKA CLIENT
OF THEIR OWN
EUREKA IS NOT USED TO IT
(CONFIGURATIONS ARE STORED IN A CONTEXT BEAN)
RESULTING IN OVERRIDES UNLESS
CONTROLLED

59. TESTING IS PROBLEMATIC
AS REQUIRES A FULL COPY OF PRODUCTION
TESTING THE PIECES IS COMPLETELY ON
THE ADOPTING ENGINEER

60. CHALLENGES

61. VARIATION IN HTTP COMMS AND
IMPLEMENTATIONS

62. FEIGN AS A CLIENT BUILDER
DIRECT FALLBACKS
HYSTRIX CIRCUIT BREAKING
SPLITTING BALANCING AND DISCOVERY
BETWEEN SERVICES

63. SOLUTION COMPATIBILITY
SPRING BOOT
EUREKA
RIBBON
SUPPORTING TOOLSET
AROUND 75% ADOPTION PROBLEMS

64. 3 MONTHS
30 SERVICES IN PRODUCTION
20 MORE IN THE MAKING

65. E

66. GREAT
NOW WE HAVE LOTS OF SERVICE
CONFIG TO TRACK

67. LET’S MAKE THINGS AS BORING AS
POSSIBLE

68. LET’S PUT EVERYTHING IN
(PUPPET | CHEF | SALT | ANSIBLE)

69. JOB DONE

70. EXCEPT…

71. SECRET MANAGEMENT IS TRICKY

72. WE CAN MAKE IT WORK. BUT IS
THERE A LESS TEDIOUS WAY?

73. SPRING CLOUD CONFIG SERVER
NETFLIX ARCHAIUS
HASHICORP VAULT

74. NICE DOCUMENTATION:
HTTPS://CLOUD.SPRING.IO/SPRING-CLOUD-
CONFIG/SPRING-CLOUD-CONFIG.HTML

75. NAME/VALUE PAIRS
/ENCRYPT & /DECRYPT ENDPOINTS
EASY TO EMBED IN SPRING BOOT
APPLICATIONS

76. SIMPLE REST API, SO PLAYS WELL
WITH NON-SPRING TOO!

77. SUPPORTS TEMPLATE FILES… BUT
WE’VE NOT USED THEM

79. SECRETS DON’T NEED TO BE STORED
IN PLAIN TEXT:
/ENCRYPT
/DECRYPT

80. VAULT BACKEND:
REQUIRES A TOKEN FROM CLIENT
FAILFAST VS RETRY

81. GETTING IT INTO PRODUCTION:
“VOLUNTEERED” A FEW SERVICE
OWNERS

82. EVERYTHING WORKED SMOOTHLY. VERY
SUSPICIOUS.

83. BUGS FOUND!

84. GITHUB DOWN: CONFIG SERVER
GOES MENTAL
CONFIG SERVER CACHES LOCALLY,
BUT ALSO WILL ALWAYS CHECK FOR
NEW CONFIG IN THE REPO

85. VERY INFREQUENTLY… IT JUST DIES
AND WE DON’T KNOW WHY (YET)
(NEVER HAD >1 NODE DIE AT ONCE, SO
NOT AWFUL IMPACT, BUT CONCERNING)

86. RESULT: SERVICES ARE GETTING INTO
PRODUCTION FASTER

87. Let us change money transfer forever.
Together.
https://transferwise.com/jobs/