隨著公司的業務發展，我們開始注意到要快速的部署一個新的環境、市場越來越沒有效率，而 CM（Configuration Management）在我們解決這些問題中扮演著重要角色。本演講將專注於解決下列問題： 1. 為什麼我們需要 CM？ 2. 如何管理不同種類的 Configuration: Realtime, Service Discovery and Provision? 3. 如何降低團隊的導入 CM 複雜度？

1. 從零開始的 Configuration Management
Levi Chen @ DevOpsDays Taipei 2019

2. About Me
陳俊毅 (Levi Chen)
Engineer in Common Service & Infra Team @ 91APP
Contact me
● FB: https://fb.me/chenjiunyi
● Blog: https://blog.levichen.tw
2

3. Responsibilities @ 91APP
● Develop 3 common services
○ Product, Notification and Rate Limiter
● Help RDs to use cloud technologies
○ Cloud Native, Serverless
● Help architecture team to implement new policies
About Me
3

4. 4

5. 5
https://paulhammant.com/2014/08/27/provisioning-deployment-and-app-config-cycles/

6. Agenda
● Why configuration management?
● How to manage configuration?
● How to go live?
6

7. Agenda
● Why configuration management?
● How to manage configuration?
● How to go live?
7

8. 台灣最大 & 成長最快
品牌新零售解決方案公司
- 2013 年成立
- 總部在台北，馬來西亞/香港分公司
- 員工人數 400 人
8

9. 9

10. 品牌客戶超過 10,000 家
10

11. Challenge #1
Deploying to Mutiple Environments
11

12. TW
HK
MY
Expand to New Markets
2013 TW
2018 MY
2019 HK
12

13. Load Test
https://www.monitis.com/blog/load-test-and-be-sure-your-website-is-ready/13

14. RD + PD Team: 150 people, 9 function teams
Multiple Test Environments
14

15. Team 1
15
Team 2
Team 3
Team 4
Team 5
SIT Staging Production

16. Team 1
16
Team 2
Team 3
Team 4
Team 5
SIT Staging Production
Env 1
Env 2
Env 3
Env 4
Env 5

17. The Complexity of Deployment
=
Number of Environments
x
Number of Services
x
Number of Instances
17

18. Environment 1
Service CD Server
1. Build
2. Deploy
18

19. Environment 1
Service CD Server
1. Build
2. Deploy
Environment 2
Service CD Server
1. Build
2. Deploy
19

20. Challenge #2
Multiple Tenancies
20

21. Different configurations between tenancies
https://sodelicious.recipes/in-the-kitchen/christmas-food-around-the-world/
21

22. The Complexity of Deployment
=
Number of Environments
x
Number of Services
x
Number of Instances
x
Number of Configurations22

23. If we have 10 environments, 40 services, 2 instances, 20
configuration…
10 environments x 40 services x 2 instances x 20 configuration
= 16,000 configurations
23

24. Challenge #3
Changing Configurations in Realtime
24

25. Change configurations != Change systems
http://ellenhorn.com/blog/the-ten-reasons-not-to-change/25

26. Missing configuration after deployment
https://www.express.co.uk/life-style/health/431422/Busy-Then-forget-it-Hectic-lifestyle-affects-our-memory26

27. Challenge #4
Too Many Domain Names
27

28. 28
Instance
#1
Instance
#2
Instance
#3
LBClient
Server-Side Service Discovery
https://www.nginx.com/blog/service-discovery-in-a-microservices-architecture/

29. Environment 1
A
B
C
D
E
a.environment1.com
b.environment1.com
c.environment1.com
d.environment1.com
e.environment1.com
Environment 2
A
B
C
D
E
a.environment2.com
b.environment2.com
c.environment2.com
d.environment2.com
e.environment2.com29

30. 40 services = 40 domains
30

31. If we have 10 environments, 40 services…
10 environments x 40 services = 400 domain names
31

32. ● Deploying to multiple environments
● Multiple tenancies
● Changing configuration in realtime
● Too many domain names
Recap: Why configuration management?
32

33. Agenda
● Why configuration management?
● How to manage configuration?
● How to go live?
33

34. Step #1
Make Service Artifacts Stateless
34

35. Environment 1
Service CD Server
1. Build
2. Deploy
Environment 2
Service CD Server
1. Build
2. Deploy
35

36. Responsibility of CD Server
● Build Artifacts for a specific environment
● Deployment
○ Permission
○ Network Topology
○ Deployment Strategy
36

37. number of artifacts
=
number of environments
=
number of CD server
37

38. What is your performance?
https://www.humanresourcesonline.net/how-to-effectively-set-and-communicate-performance-expectations/
38

39. How does open source deal with it?
39

40. We should ….
Single Codebase Multiple Deployment
Deployment = Artifact + Configurations
40

41. Environment 1
AM
CI Server
CMService
Environment 2
CI Server
CMService
41

42. Environment 1
AM
CI Server
CMService
Environment 2
CI Server
CM
1. Build Config 1. Build Code
Service
42

43. Environment 1
AM
CI Server
CMService
Environment 2
CI Server
2. Get Artifact
2. Get Artifact
CM
1. Build Config 1. Build Code
Service
43

44. Environment 1
AM
CI Server
CMService
Environment 2
CI Server
3. Get Config
2. Get Artifact
2. Get Artifact
CM
1. Build Config 1. Build Code
3. Get ConfigService
44

45. Everyone could use artifacts to build services everywhere,
by different configurations
https://www.sc.qa/en/news/2022-fifa-world-cup-stadium-workers-given-5m-repayment45

46. Step 2. Configuration Hierarchy
46

47. Global Region Market Shop
Default Settings
Schema
Comments
Regional Settings
ap-northeast-1
ap-southeast-1
tw
hk
my
Shop Id: 9527
Shop Id: 9528
/{module}/global.json
/{module}/{region}/global.json
/{module}/{region}/{market}.json
/{module}/{region}/{market}-{shop}.json
Configuration Hierarchy (Source)
47

48. 48

49. 49

50. 50

51. 51

52. global
region
market
shop
52

53. Global Region Market Shop
Default Settings
Schema
Comments
Regional Settings
ap-northeast-1
ap-southeast-1
tw
hk
my
Shop Id: 9527
Shop Id: 9528
Configuration Hierarchy (Build)
/{module}/index.json
/{module}/{market}.json
/{module}/{market}-{shop}.json
53

54. 54

55. $ docker run
-v ${PWD}:/build
$DOCKER_IMG
cfgctl build_local_file
--config ./config
--artifacts ./artifacts
--module MsaDemo
--datacenter dc-01
--tag 1.1.0
55

56. shop 8888 market
56

57. 57
$ docker run
-v ${PWD}:/build
$DOCKER_IMG
cfgctl build_local_file
--config ./config
--artifacts ./artifacts
--module MsaDemo
--datacenter dc-01
--tag 1.1.0
--map true

58. 58

59. Benefits of Configuration Hierarchy
● Tenancies level configuration
● No copy paste (remove redundant configuration)
59

60. Step #3
Configuration Build Tools: Cfgctl
60

61. ● Command line interface for building configuration hierarchy
● Test for Configuration
● JSON -> Key-Value pairs
● Build and deploy configuration to 3 types of storages
○ Local files
○ Consul
○ S3
Cfgctl
61

62. Play with cfgctl: For developer in local environment
Local
Files
Configuration
Local
Consul
Cfgctl build for
local files
Cfgctl build for consul
CodeSDK
62

63. Play with cfgctl: For OOO Environment
S3
Configuration
Consul
Cfgctl build for S3
Cfgctl build for
consul
ServiceASDK
ServiceBSDK
63

64. Step #4
Configuration SDK
64

65. Configuration SDK
● Languages Supported: C#, Node.js
● Features
○ Interface
○ Supported multiple configuration storages
○ Caching
○ Hot reload
65

66. Service SDK
Consul
S3
Local Files
66

67. Environment 1
AM
CI Server
CMService
Environment 2
CI Server
3. Get Config
2. Get Artifact
2. Get Artifact
CM
1. Build Config 1. Build Code
3. Get ConfigService
67

68. Environment 1
AM
CI Server
CMService
Environment 2
CI Server
3. Get Config
2. Get Artifact
2. Get Artifact
CM
1. Build Config
By Cfgctl
1. Build Code
3. Get ConfigService
S
D
K
S
D
K
68

69. Step #5
Service Discovery
69

70. 70
Instance
#1
Instance
#2
Instance
#3
LBClient
Server-Side Service Discovery
https://www.nginx.com/blog/service-discovery-in-a-microservices-architecture/

71. 71
Service A
Service
Registery
Service B
#1
Service B
#2
Service A
Service
Registery
Service B
#1
Service B
#2

72. 72
Service A
Service
Registery
Service B
#1
Service B
#2
Service A
Service
Registery
Service B
#1
Service B
#2
Hi, I am Service B (register)

73. 73
Service A
Service
Registery
Service B
#1
Service B
#2
Service A
Service
Registery
Service B
#1
Service B
#2
Hi, I am Service B (register)
Hi, I still alive (send heartbeat)

74. 74
Service A
Service
Registery
Service B
#1
Service B
#2
Service A
Service
Registery
Service B
#1
Service B
#2
Hi, I am Service B (register)
Hi, I still alive (send heartbeat)
I want to call Service B

75. 75
Service A
Service
Registery
Service B
#1
Service B
#2
Service A
Service
Registery
Service B
#1
Service B
#2
Hi, I am Service B (register)
Hi, I still alive (send heartbeat)
I want to call Service B
Service B has a Instance #1

76. 76
Service A
Service
Registery
Service B
#1
Service B
#2
Service A
Service
Registery
Service B
#1
Service B
#2
Hi, I am Service B (register)
Hi, I still alive (send heartbeat)
I want to call Service B
Service B has a Instance #1
Call Service B Instance #1

77. 77
Service A
Service
Registery
Service B
#1
Service B
#2
Service A
Service
Registery
Service B
#1
Service B
#2
Hi, I am Service B (register)
Hi, I still alive (send heartbeat)
I want to call Service B
Service B has a Instance #1
Call Service B Instance #1
Hi, I am Service B (register)

78. 78
Service A
Service
Registery
Service B
#1
Service B
#2
Service A
Service
Registery
Service B
#1
Service B
#2
Hi, I am Service B (register)
Hi, I still alive (send heartbeat)
I want to call Service B
Service B has a Instance #1
Call Service B Instance #1
Hi, I am Service B (register)
Hi, I still alive (send heartbeat)

79. 79
Service A
Service
Registery
Service B
#1
Service B
#2
Service A
Service
Registery
Service B
#1
Service B
#2
I want to call Service B

80. 80
Service A
Service
Registery
Service B
#1
Service B
#2
Service A
Service
Registery
Service B
#1
Service B
#2
I want to call Service B
Service B has a Instance #1, #2

81. 81
Service A
Service
Registery
Service B
#1
Service B
#2
Service A
Service
Registery
Service B
#1
Service B
#2
I want to call Service B
Service B has a Instance #1, #2
Call Service B Instance #2 (50%)

82. 82
Service A
Service
Registery
Service B
#1
Service B
#2
Service A
Service
Registery
Service B
#1
Service B
#2
I want to call Service B
Service B has a Instance #1, #2
Call Service B Instance #2 (50%)
I wanna to leave (deregister)

83. 83
Service A
Service
Registery
Service B
#1
Service B
#2
Service A
Service
Registery
Service B
#1
Service B
#2
I want to call Service B
Service B has a Instance #1, #2
Call Service B Instance #2 (50%)
I wanna to leave (deregister)
I want to call Service B

84. 84
Service A
Service
Registery
Service B
#1
Service B
#2
Service A
Service
Registery
Service B
#1
Service B
#2
I want to call Service B
Service B has a Instance #1, #2
Call Service B Instance #2 (50%)
I wanna to leave (deregister)
I want to call Service B
Service B has a Instance #2

85. 85
Service A
Service
Registery
Service B
#1
Service B
#2
Service A
Service
Registery
Service B
#1
Service B
#2
I want to call Service B
Service B has a Instance #1, #2
Call Service B Instance #2 (50%)
I wanna to leave (deregister)
I want to call Service B
Service B has a Instance #2
Call Service B Instance #2

86. Environment 1
A
B
C D
E
cm.environment1.com
cm.environment2.com
CM
Environment 2
A
B
C D
ECM
86

87. Multiple tenancies infrastructure in client-side service discovery
https://www.nginx.com/blog/service-discovery-in-a-microservices-architecture/
VIP
VIP
Normal
87

88. VIP
Normal
Cleint
VIP.client.service.com
Normal.client.service.com
Multiple tenancies infrastructure in server-side service discovery
88

89. Step #6
Mininum Configuration
(How to Discover Service Discovery?)
89

90. 90
https://www.slideshare.net/AmazonWebServices/leveraging-elastic-webscale-computing-with-aws

91. 91
Put bootstring configuration on AWS EC2 tags

92. Convert to Environment Variables
It Could Cover Container, VM base Application
92

93. Environment 1
AM
CI Server
CMService
Environment 2
CI Server
CM
0. Build Config
By Cfgctl
0. Build Code
Service
S
D
K
S
D
K
93

94. Environment 1
AM
CI Server
CMService
Environment 2
CI Server
CM
0. Build Config
By Cfgctl
0. Build Code
Service
S
D
K
S
D
K
94
1. Get Metadata
1. Get Metadata
Service Name, Version, AM, CM, etc..
Service Name, Version, AM, CM, etc..

95. Environment 1
AM
CI Server
CMService
Environment 2
CI Server
2. Get Artifact
2. Get Artifact
CM
0. Build Config
By Cfgctl
0. Build Code
Service
S
D
K
S
D
K
95
1. Get Metadata
1. Get Metadata
Service Name, Version, AM, CM, etc..
Service Name, Version, AM, CM, etc..

96. Environment 1
AM
CI Server
CMService
Environment 2
CI Server
3. Get Config
2. Get Artifact
2. Get Artifact
CM
0. Build Config
By Cfgctl
0. Build Code
3. Get ConfigService
S
D
K
S
D
K
96
1. Get Metadata
1. Get Metadata
Service Name, Version, AM, CM, etc..
Service Name, Version, AM, CM, etc..

97. Service Provisioning
● Build code and configuration
● Get Metadata
● Get Artifacts
● Get Confugration
● Run Service
97

98. Step #7
Types of Configuration
98

99. 3 Levels of Configuration
● Realtime
○ Configuration SDK + Configuration Storages (Consul
or S3)
● Service Discovery
○ Configuration SDK + Advance Configuration Services
(Consul)
● Provision
○ Configuration Storages (Consul or S3)
99

100. Agenda
● Why configuration management?
● How to manage configuration?
● How to go live?
100

101. Rule #1.
Consider what you can contribute to teams and minimize the
impact on them
101

102. 102
https://www.fabiaoqing.com/search/search/keyword/我委屈但是我不哭

103. ● Arch Team
○ Design
○ Configuration SDK
■ Easy to change configuration storages
■ Cache
■ Hotreload
■ Service Registration
Consider what you can contribute to teams and
minimize the impact on them
103

104. Consider what you can contribute to teams and
minimize the impact on them
104
● Infra Team & Common Service Team
○ PoC / Provision / Maintain Configuration Storage
○ Consider network topology / security
○ Build tools: Cfgctl
■ Resolve level configuration issue
○ Use it in our own services first

105. Rule #2.
Prove it works
105

106. 106

107. Rule #3.
Use it in small project or your services
107

108. Rule #4.
Prove it is stable
108

109. Service SDK
Consul
S3
Local Files
109

110. 110
Performance across the S3 Storage Classes
https://aws.amazon.com/s3/storage-classes/?nc1=h_ls

111. Server
(Follower)
Server
(Leader)
Server
(Follower)
GOSSIP
Replication
Leader
Forwarding
Replication
Client Client Client
RPC
Leader
Forwarding
GOSSIP
RPCGOSSIP
Consul Cluster
https://learn.hashicorp.com/consul/datacenter-deploy/deployment-guide111

112. Server
(Follower)
Server
(Leader)
Server
(Follower)
GOSSIP
Replication
Leader
Forwarding
Replication
Client Client Client
RPC
Leader
Forwarding
GOSSIP
RPCGOSSIP
Consul Cluster Availability
112

113. Server
(Follower)
Server
(Follower)
GOSSIP
Leader
Forwarding
Replication
Client Client Client
RPC
GOSSIP
RPCGOSSIP
Consul Cluster Availability, leader election
113

114. Consul Cluster Availability: Multiple AZ deployment
https://aws.amazon.com/quickstart/architecture/consul/114

115. Performance Tuning
https://www.consul.io/docs/install/performance.html115

116. 116

117. 117
Design for Operation
http://www.fgsbmc.org.tw/news_latestnews_c.aspx?News_Id=201801084

118. Questions

119. 119

120. Thanks