What are the challenges in managing hundreds or thousands of servers.

1. How you Can How you Can
Manage Hundreds Manage Hundreds
of Serversof Servers
Marian MarinovMarian Marinov
mm@siteground.commm@siteground.com
Chief System ArchitectChief System Architect
Head of the DevOps departmentHead of the DevOps department

2. ❖❖ Who am I?Who am I?
- Chief System Architect of SiteGround.com- Chief System Architect of SiteGround.com
- Sysadmin since 1996- Sysadmin since 1996
- Organizer of OpenFest, BG Perl- Organizer of OpenFest, BG Perl
Workshops, LUG-BG and othersWorkshops, LUG-BG and others
- Teaching Network Security and- Teaching Network Security and
Linux System AdministrationLinux System Administration
courses in Sofia Universitycourses in Sofia University
and SoftUniand SoftUni

3. ❖❖ The theory states
Let's skip the detailsLet's skip the details

4. ❖❖ The theory states
➢ Make all of your servers feel the
same
Let's skip the detailsLet's skip the details

5. ❖❖ The theory states
➢ Make all of your servers feel the
same
➢ Automate everything
Let's skip the detailsLet's skip the details

6. ❖❖ The theory states
➢ Make all of your servers feel the
same
➢ same hardware
➢ same software
➢ Automate everything
Let's skip the detailsLet's skip the details

7. ❖❖ The theory states
➢ Make all of your servers feel the
same
➢ same hardware
➢ same software
➢ Automate everything
➢ no manual deployments
➢ use systems like Ansible, Pupet, Chef
Let's skip the detailsLet's skip the details

8. What most people do?What most people do?

9. ➢ Do the work in parallel
What most people do?What most people do?

10. ➢ Do the work in parallel
➢ pssh pscp (even Pacemaker is using those)
What most people do?What most people do?

11. What we did?What we did?

12. ➢ Multy-command.shMulty-command.sh :)
What we did?What we did?

13. ➢ Multy-command.shMulty-command.sh :)
for srv in $srv_list; do
ssh $srv $cmd &
done
What we did?What we did?
mexec

14. ➢ Multy-command.shMulty-command.sh :)
for srv in $srv_list; do
ssh $srv $cmd &
done
for srv in $srv_list; do
scp $src $srv:$src &
done
What we did?What we did?
mexec
mcopy

15. ❖❖ This was is fine...This was is fine...

16. ❖❖ This was is fine...This was is fine...
when you have under 5-6000 machineswhen you have under 5-6000 machines

17. ➢ Time
➢ Network
➢ Memory, CPU and I/O
There are LIMITS :)There are LIMITS :)

18. ❖❖ Using most automation systems, theUsing most automation systems, the
execution of the tasks takes considerablyexecution of the tasks takes considerably
longer time then actually executing thelonger time then actually executing the
commands directlycommands directly
Time limitTime limit

19. ❖❖ Using most automation systems, theUsing most automation systems, the
execution of the tasks takes considerablyexecution of the tasks takes considerably
longer time then actually executing thelonger time then actually executing the
commands directlycommands directly
❖❖ This happens because of error handlingThis happens because of error handling
and logging of all work, that has beenand logging of all work, that has been
donedone
Time limitTime limit

20. ➢ tipical 1Gbps connection is around
100-110 MB per second
Network limitsNetwork limits

21. ➢ tipical 1Gbps connection is around
100-110 MB per second
➢ if you have 10,000 machines
Network limitsNetwork limits

22. ➢ tipical 1Gbps connection is around
100-110 MB per second
➢ if you have 10,000 machines
➢ 2 KB file is now 20 MB
Network limitsNetwork limits

23. ➢ tipical 1Gbps connection is around
100-110 MB per second
➢ if you have 10,000 machines
➢ 2 KB file is now 20 MB
➢ but 1MB file becomes 10GB...
Network limitsNetwork limits

24. ➢ tipical 1Gbps connection is around
100-110 MB per second
➢ if you have 10,000 machines
➢ 2 KB file is now 20 MB
➢ but 1MB file becomes 10GB...
➢ transferred with 100-110MBps ~ 91sec
Network limitsNetwork limits

25. ➢ tipical 1Gbps connection is around
100-110 MB per second
➢ if you have 10,000 machines
➢ 2 KB file is now 20 MB
➢ but 1MB file becomes 10GB...
➢ transferred with 100-110MBps ~ 91sec
➢ what if you have to download the latest
Apache package( ~ 5MB)?
Network limitsNetwork limits

26. ❖❖ 1 ssh process is ~6MB RAM1 ssh process is ~6MB RAM
Memory, CPU, I/O limitsMemory, CPU, I/O limits

27. ❖❖ 1 ssh process is ~6MB RAM1 ssh process is ~6MB RAM
❖❖ 10,000 * 6 = 60GB RAM10,000 * 6 = 60GB RAM
Memory, CPU, I/O limitsMemory, CPU, I/O limits

28. ❖❖ 1 ssh process is ~6MB RAM1 ssh process is ~6MB RAM
❖❖ 10,000 * 6 = 60GB RAM10,000 * 6 = 60GB RAM
❖❖ spawning 10k processes is costing CPUspawning 10k processes is costing CPU
Memory, CPU, I/O limitsMemory, CPU, I/O limits

29. ❖❖ 1 ssh process is ~6MB RAM1 ssh process is ~6MB RAM
❖❖ 10,000 * 6 = 60GB RAM10,000 * 6 = 60GB RAM
❖❖ spawning 10k processes is costing CPUspawning 10k processes is costing CPU
❖❖ logging data from 10k processes requireslogging data from 10k processes requires
a lot of I/Oa lot of I/O
Memory, CPU, I/O limitsMemory, CPU, I/O limits

30. ❖❖ 1 ssh process is ~6MB RAM1 ssh process is ~6MB RAM
❖❖ 10,000 * 6 = 60GB RAM10,000 * 6 = 60GB RAM
❖❖ spawning 10k processes is costing CPUspawning 10k processes is costing CPU
❖❖ logging data from 10k processes requireslogging data from 10k processes requires
a lot of I/Oa lot of I/O
❖❖ what if the task on each server takeswhat if the task on each server takes
10min...10min...
Memory, CPU, I/O limitsMemory, CPU, I/O limits

31. ➢ How will you handle upgrades?
The reality is even The reality is even
worse...worse...

32. ➢ How will you handle upgrades?
➢ single software update is easy
The reality is even The reality is even
worse...worse...

33. ➢ How will you handle upgrades?
➢ single software update is easy
➢ handling multiple at the same time, that's fun
The reality is even The reality is even
worse...worse...

34. ➢ How will you handle upgrades?
➢ single software update is easy
➢ handling multiple at the same time, that's fun
➢ How will you address security issues?
The reality is even The reality is even
worse...worse...

35. ➢ How will you handle upgrades?
➢ single software update is easy
➢ handling multiple at the same time, that's fun
➢ How will you address security issues?
➢ Tipical timeframe for kernel upgrade for us
is around 45 days
The reality is even The reality is even
worse...worse...

36. ➢ How will you handle upgrades?
➢ single software update is easy
➢ handling multiple at the same time, that's fun
➢ How will you address security issues?
➢ Tipical timeframe for kernel upgrade for us
is around 45 days
➢ it is highly possible that there may be an issue
that will require you to switch to a newer kernel
version in that 45days period
The reality is even The reality is even
worse...worse...

37. ➢ Designing a very effective systems means:
➢ You WILL break stuff VERRY effectively
The reality is even The reality is even
worse...worse...

38. Marian MarinovMarian Marinov
mm@siteground.commm@siteground.com