SlideShare une entreprise Scribd logo
1  sur  30
Télécharger pour lire hors ligne
Effective Service + Resource
Management with systemd
Adventures running millions of systemd services for
About Me and Pantheon
● Production users
of systemd since 2011
● Millions of units in
deployment across hundreds
of servers
● Committer since 2012
● Focus has been on journal
logging, control group
scalability, and general
systemd scalability
The Basic Steps
1 Define expected behavior and control
2 Plan for the unexpected
3 Tighten security
4 Manage, monitor, and automate
Service Types
1 Define expected behavior and control
Type=simple (the default)
systemctl start foo.service systemctl stop foo.service
ExecStart=/usr/bin/foo
/etc/systemd/system/foo.service
Considered started for dependencies
Considered stopped for dependencies
[Service]
ExecStart=/usr/bin/foo
# systemctl daemon-reload
Type=oneshot
systemctl start foo.service systemctl stop foo.service
*Unless RemainAfterExit=true
*
ExecStart=/usr/bin/foo
[Service]
Type=oneshot
ExecStart=/usr/bin/foo
RuntimeMaxSec=30
/etc/systemd/system/foo.service
RuntimeMaxSec=30
Type=forking
systemctl start foo.service
systemctl stop foo.service
ExecStart...
PIDFile=/var/run/foo.pid
[Service]
Type=forking
ExecStart=/usr/bin/foo
PIDFile=/var/run/foo.pid
TimeoutStartSec=30
/etc/systemd/system/foo.service
TimeoutStartSec=30
Type=notify
systemctl start foo.service systemctl stop foo.service
ExecStart...
[Service]
Type=notify
ExecStart=/usr/bin/foo
TimeoutStartSec=30
NotifyAccess=all ⬅maybe
/etc/systemd/system/foo.service
Called from daemon:
systemd-notify --ready
Best of
All
Types
Service Shutdown and Reloading
1 Define expected behavior and control
KillMode=control-group (the default)
systemctl stop foo.service
[Service]
ExecStart=/usr/bin/foo
KillMode=control-group
TimeoutStopSec=30
/etc/systemd/system/foo.service
PID=100
101
102
103
…or “Oprah’s Favorite Signals”
SIGTERM
PID=100
101
102
103
SIGKILL
TimeoutStopSec=30
KillMode=none
systemctl stop foo.service
[Service]
ExecStart=/usr/bin/foo
KillMode=none
ExecStop=/usr/bin/fooctl
stop
/etc/systemd/system/foo.service
PID=100
101
102
103
PID=100
101
102
103
No CleanupExecStop=/usr/bin/fooctl stop
KillMode=process
systemctl stop foo.service
[Service]
ExecStart=/usr/bin/foo
KillMode=process
/etc/systemd/system/foo.service
PID=100
101
102
103
SIGTERM PID=100
101
102
103
No Cleanup
KillMode=mixed
systemctl stop foo.service
[Service]
ExecStart=/usr/bin/foo
KillMode=mixed
TimeoutStopSec=30
/etc/systemd/system/foo.service
PID=100
101
102
103
SIGTERM PID=100
101
102
103
SIGKILL
TimeoutStopSec=30
Best
for
Most
ExecReload=
systemctl reload foo.service
[Service]
ExecStart=/usr/bin/foo
ExecReload=/bin/kill -HUP $MAINPID
/etc/systemd/system/foo.service
Use Me
ExecReload=/bin/kill -HUP $MAINPID
Dependencies and Transactions
1 Define expected behavior and control
WantedBy=
Implicit in late bootup:
systemctl start multi-user.target
[Service]
ExecStart=/usr/bin/foo
[Install]
WantedBy=multi-user.target
/etc/systemd/system/foo.service
Use Me
# systemctl enable foo.service
Added to transaction by wants:
systemctl start foo.service
multi-user.target completes startup
Operations in systemd happen in transactions, which are ordered sets of jobs.
…the successor to runlevels
Other Dependencies
Inclusion
These dependencies will add more units to a
transaction. There is no effect on ordering.
● Requires=bar.service
○ If foo.service is starting, starting bar.service
will also happen. A failure to start bar.service
will cause the entire transaction to fail.
○ Inverse of RequiredBy=
● Wants=bar.service
○ A weak form of Requires=. If bar.service fails
to start, the transaction will still succeed.
○ Inverse of WantedBy=
● Also=bar.service
○ When foo.service is enabled to start by
default, bar.service will also be enabled.
Ordering
These dependencies will order units in the
transaction. They will not add specified units if
they are not already in the transaction.
● Before=bar.service
○ If bar.service is in the same transaction, bar.
service will not begin starting until foo.
service is finished starting.
● After=bar.service
○ If bar.service is in the same transaction, foo.
service will not begin starting until bar.
service is finished starting.
[Unit]
Requires=bar.service
After=bar.service
...
/etc/systemd/system/foo.service
Controlling Resources
1 Define expected behavior and control
Control Groups Options for Resources
Absolute Limits
● MemoryLimit=
○ Caution: Certain limits cause further
allocation for a group to use swap, impacting
system performance.
● TasksMax=
○ Maximum combined processes and threads,
including kernel threads.
● BlockIOReadBandwidth=
○ Limits reading block I/O to the specified
bytes
per second.
● BlockIOWriteBandwidth=
○ Limits writing block I/O to the specified
bytes
Relative Controls and More
● CPUShares=
○ When under contention, CPU is allocated by
the kernel proportionally using the number
for this service versus the combined shares of
all others.
● BlockIOWeight=
○ When under contention, block I/O is
allocated by the kernel proportionally using
the number for this service versus the
combined weights of all others.
● nftables for network traffic
○ Not configured in systemd, but nftables can
leverage systemd’s control groups for traffic
shaping and other rules.
Using Traditional ulimit/rlimit Options
● CPU
○ LimitCPU=
○ LimitNPROC=
○ LimitRTPRIO=
○ LimitRTTIME=
○ LimitNICE=
● Disk
○ LimitCORE=
● Memory
○ LimitDATA=
○ LimitFSIZE=
○ LimitSTACK=
○ LimitMSGQUEUE=
○ LimitAS=
○ LimitRSS=
○ LimitMEMLOCK=
● Other
○ LimitSIGPENDING=
○ LimitNOFILE=
○ LimitLOCKS=
Handling Timeouts and Abnormal Exits
2 Plan for the unexpected
Directives for Detecting and Responding to Failure
Detecting Failure
● SuccessExitStatus=
○ Whitelist of exit codes and signals to indicate a
normal exit. Defaults to zero and the usual process
signals for healthy processes.
● RestartPreventExitStatus=
○ Blacklist of exit codes and signals to not trigger
restarts. Useful to restart on most failures but not
unrecoverable ones like a bad configuration.
● RestartForceExitStatus=
○ The opposite of the previous option.
● StartLimitInterval= and StartLimitBurst=
○ Thresholds at which attempted failure recovery
becomes a stickier failure.
Responding to Failure
● Restart=
○ Allows many options, but on-failure is
probably best for most cases.
● FailureAction=
○ Supports options like rebooting or shutting
down the system on service failure.
● StartLimitAction=
○ Same as FailureAction= but triggered when
StartLimit… thresholds get hit.
● systemctl reset-failed
○ Resets status units marked as failed.
Built-In Service Monitoring with Watchdog
Services
● WatchdogSec=
○ Configures the maximum interval for the
healthy service to ping systemd.
● $WATCHDOG_USEC and $WATCHDOG_PID
○ Environmental variables set for a service that
is expected to provide systemd with
watchdog pings.
● systemd-notify WATCHDOG=1
○ CLI; the most basic way for a service to send
systemd a watchdog ping.
● sd_notify(0, “WATCHDOG=1”);
○ A better way that requires linking to a
systemd library.
Overall System
● RuntimeWatchdogSec=
○ Configures the maximum interval for
systemd to ping the hardware watchdog
service (if it exists). If the hardware fails to
receive an expected ping, it will reboot the
system.
● ShutdownWatchdogSec=
○ Bounds the time the watchdog hardware is
willing to wait for a clean shutdown for the
triggered reboot.
Dropping Privileges and Access Early
3 Tighten security
Dropping Privileges and Access Early
● Hardening options that mostly just work
○ User=<service-user>
○ PrivateTmp=true
○ PrivateDevices=true
○ ProtectSystem=full
○ ProtectHome=read-only
○ NoNewPrivileges=true
○ MountFlags=private
○ SystemCallArchitectures=native
○ SecureBits=noroot noroot-locked
● Restrict visible directories
○ ReadWriteDirectories=
○ ReadOnlyDirectories=
○ InaccessibleDirectories=
○ RootDirectory=
runs the service in chroot
● Whitelist capabilities and system calls
○ AmbientCapabilities=
○ CapabilityBoundingSet=
○ SystemCallFilter=
○ SystemCallErrorNumber=EPERM
tests filters in a non-enforcing mode
● Control sockets
○ RestrictAddressFamilies=
○ PrivateNetwork=true, which is best
combined with socket activation
● Bridge to mandatory access control (MAC)
○ SELinuxContext=
○ AppArmorProfile=
○ SmackProcessLabel=
Monitoring
4 Manage, monitor, and automate
Monitor at the Box Level
Plug a systemctl call into your monitoring tool:
# systemctl --state=failed --all
0 loaded units listed.
To show all installed unit files use 'systemctl list-unit-files'.
Automation
4 Manage, monitor, and automate
Pantheon is a Chef Shop
template '/etc/systemd/system/foo.service' do
mode '0644'
source 'foo.service.erb'
end
service 'foo.service' do
provider Chef::Provider::Service::Systemd
supports :status => true, :restart => true, :reload => true
action [ :enable, :start ]
end
Questions? Follow Ups?
Reach out to me @DavidStrauss.
Want to get more hands-on? We’re hiring!
pantheon.io/careers

Contenu connexe

Tendances

Tiered Compilation in Hotspot JVM
Tiered Compilation in Hotspot JVMTiered Compilation in Hotspot JVM
Tiered Compilation in Hotspot JVMIgor Veresov
 
Debugging linux kernel tools and techniques
Debugging linux kernel tools and  techniquesDebugging linux kernel tools and  techniques
Debugging linux kernel tools and techniquesSatpal Parmar
 
Android Security Internals
Android Security InternalsAndroid Security Internals
Android Security InternalsOpersys inc.
 
Embedded Linux/ Debian with ARM64 Platform
Embedded Linux/ Debian with ARM64 PlatformEmbedded Linux/ Debian with ARM64 Platform
Embedded Linux/ Debian with ARM64 PlatformSZ Lin
 
Linux User Space Debugging & Profiling
Linux User Space Debugging & ProfilingLinux User Space Debugging & Profiling
Linux User Space Debugging & ProfilingAnil Kumar Pugalia
 
RISC-V on Edge: Porting EVE and Alpine Linux to RISC-V
RISC-V on Edge: Porting EVE and Alpine Linux to RISC-VRISC-V on Edge: Porting EVE and Alpine Linux to RISC-V
RISC-V on Edge: Porting EVE and Alpine Linux to RISC-VScyllaDB
 
Linux Kernel Booting Process (1) - For NLKB
Linux Kernel Booting Process (1) - For NLKBLinux Kernel Booting Process (1) - For NLKB
Linux Kernel Booting Process (1) - For NLKBshimosawa
 
Enable DPDK and SR-IOV for containerized virtual network functions with zun
Enable DPDK and SR-IOV for containerized virtual network functions with zunEnable DPDK and SR-IOV for containerized virtual network functions with zun
Enable DPDK and SR-IOV for containerized virtual network functions with zunheut2008
 
Linux Initialization Process (2)
Linux Initialization Process (2)Linux Initialization Process (2)
Linux Initialization Process (2)shimosawa
 
Sisteme de Operare: Sisteme de fisiere
Sisteme de Operare: Sisteme de fisiereSisteme de Operare: Sisteme de fisiere
Sisteme de Operare: Sisteme de fisiereAlexandru Radovici
 
Linux Kernel Crashdump
Linux Kernel CrashdumpLinux Kernel Crashdump
Linux Kernel CrashdumpMarian Marinov
 
BUD17-302: LLVM Internals #2
BUD17-302: LLVM Internals #2 BUD17-302: LLVM Internals #2
BUD17-302: LLVM Internals #2 Linaro
 
Implementing generic JNI hardware control for Kotlin based app on AOSP
Implementing generic JNI hardware control for Kotlin based app on AOSPImplementing generic JNI hardware control for Kotlin based app on AOSP
Implementing generic JNI hardware control for Kotlin based app on AOSPCheng Wig
 
Linux Initialization Process (1)
Linux Initialization Process (1)Linux Initialization Process (1)
Linux Initialization Process (1)shimosawa
 
Learning AOSP - Android Booting Process
Learning AOSP - Android Booting ProcessLearning AOSP - Android Booting Process
Learning AOSP - Android Booting ProcessNanik Tolaram
 
Kernel Recipes 2015: Anatomy of an atomic KMS driver
Kernel Recipes 2015: Anatomy of an atomic KMS driverKernel Recipes 2015: Anatomy of an atomic KMS driver
Kernel Recipes 2015: Anatomy of an atomic KMS driverAnne Nicolas
 
Android Booting Sequence
Android Booting SequenceAndroid Booting Sequence
Android Booting SequenceJayanta Ghoshal
 

Tendances (20)

Tiered Compilation in Hotspot JVM
Tiered Compilation in Hotspot JVMTiered Compilation in Hotspot JVM
Tiered Compilation in Hotspot JVM
 
Debugging linux kernel tools and techniques
Debugging linux kernel tools and  techniquesDebugging linux kernel tools and  techniques
Debugging linux kernel tools and techniques
 
Android Security Internals
Android Security InternalsAndroid Security Internals
Android Security Internals
 
Embedded Linux/ Debian with ARM64 Platform
Embedded Linux/ Debian with ARM64 PlatformEmbedded Linux/ Debian with ARM64 Platform
Embedded Linux/ Debian with ARM64 Platform
 
Linux User Space Debugging & Profiling
Linux User Space Debugging & ProfilingLinux User Space Debugging & Profiling
Linux User Space Debugging & Profiling
 
RISC-V on Edge: Porting EVE and Alpine Linux to RISC-V
RISC-V on Edge: Porting EVE and Alpine Linux to RISC-VRISC-V on Edge: Porting EVE and Alpine Linux to RISC-V
RISC-V on Edge: Porting EVE and Alpine Linux to RISC-V
 
Linux Kernel Booting Process (1) - For NLKB
Linux Kernel Booting Process (1) - For NLKBLinux Kernel Booting Process (1) - For NLKB
Linux Kernel Booting Process (1) - For NLKB
 
Enable DPDK and SR-IOV for containerized virtual network functions with zun
Enable DPDK and SR-IOV for containerized virtual network functions with zunEnable DPDK and SR-IOV for containerized virtual network functions with zun
Enable DPDK and SR-IOV for containerized virtual network functions with zun
 
Linux Initialization Process (2)
Linux Initialization Process (2)Linux Initialization Process (2)
Linux Initialization Process (2)
 
Sisteme de Operare: Sisteme de fisiere
Sisteme de Operare: Sisteme de fisiereSisteme de Operare: Sisteme de fisiere
Sisteme de Operare: Sisteme de fisiere
 
Linux Kernel Crashdump
Linux Kernel CrashdumpLinux Kernel Crashdump
Linux Kernel Crashdump
 
BUD17-302: LLVM Internals #2
BUD17-302: LLVM Internals #2 BUD17-302: LLVM Internals #2
BUD17-302: LLVM Internals #2
 
Implementing generic JNI hardware control for Kotlin based app on AOSP
Implementing generic JNI hardware control for Kotlin based app on AOSPImplementing generic JNI hardware control for Kotlin based app on AOSP
Implementing generic JNI hardware control for Kotlin based app on AOSP
 
Linux Internals - Part II
Linux Internals - Part IILinux Internals - Part II
Linux Internals - Part II
 
Linux Initialization Process (1)
Linux Initialization Process (1)Linux Initialization Process (1)
Linux Initialization Process (1)
 
Android Things : Building Embedded Devices
Android Things : Building Embedded DevicesAndroid Things : Building Embedded Devices
Android Things : Building Embedded Devices
 
Learning AOSP - Android Booting Process
Learning AOSP - Android Booting ProcessLearning AOSP - Android Booting Process
Learning AOSP - Android Booting Process
 
Android Internals
Android InternalsAndroid Internals
Android Internals
 
Kernel Recipes 2015: Anatomy of an atomic KMS driver
Kernel Recipes 2015: Anatomy of an atomic KMS driverKernel Recipes 2015: Anatomy of an atomic KMS driver
Kernel Recipes 2015: Anatomy of an atomic KMS driver
 
Android Booting Sequence
Android Booting SequenceAndroid Booting Sequence
Android Booting Sequence
 

Similaire à Effective service and resource management with systemd

Linux : Booting and runlevels
Linux : Booting and runlevelsLinux : Booting and runlevels
Linux : Booting and runlevelsJohn Ombagi
 
Fully Automated Nagios (FAN)
Fully Automated Nagios (FAN)Fully Automated Nagios (FAN)
Fully Automated Nagios (FAN)Kaustubh Padwad
 
linux monitoring and performance tunning
linux monitoring and performance tunning linux monitoring and performance tunning
linux monitoring and performance tunning iman darabi
 
How To Get The Most Out Of Your Hibernate, JBoss EAP 7 Application (Ståle Ped...
How To Get The Most Out Of Your Hibernate, JBoss EAP 7 Application (Ståle Ped...How To Get The Most Out Of Your Hibernate, JBoss EAP 7 Application (Ståle Ped...
How To Get The Most Out Of Your Hibernate, JBoss EAP 7 Application (Ståle Ped...Red Hat Developers
 
10 Tips for AIX Security
10 Tips for AIX Security10 Tips for AIX Security
10 Tips for AIX SecurityHelpSystems
 
Summit demystifying systemd1
Summit demystifying systemd1Summit demystifying systemd1
Summit demystifying systemd1Susant Sahani
 
FALLSEM2023-24_BCSE302L_TH_VL2023240100957_2023-06-21_Reference-Material-I.pptx
FALLSEM2023-24_BCSE302L_TH_VL2023240100957_2023-06-21_Reference-Material-I.pptxFALLSEM2023-24_BCSE302L_TH_VL2023240100957_2023-06-21_Reference-Material-I.pptx
FALLSEM2023-24_BCSE302L_TH_VL2023240100957_2023-06-21_Reference-Material-I.pptxhritikraj888
 
Computer system architecture
Computer system architectureComputer system architecture
Computer system architecturejeetesh036
 
101 1.3 runlevels , shutdown, and reboot
101 1.3 runlevels , shutdown, and reboot101 1.3 runlevels , shutdown, and reboot
101 1.3 runlevels , shutdown, and rebootAcácio Oliveira
 
Process Management Operating Systems .pptx
Process Management        Operating Systems .pptxProcess Management        Operating Systems .pptx
Process Management Operating Systems .pptxSAIKRISHNADURVASULA2
 
Kernel Process Management
Kernel Process ManagementKernel Process Management
Kernel Process Managementpradeep_tewani
 
When the OS gets in the way
When the OS gets in the wayWhen the OS gets in the way
When the OS gets in the wayMark Price
 
LISA15: systemd, the Next-Generation Linux System Manager
LISA15: systemd, the Next-Generation Linux System Manager LISA15: systemd, the Next-Generation Linux System Manager
LISA15: systemd, the Next-Generation Linux System Manager Alison Chaiken
 
Archivematica Technical Training Diagnostics Guide (September 2018)
Archivematica Technical Training Diagnostics Guide (September 2018)Archivematica Technical Training Diagnostics Guide (September 2018)
Archivematica Technical Training Diagnostics Guide (September 2018)Artefactual Systems - Archivematica
 
Nagios Conference 2014 - Eric Mislivec - Getting Started With Nagios Core
Nagios Conference 2014 - Eric Mislivec - Getting Started With Nagios CoreNagios Conference 2014 - Eric Mislivec - Getting Started With Nagios Core
Nagios Conference 2014 - Eric Mislivec - Getting Started With Nagios CoreNagios
 
Unit 2_OS process management
Unit 2_OS process management Unit 2_OS process management
Unit 2_OS process management JayeshGadhave1
 
Multi-Threading.pptx
Multi-Threading.pptxMulti-Threading.pptx
Multi-Threading.pptxCHANDRUG31
 

Similaire à Effective service and resource management with systemd (20)

Optimizing Linux Servers
Optimizing Linux ServersOptimizing Linux Servers
Optimizing Linux Servers
 
Linux : Booting and runlevels
Linux : Booting and runlevelsLinux : Booting and runlevels
Linux : Booting and runlevels
 
Fully Automated Nagios (FAN)
Fully Automated Nagios (FAN)Fully Automated Nagios (FAN)
Fully Automated Nagios (FAN)
 
linux monitoring and performance tunning
linux monitoring and performance tunning linux monitoring and performance tunning
linux monitoring and performance tunning
 
How To Get The Most Out Of Your Hibernate, JBoss EAP 7 Application (Ståle Ped...
How To Get The Most Out Of Your Hibernate, JBoss EAP 7 Application (Ståle Ped...How To Get The Most Out Of Your Hibernate, JBoss EAP 7 Application (Ståle Ped...
How To Get The Most Out Of Your Hibernate, JBoss EAP 7 Application (Ståle Ped...
 
Pdf c1t tlawaxb
Pdf c1t tlawaxbPdf c1t tlawaxb
Pdf c1t tlawaxb
 
10 Tips for AIX Security
10 Tips for AIX Security10 Tips for AIX Security
10 Tips for AIX Security
 
Operating System.pdf
Operating System.pdfOperating System.pdf
Operating System.pdf
 
Summit demystifying systemd1
Summit demystifying systemd1Summit demystifying systemd1
Summit demystifying systemd1
 
FALLSEM2023-24_BCSE302L_TH_VL2023240100957_2023-06-21_Reference-Material-I.pptx
FALLSEM2023-24_BCSE302L_TH_VL2023240100957_2023-06-21_Reference-Material-I.pptxFALLSEM2023-24_BCSE302L_TH_VL2023240100957_2023-06-21_Reference-Material-I.pptx
FALLSEM2023-24_BCSE302L_TH_VL2023240100957_2023-06-21_Reference-Material-I.pptx
 
Computer system architecture
Computer system architectureComputer system architecture
Computer system architecture
 
101 1.3 runlevels , shutdown, and reboot
101 1.3 runlevels , shutdown, and reboot101 1.3 runlevels , shutdown, and reboot
101 1.3 runlevels , shutdown, and reboot
 
Process Management Operating Systems .pptx
Process Management        Operating Systems .pptxProcess Management        Operating Systems .pptx
Process Management Operating Systems .pptx
 
Kernel Process Management
Kernel Process ManagementKernel Process Management
Kernel Process Management
 
When the OS gets in the way
When the OS gets in the wayWhen the OS gets in the way
When the OS gets in the way
 
LISA15: systemd, the Next-Generation Linux System Manager
LISA15: systemd, the Next-Generation Linux System Manager LISA15: systemd, the Next-Generation Linux System Manager
LISA15: systemd, the Next-Generation Linux System Manager
 
Archivematica Technical Training Diagnostics Guide (September 2018)
Archivematica Technical Training Diagnostics Guide (September 2018)Archivematica Technical Training Diagnostics Guide (September 2018)
Archivematica Technical Training Diagnostics Guide (September 2018)
 
Nagios Conference 2014 - Eric Mislivec - Getting Started With Nagios Core
Nagios Conference 2014 - Eric Mislivec - Getting Started With Nagios CoreNagios Conference 2014 - Eric Mislivec - Getting Started With Nagios Core
Nagios Conference 2014 - Eric Mislivec - Getting Started With Nagios Core
 
Unit 2_OS process management
Unit 2_OS process management Unit 2_OS process management
Unit 2_OS process management
 
Multi-Threading.pptx
Multi-Threading.pptxMulti-Threading.pptx
Multi-Threading.pptx
 

Plus de David Timothy Strauss

Plus de David Timothy Strauss (14)

Advanced Drupal 8 Caching
Advanced Drupal 8 CachingAdvanced Drupal 8 Caching
Advanced Drupal 8 Caching
 
LCache DrupalCon Dublin 2016
LCache DrupalCon Dublin 2016LCache DrupalCon Dublin 2016
LCache DrupalCon Dublin 2016
 
Container Security via Monitoring and Orchestration - Container Security Summit
Container Security via Monitoring and Orchestration - Container Security SummitContainer Security via Monitoring and Orchestration - Container Security Summit
Container Security via Monitoring and Orchestration - Container Security Summit
 
Don't Build "Death Star" Security - O'Reilly Software Architecture Conference...
Don't Build "Death Star" Security - O'Reilly Software Architecture Conference...Don't Build "Death Star" Security - O'Reilly Software Architecture Conference...
Don't Build "Death Star" Security - O'Reilly Software Architecture Conference...
 
Containers > VMs
Containers > VMsContainers > VMs
Containers > VMs
 
PHP at Density and Scale (Lone Star PHP 2014)
PHP at Density and Scale (Lone Star PHP 2014)PHP at Density and Scale (Lone Star PHP 2014)
PHP at Density and Scale (Lone Star PHP 2014)
 
PHP at Density and Scale
PHP at Density and ScalePHP at Density and Scale
PHP at Density and Scale
 
PHP at Density and Scale
PHP at Density and ScalePHP at Density and Scale
PHP at Density and Scale
 
Valhalla at Pantheon
Valhalla at PantheonValhalla at Pantheon
Valhalla at Pantheon
 
Cassandra-Powered Distributed DNS
Cassandra-Powered Distributed DNSCassandra-Powered Distributed DNS
Cassandra-Powered Distributed DNS
 
Scalable Drupal Infrastructure
Scalable Drupal InfrastructureScalable Drupal Infrastructure
Scalable Drupal Infrastructure
 
Planning LAMP infrastructure
Planning LAMP infrastructurePlanning LAMP infrastructure
Planning LAMP infrastructure
 
Is Drupal Secure?
Is Drupal Secure?Is Drupal Secure?
Is Drupal Secure?
 
Cassandra queuing
Cassandra queuingCassandra queuing
Cassandra queuing
 

Dernier

[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdfhans926745
 
Tata AIG General Insurance Company - Insurer Innovation Award 2024
Tata AIG General Insurance Company - Insurer Innovation Award 2024Tata AIG General Insurance Company - Insurer Innovation Award 2024
Tata AIG General Insurance Company - Insurer Innovation Award 2024The Digital Insurer
 
GenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day PresentationGenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day PresentationMichael W. Hawkins
 
Developing An App To Navigate The Roads of Brazil
Developing An App To Navigate The Roads of BrazilDeveloping An App To Navigate The Roads of Brazil
Developing An App To Navigate The Roads of BrazilV3cube
 
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...apidays
 
Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024The Digital Insurer
 
Data Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt RobisonData Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt RobisonAnna Loughnan Colquhoun
 
HTML Injection Attacks: Impact and Mitigation Strategies
HTML Injection Attacks: Impact and Mitigation StrategiesHTML Injection Attacks: Impact and Mitigation Strategies
HTML Injection Attacks: Impact and Mitigation StrategiesBoston Institute of Analytics
 
Handwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed textsHandwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed textsMaria Levchenko
 
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...Miguel Araújo
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerThousandEyes
 
Tech Trends Report 2024 Future Today Institute.pdf
Tech Trends Report 2024 Future Today Institute.pdfTech Trends Report 2024 Future Today Institute.pdf
Tech Trends Report 2024 Future Today Institute.pdfhans926745
 
A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)Gabriella Davis
 
GenAI Risks & Security Meetup 01052024.pdf
GenAI Risks & Security Meetup 01052024.pdfGenAI Risks & Security Meetup 01052024.pdf
GenAI Risks & Security Meetup 01052024.pdflior mazor
 
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, AdobeApidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobeapidays
 
Automating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps ScriptAutomating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps Scriptwesley chun
 
Boost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivityBoost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivityPrincipled Technologies
 
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot TakeoffStrategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoffsammart93
 
🐬 The future of MySQL is Postgres 🐘
🐬  The future of MySQL is Postgres   🐘🐬  The future of MySQL is Postgres   🐘
🐬 The future of MySQL is Postgres 🐘RTylerCroy
 
Exploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone ProcessorsExploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone Processorsdebabhi2
 

Dernier (20)

[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf
 
Tata AIG General Insurance Company - Insurer Innovation Award 2024
Tata AIG General Insurance Company - Insurer Innovation Award 2024Tata AIG General Insurance Company - Insurer Innovation Award 2024
Tata AIG General Insurance Company - Insurer Innovation Award 2024
 
GenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day PresentationGenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day Presentation
 
Developing An App To Navigate The Roads of Brazil
Developing An App To Navigate The Roads of BrazilDeveloping An App To Navigate The Roads of Brazil
Developing An App To Navigate The Roads of Brazil
 
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
 
Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024
 
Data Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt RobisonData Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt Robison
 
HTML Injection Attacks: Impact and Mitigation Strategies
HTML Injection Attacks: Impact and Mitigation StrategiesHTML Injection Attacks: Impact and Mitigation Strategies
HTML Injection Attacks: Impact and Mitigation Strategies
 
Handwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed textsHandwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed texts
 
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
 
Tech Trends Report 2024 Future Today Institute.pdf
Tech Trends Report 2024 Future Today Institute.pdfTech Trends Report 2024 Future Today Institute.pdf
Tech Trends Report 2024 Future Today Institute.pdf
 
A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)
 
GenAI Risks & Security Meetup 01052024.pdf
GenAI Risks & Security Meetup 01052024.pdfGenAI Risks & Security Meetup 01052024.pdf
GenAI Risks & Security Meetup 01052024.pdf
 
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, AdobeApidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
 
Automating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps ScriptAutomating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps Script
 
Boost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivityBoost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivity
 
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot TakeoffStrategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
 
🐬 The future of MySQL is Postgres 🐘
🐬  The future of MySQL is Postgres   🐘🐬  The future of MySQL is Postgres   🐘
🐬 The future of MySQL is Postgres 🐘
 
Exploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone ProcessorsExploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone Processors
 

Effective service and resource management with systemd

  • 1. Effective Service + Resource Management with systemd Adventures running millions of systemd services for
  • 2. About Me and Pantheon ● Production users of systemd since 2011 ● Millions of units in deployment across hundreds of servers ● Committer since 2012 ● Focus has been on journal logging, control group scalability, and general systemd scalability
  • 3. The Basic Steps 1 Define expected behavior and control 2 Plan for the unexpected 3 Tighten security 4 Manage, monitor, and automate
  • 4. Service Types 1 Define expected behavior and control
  • 5. Type=simple (the default) systemctl start foo.service systemctl stop foo.service ExecStart=/usr/bin/foo /etc/systemd/system/foo.service Considered started for dependencies Considered stopped for dependencies [Service] ExecStart=/usr/bin/foo # systemctl daemon-reload
  • 6. Type=oneshot systemctl start foo.service systemctl stop foo.service *Unless RemainAfterExit=true * ExecStart=/usr/bin/foo [Service] Type=oneshot ExecStart=/usr/bin/foo RuntimeMaxSec=30 /etc/systemd/system/foo.service RuntimeMaxSec=30
  • 7. Type=forking systemctl start foo.service systemctl stop foo.service ExecStart... PIDFile=/var/run/foo.pid [Service] Type=forking ExecStart=/usr/bin/foo PIDFile=/var/run/foo.pid TimeoutStartSec=30 /etc/systemd/system/foo.service TimeoutStartSec=30
  • 8. Type=notify systemctl start foo.service systemctl stop foo.service ExecStart... [Service] Type=notify ExecStart=/usr/bin/foo TimeoutStartSec=30 NotifyAccess=all ⬅maybe /etc/systemd/system/foo.service Called from daemon: systemd-notify --ready Best of All Types
  • 9. Service Shutdown and Reloading 1 Define expected behavior and control
  • 10. KillMode=control-group (the default) systemctl stop foo.service [Service] ExecStart=/usr/bin/foo KillMode=control-group TimeoutStopSec=30 /etc/systemd/system/foo.service PID=100 101 102 103 …or “Oprah’s Favorite Signals” SIGTERM PID=100 101 102 103 SIGKILL TimeoutStopSec=30
  • 14. ExecReload= systemctl reload foo.service [Service] ExecStart=/usr/bin/foo ExecReload=/bin/kill -HUP $MAINPID /etc/systemd/system/foo.service Use Me ExecReload=/bin/kill -HUP $MAINPID
  • 15. Dependencies and Transactions 1 Define expected behavior and control
  • 16. WantedBy= Implicit in late bootup: systemctl start multi-user.target [Service] ExecStart=/usr/bin/foo [Install] WantedBy=multi-user.target /etc/systemd/system/foo.service Use Me # systemctl enable foo.service Added to transaction by wants: systemctl start foo.service multi-user.target completes startup Operations in systemd happen in transactions, which are ordered sets of jobs. …the successor to runlevels
  • 17. Other Dependencies Inclusion These dependencies will add more units to a transaction. There is no effect on ordering. ● Requires=bar.service ○ If foo.service is starting, starting bar.service will also happen. A failure to start bar.service will cause the entire transaction to fail. ○ Inverse of RequiredBy= ● Wants=bar.service ○ A weak form of Requires=. If bar.service fails to start, the transaction will still succeed. ○ Inverse of WantedBy= ● Also=bar.service ○ When foo.service is enabled to start by default, bar.service will also be enabled. Ordering These dependencies will order units in the transaction. They will not add specified units if they are not already in the transaction. ● Before=bar.service ○ If bar.service is in the same transaction, bar. service will not begin starting until foo. service is finished starting. ● After=bar.service ○ If bar.service is in the same transaction, foo. service will not begin starting until bar. service is finished starting. [Unit] Requires=bar.service After=bar.service ... /etc/systemd/system/foo.service
  • 18. Controlling Resources 1 Define expected behavior and control
  • 19. Control Groups Options for Resources Absolute Limits ● MemoryLimit= ○ Caution: Certain limits cause further allocation for a group to use swap, impacting system performance. ● TasksMax= ○ Maximum combined processes and threads, including kernel threads. ● BlockIOReadBandwidth= ○ Limits reading block I/O to the specified bytes per second. ● BlockIOWriteBandwidth= ○ Limits writing block I/O to the specified bytes Relative Controls and More ● CPUShares= ○ When under contention, CPU is allocated by the kernel proportionally using the number for this service versus the combined shares of all others. ● BlockIOWeight= ○ When under contention, block I/O is allocated by the kernel proportionally using the number for this service versus the combined weights of all others. ● nftables for network traffic ○ Not configured in systemd, but nftables can leverage systemd’s control groups for traffic shaping and other rules.
  • 20. Using Traditional ulimit/rlimit Options ● CPU ○ LimitCPU= ○ LimitNPROC= ○ LimitRTPRIO= ○ LimitRTTIME= ○ LimitNICE= ● Disk ○ LimitCORE= ● Memory ○ LimitDATA= ○ LimitFSIZE= ○ LimitSTACK= ○ LimitMSGQUEUE= ○ LimitAS= ○ LimitRSS= ○ LimitMEMLOCK= ● Other ○ LimitSIGPENDING= ○ LimitNOFILE= ○ LimitLOCKS=
  • 21. Handling Timeouts and Abnormal Exits 2 Plan for the unexpected
  • 22. Directives for Detecting and Responding to Failure Detecting Failure ● SuccessExitStatus= ○ Whitelist of exit codes and signals to indicate a normal exit. Defaults to zero and the usual process signals for healthy processes. ● RestartPreventExitStatus= ○ Blacklist of exit codes and signals to not trigger restarts. Useful to restart on most failures but not unrecoverable ones like a bad configuration. ● RestartForceExitStatus= ○ The opposite of the previous option. ● StartLimitInterval= and StartLimitBurst= ○ Thresholds at which attempted failure recovery becomes a stickier failure. Responding to Failure ● Restart= ○ Allows many options, but on-failure is probably best for most cases. ● FailureAction= ○ Supports options like rebooting or shutting down the system on service failure. ● StartLimitAction= ○ Same as FailureAction= but triggered when StartLimit… thresholds get hit. ● systemctl reset-failed ○ Resets status units marked as failed.
  • 23. Built-In Service Monitoring with Watchdog Services ● WatchdogSec= ○ Configures the maximum interval for the healthy service to ping systemd. ● $WATCHDOG_USEC and $WATCHDOG_PID ○ Environmental variables set for a service that is expected to provide systemd with watchdog pings. ● systemd-notify WATCHDOG=1 ○ CLI; the most basic way for a service to send systemd a watchdog ping. ● sd_notify(0, “WATCHDOG=1”); ○ A better way that requires linking to a systemd library. Overall System ● RuntimeWatchdogSec= ○ Configures the maximum interval for systemd to ping the hardware watchdog service (if it exists). If the hardware fails to receive an expected ping, it will reboot the system. ● ShutdownWatchdogSec= ○ Bounds the time the watchdog hardware is willing to wait for a clean shutdown for the triggered reboot.
  • 24. Dropping Privileges and Access Early 3 Tighten security
  • 25. Dropping Privileges and Access Early ● Hardening options that mostly just work ○ User=<service-user> ○ PrivateTmp=true ○ PrivateDevices=true ○ ProtectSystem=full ○ ProtectHome=read-only ○ NoNewPrivileges=true ○ MountFlags=private ○ SystemCallArchitectures=native ○ SecureBits=noroot noroot-locked ● Restrict visible directories ○ ReadWriteDirectories= ○ ReadOnlyDirectories= ○ InaccessibleDirectories= ○ RootDirectory= runs the service in chroot ● Whitelist capabilities and system calls ○ AmbientCapabilities= ○ CapabilityBoundingSet= ○ SystemCallFilter= ○ SystemCallErrorNumber=EPERM tests filters in a non-enforcing mode ● Control sockets ○ RestrictAddressFamilies= ○ PrivateNetwork=true, which is best combined with socket activation ● Bridge to mandatory access control (MAC) ○ SELinuxContext= ○ AppArmorProfile= ○ SmackProcessLabel=
  • 27. Monitor at the Box Level Plug a systemctl call into your monitoring tool: # systemctl --state=failed --all 0 loaded units listed. To show all installed unit files use 'systemctl list-unit-files'.
  • 29. Pantheon is a Chef Shop template '/etc/systemd/system/foo.service' do mode '0644' source 'foo.service.erb' end service 'foo.service' do provider Chef::Provider::Service::Systemd supports :status => true, :restart => true, :reload => true action [ :enable, :start ] end
  • 30. Questions? Follow Ups? Reach out to me @DavidStrauss. Want to get more hands-on? We’re hiring! pantheon.io/careers