NixCon 2020 talk about an experimental framework that integrates the Nix package manager with all kinds of process managers, such as : sysvinit, systemd, launchd, and even Docker
The Essentials of Digital Experience Monitoring_ A Comprehensive Guide.pdf
nix-processmgmt: An experimental Nix-based process manager-agnostic framework
1. nix-processmgmt: An experimental Nix-based
process manager-agnostic framework
Sander van der Burg
October 17, 2020
Sander van der Burg nix-processmgmt
2. Nix package manager: a powerful solution
Conveniently construct packages from source code and all its
required build-time dependencies
Build determinism.
Same hash code → (nearly) bit-identical build
Transparent binary deployments (by using substitutes)
Store multiple versions and variants safely next to each other
Thanks to the hash prefixes and the Nix store
Unprivileged user deployments
Multiple operating systems: Linux, macOS, FreeBSD, others
Sander van der Burg nix-processmgmt
3. Nix: development environments
We can conveniently install and use all kinds of packages without
interfering with the host system’s packages.
nix-shell example
$ python --version
Python 2.7.15
$ node --version
node: command not found
$ nix-shell -p python3 nodejs
$ python --version
Python 3.8.5
$ which python
/nix/store/z65l1jqvxa58zzwwa3bvglb6asj4y8cv-python3-3.8.5/bin/python
$ node --version
v12.18.4
$ which node
/nix/store/2w6ilfh7zmbz9zqvphgxinmbn3wdqa1b-nodejs-12.18.4/bin/node
Sander van der Burg nix-processmgmt
5. Nix package manager: not a service manager
Nix does not manage application services/processes’ life-cycles.
Sander van der Burg nix-processmgmt
6. Nix: service deployment integrations
There are sister projects that complement Nix with process manage-
ment:
NixOS. Generates systemd unit files to manage services.
Requires you to switch to a fully Nix-managed Linux
distribution.
nix-darwin. Generates launchd daemon configuration files.
Only works on macOS.
Sander van der Burg nix-processmgmt
7. Nix: service deployment integrations
Nix can also be used to augment other process management solu-
tions:
Docker. Docker uses a package manager in Dockerfiles for
the construction of images.
Nix can be used as a replacement for conventional package
managers.
Nix can be used to fully build Docker images.
Not always not a compelling use case for Nix beginners →
they typically download prebuilt images from Docker Hub.
Docker is built around Linux technologies (e.g. namespaces)
and deploys Linux software
Sander van der Burg nix-processmgmt
8. nix-processmgmt: A general solution complementing Nix
with process management
Sander van der Burg nix-processmgmt
9. nix-processmgmt: A general solution complementing Nix
with process management
Driven by Nix and the Nix expression language
Based on simple conventions: function definitions and an
attribute set with function invocations
Similar to how packages are organized in Nixpkgs
Works with process dependencies as well: the framework
arranges the ordering, if needed
Process-manager agnostic: Integrates with sysvinit scripts,
supervisord, systemd, launchd, bsdrc scripts, cygrunsrv
Even with systems that are not qualified as process managers:
disnix, docker
Sander van der Burg nix-processmgmt
10. nix-processmgmt: A general solution complementing Nix
with process management
Operating system agnostic: Works on NixOS, but it is not a
requirement
Conventional Linux distros, macOS, FreeBSD, Cygwin
Unprivileged user installations
A switch to disable creation of users, and changing user
permissions
No advanced concepts required, such as namespaces and
cgroups
The solution relies on conflict avoidance, rather than isolation
Sander van der Burg nix-processmgmt
11. Example: a simple web application system
Sander van der Burg nix-processmgmt
12. Packaging the webapp process (sysvinit, verbose)
{createSystemVInitScript, webapp, port ? 5000}:
createSystemVInitScript {
name = "webapp";
description = "Example web application";
environment.PORT = port;
activities = {
start = ’’
log_info_msg "Starting Example web application..."
loadproc ${webapp}/bin/webapp -D
evaluate_retval
’’;
stop = ’’
log_info_msg "Stopping Example web application..."
killproc ${webapp}/bin/webapp
evaluate_retval
’’;
restart = "$0 stop; sleep 1; $0 start";
status = "statusproc ${webapp}/bin/webapp";
};
runlevels = [ 3 4 5 ];
}
Sander van der Burg nix-processmgmt
13. Packaging the webapp process (sysvinit, declarative)
{createSystemVInitScript, webapp, port ? 5000}:
createSystemVInitScript {
name = "webapp";
process = "${webapp}/bin/webapp";
args = [ "-D" ];
runlevels = [ 3 4 5 ];
environment.PORT = port;
}
We can also specify the daemon that we want to manage, instead of
the activity implementations. Most sysvinit activities (start, stop,
restart, reload, status) can be inferred.
Sander van der Burg nix-processmgmt
14. Packaging the webapp process (systemd)
The following function composes a systemd unit instead of a
sysvinit script:
{createSystemdService, webapp, port ? 5000}:
createSystemdService {
name = "webapp";
Unit = {
Description = "Example web application";
};
Service = {
ExecStart = "${webapp}/bin/webapp";
Environment.PORT = port;
Type = "simple";
};
}
The framework contains many other process manager-
specific abstraction functions: createSupervisordProgram,
createLaunchdDaemon, createBSDRCScript etc.
Sander van der Burg nix-processmgmt
15. Packaging the webapp process (agnostic)
Process manager-agnostic abstraction of the webapp service:
{createManagedProcess, webapp, port ? 5000}:
createManagedProcess {
name = "webapp";
description = "Example web application";
process = "${webapp}/bin/webapp";
daemonArgs = [ "-D" ]; # For process managers that prefer daemons
environment.PORT = port;
overrides = {
sysvinit.runlevels = [ 3 4 5 ];
};
}
Invokes the required target-specific abstraction function, e.g.
createSystemVInitScript, createSystemdService
overrides override/augment process manager-specific
parameters
You can treat foreground processes and daemons separately,
for optimal user experience
Sander van der Burg nix-processmgmt
16. Instantiatable webapp processes
{createManagedProcess, webapp}:
{instanceSuffix ? "", instanceName ? "webapp${instanceSuffix}", port ? 5000}:
createManagedProcess {
name = instanceName;
inherit instanceName; # To ensure a unique PID file name
description = "Example web application";
process = "${webapp}/bin/webapp";
daemonArgs = [ "-D" ]; # For process managers that prefer daemons
environment.PORT = port;
overrides = {
sysvinit.runlevels = [ 3 4 5 ];
};
}
instanceName: ensures that the daemon command generates
unique PID file
Outer-function header: parameters that apply to all instances.
Inner-function header: instance parameters. A unique
combination ensures that multiple instances can co-exist.
Sander van der Burg nix-processmgmt
17. Composing process instances
{ pkgs ? import <nixpkgs> {}
, stateDir ? "/var" , forceDisableUserChange ? false, processManager}:
let constructors = import ./constructors.nix {
inherit pkgs stateDir forceDisableUserChange processManager;
}; in
rec {
webapp1 = rec { # First webapp instance
port = 5000;
dnsName = "webapp1.local";
pkg = constructors.webapp {
inherit port; instanceSuffix = "1";
};
};
webapp2 = rec { # Second webapp instance
port = 5001;
dnsName = "webapp2.local";
pkg = constructors.webapp {
inherit port; instanceSuffix = "2";
};
};
...
}
Sander van der Burg nix-processmgmt
18. Composing process instances
{ pkgs ? import <nixpkgs> {}
, stateDir ? "/var" , forceDisableUserChange ? false, processManager}:
let constructors = import ./constructors.nix {
inherit pkgs stateDir forceDisableUserChange processManager;
}; in
rec {
...
# Nginx with a config that redirects users to the appropriate webapp
# instance based on the virtual host header
nginx = {
pkg = constructors.nginxReverseProxyHostBased {
# Process dependencies used to set up redirections in nginx.conf
webapps = [ webapp1 webapp2 ];
port = 8080;
} {};
};
}
Sander van der Burg nix-processmgmt
19. Demo: deploying the system and using it
Deploy the process instances as sysvinit scripts:
$ nixproc-sysvinit-switch processes.nix
Open the first webapp instance (via the Nginx reverse proxy):
$ curl -H ’Host: webapp1.local’ http://localhost:8080
Open the second webapp instance (via the Nginx reverse proxy):
$ curl -H ’Host: webapp2.local’ http://localhost:8080
Sander van der Burg nix-processmgmt
20. Demo: all kinds of process manager integrations
Deploy as systemd units (in a user session):
$ nixproc-systemd-switch --user processes.nix
Deploy as supervisord programs (stateless):
$ nixproc-supervisord-deploy-stateless processes.nix
Deploy on FreeBSD as BSD rc scripts:
$ nixproc-bsdrc-switch processes.nix
Deploy as Docker containers per service (with shared Nix store and
host networking):
$ nixproc-docker-switch processes.nix
Sander van der Burg nix-processmgmt
22. Other features of nix-processmgmt
Automatic creation of users and groups (createCredentials
function)
nixproc-id-assign: Automated assignment of unique IDs
for TCP/UDP ports, UIDs, GIDs to process instances
Using Disnix as a process orchestrator (works on all platforms
where Nix/Disnix is supported)
Sander van der Burg nix-processmgmt
23. Combining nix-processmgmt with Disnix
We can use any process manager, do distributed deployment and
combine processes with non-process based services (e.g. Java web
applications in an Apache Tomcat container):
Sander van der Burg nix-processmgmt
24. Example services packaged for nix-processmgmt
HTTP/application servers:
Apache HTTP server
Nginx
Apache Tomcat
Database:
PostgreSQL
MariaDB/MySQL
MongoDB
InfluxDB
Misc:
Docker
Supervisord
svnserve
Sander van der Burg nix-processmgmt
25. Future work
Deploy containers with multiple processes (WIP)
Mutable service containers (WIP)
s6 / s6-rc backend
Work on a test strategy for services
Maybe write an RFC?
Sander van der Burg nix-processmgmt
27. References
Blog posts:
A Nix-based functional organization for managing processes,
https://sandervanderburg.blogspot.com/2019/11/
a-nix-based-functional-organization-for.html
A declarative process manager-agnostic deployment
framework based on Nix tooling,
https://sandervanderburg.blogspot.com/2020/02/
a-declarative-process-manager-agnostic.html
Deploying container and application services with Disnix,
https://sandervanderburg.blogspot.com/2020/04/
deploying-container-and-application.html
Sander van der Burg nix-processmgmt
28. References
Blog posts:
Using Disnix as a simple and minimalistic dependency-based
process manager,
https://sandervanderburg.blogspot.com/2020/06/
using-disnix-as-simple-and-minimalistic.html
Experimenting with Nix and the service management
properties of Docker,
https://sandervanderburg.blogspot.com/2020/08/
experimenting-with-nix-and-service.html
Assigning unique IDs to services in Disnix deployment models,
https://sandervanderburg.blogspot.com/2020/09/
assigning-unique-ids-to-services-in.html
Sander van der Burg nix-processmgmt
29. References
E. Dolstra, The Purely Functional Software Deployment
Model, PhD thesis, Chapter 9,
https://edolstra.github.io/pubs/phd-thesis.pdf
Sander van der Burg nix-processmgmt