CNDT 2021にて、eStargzを用いたlazy pullingによる高速なコンテナ起動について発表しました (2021年11月5日)。 https://event.cloudnativedays.jp/cndt2021/talks/1250

1. Copyright(c)2021 NTT Corp. All Rights Reserved
eStargz lazy pulling
2021/11/5
CloudNative Days Tokyo 2021

2. Copyright(c)2021 NTT Corp. All Rights Reserved
GitHub:@ktock / Twitter:@TokunagaKohei
containerd BuildKit
Container Runtime Meetup

3. Copyright(c)2021 NTT Corp. All Rights Reserved
TL;DR
l Pull 3
l eStargz lazy pulling pull
• https://github.com/containerd/stargz-snapshotter
l eStargz Lazy pulling
• Kubernetes k3s KinD containerd nerdctl Podman CRI-O BuildKit Kaniko
go-containerregistry ko buildpacks.io
l eStargz OCI Image Specification

4. Copyright(c)2021 NTT Corp. All Rights Reserved
pull
pulling packages accounts for 76% of container start time,
but only 6.4% of that data is read [Harter et al. 2016]
[Harter et al. 2016] Tyler Harter, Brandon Salmon, Rose Liu, Andrea C. Arpaci-Dusseau, Remzi H. Arpaci-Dusseau. "Slacker: Fast Distribution with
Lazy Docker Containers". 14th USENIX Conference on File and Storage Technologies (FAST ’16). February 22–25, 2016, Santa Clara, CA, USA
l pull
l cold start pull
l
• e.g. 4

5. Copyright(c)2021 NTT Corp. All Rights Reserved
OCI
l pull
l tar+{gzip,zstd} 5
l

6. Copyright(c)2021 NTT Corp. All Rights Reserved
eStargz lazy pulling
Standard Container Registry
eStargz-aware runtimes
BuildKit
Lazy pull
Lazy pull
legacy
ctr-remote
nerdctl
BuildKit
Kaniko
Image builders/convertes
eStargz
creates
eStargz
still runnable on
eStargz-agnostic tools (e.g. Docker)
etc…
etc…
legacy
l OCI lazy pulling
• Pull
• 6
• 6
l Lazy pulling OCI
l BuildKit Kaniko nerdctl eStargz
https://github.com/containerd/stargz-snapshotter

7. Copyright(c)2021 NTT Corp. All Rights Reserved
eStargz
eStargz
bin/ls
usr/bin/apt
entrypoint.sh
bin/bash
Range Request
TOC(Table Of Contents) footer
/
gzip
HTTP Range Request
/
stargz
bin/ls
usr/bin/apt
entrypoint.sh
bin/bash
For more details: https://github.com/containerd/stargz-snapshotter/blob/master/docs/stargz-estargz.md
l
l OCI
l CRFS stargz
l 7

8. Copyright(c)2021 NTT Corp. All Rights Reserved
README https://github.com/containerd/stargz-snapshotter
Lower is better

9. Copyright(c)2021 NTT Corp. All Rights Reserved
lazy pulling

10. Copyright(c)2021 NTT Corp. All Rights Reserved
stargz-snapshotter 1 0
https://github.com/containerd/stargz-snapshotter
Kubernetes
Containerd
Stargz
Snapshotter
external process
gRPC API
kubelet
Lazy pull
Container Registry
eStargz
CRI-O
Stargz Store
external process
exposed on dir
kubelet
l CRI 0 lazy pulling
• Containerd Stargz Snapshotter
• CRI-O Stargz Store

11. Copyright(c)2021 NTT Corp. All Rights Reserved
K3s
l 1 Kubernetes
l containerd CRI
l v1.22.2 Stargz Snapshotter lazy pulling
https://rancher.com/docs/k3s/latest/en/advanced/#enabling-lazy-pulling-of-estargz-experimental
k3s server --snapshotter=stargz
K3s server/agent
Stargz
Snapshotter (library)
k8s
Lazy pull
Container Registry
eStargz
k3d cluster create --image=rancher/k3s:v1.22.2-k3s2 --k3s-arg='--snapshotter=stargz@server:*;agent:*'
l K3d (k3s in Docker) lazy pulling
builtin
https://github.com/k3s-io/k3s

12. Copyright(c)2021 NTT Corp. All Rights Reserved
KinD
l Kubernetes in Docker Kubernetes
l 1CRI containerd
l Stargz Snapshotter lazy pulling
• ghcr.io/stargz-containers/estargz-kind-node:0.9.0
• KinD 21 :https://github.com/kubernetes-
sigs/kind/pull/2076
$ kind create cluster --name estargz-demo --image ghcr.io/stargz-containers/estargz-kind-node:0.9.0
Stargz
Snapshotter
kubelet
Lazy pull
Container Registry
eStargz
https://github.com/kubernetes-sigs/kind

13. Copyright(c)2021 NTT Corp. All Rights Reserved
containerd nerdctl
proc
Stargz
Snapshotter
Fetches files/chunks on demand
Mounts rootfs as FUSE
Node
Lazy pull
Container Registry
eStargz
container
l Containerd Stargz Snapshotter3 lazy pulling
• Containerd remote snapshotter 3
l nerdctl Docker containerd CLI Stargz Snapshotter lazy pulling 1
nerdctl --snapshotter=stargz run ghcr.io/stargz-containers/python:3.9-esgz
nerdctl --snapshotter=stargz compose -f docker-compose.stargz.yaml up
https://github.com/containerd/containerd
https://github.com/containerd/nerdctl

14. Copyright(c)2021 NTT Corp. All Rights Reserved
Podman CRI-O
proc
container
Stargz Store
Provides layers
(mounted as FUSE)
Podman, CRI-O
Uses layers for rootfs
Node
Lazy pull
Container Registry
eStargz
l Stargz Store lazy pulling
l systemd service
yum install -y stargz-snaphsotter
systemctl enable --now stargz-store
podman run ghcr.io/stargz-containers/python:3.9-esgz
l Additional Layer Store Podman >= v3.3.0, CRI-O >= v1.22.0
• Stargz Store Podman/CRI-O FUSE 4 1
https://github.com/cri-o/cri-o
https://github.com/containers/podman
Additional Layer Store

15. Copyright(c)2021 NTT Corp. All Rights Reserved
BuildKit
FROM ghcr.io/stargz-containers/golang:1.15.3-esgz as dev
COPY ./hello.go /hello.go
RUN go build -o hello /hello.go
COPY and RUN without waiting for
the pull completion
• /usr/local/go/bin/go
• /usr/local/go/src/fmt/…
etc...
Fetch files/chunks on demand
Build on node
Lazy pull
Container Registry
eStargz
golang:1.15.3-esgz
l 5 lazy pulling1 v0.8.0
l 5 1pull 5
• 5
$ docker buildx create --use --name lazy-builder ¥
--driver docker-container --driver-opt image=moby/buildkit:master ¥
--buildkitd-flags '--oci-worker-snapshotter=stargz'
$ docker buildx inspect --bootstrap lazy-builder
$ docker buildx build .
https://github.com/moby/buildkit

16. Copyright(c)2021 NTT Corp. All Rights Reserved
eStargz

17. Copyright(c)2021 NTT Corp. All Rights Reserved
eStargz
l Kaniko : https://github.com/GoogleContainerTools/kaniko
• Kubernetes
• GGCR_EXPERIMENT_ESTARGZ=17
• eStargz
$ docker run --rm -e GGCR_EXPERIMENT_ESTARGZ=1 ¥
-v /tmp/context:/workspace -v ~/.docker/config.json:/kaniko/.docker/config.json:ro ¥
gcr.io/kaniko-project/executor:v1.6.0 --destination "ghcr.io/ktock/sample:esgz"
$ docker buildx build ¥
-o type=registry,name=ktokunaga/hello:esgz,oci-mediatypes=true,compression=estargz .
l BuildKit : https://github.com/moby/buildkit
• eStargz7compression type1 master
• Docker Buildx
https://github.com/moby/buildkit/blob/master/docs/stargz-estargz.md#creating-
stargzestargz-images

18. Copyright(c)2021 NTT Corp. All Rights Reserved
eStargz
l nerdctl : https://github.com/containerd/nerdctl
• Docker 1containerd CLI
• 1 eStargz
• nerdctl build 1eStargz
l ctr-remote : https://github.com/containerd/stargz-snapshotter
• Stargz Snapshotter containerd CLI
• 1 eStargz
• 1
l go-containerregistry : https://github.com/google/go-containerregistry
• Kaniko, ko, buildpacks.io 8
• GGCR_EXPERIMENT_ESTARGZ=1 eStargz
• ggcr 1 1 eStargz
$ nerdctl image convert --estargz --oci ghcr.io/ktock/foo:1 ghcr.io/ktock/foo:1-esgz
$ go install github.com/google/go-containerregistry/cmd/crane@latest
$ GGCR_EXPERIMENT_ESTARGZ=1 crane optimize –-insecure ghcr.io/ktock/foo:1 ghcr.io/ktock/foo:1-esgz
ctr-remote image optimize --oci ghcr.io/ktock/foo:1 ghcr.io/ktock/foo:1-esgz

19. Copyright(c)2021 NTT Corp. All Rights Reserved
eStargz 2021/10
*1 eStargz OCI OCI eStargz
*2 GGCR_EXPERIMENT_ESTARGZ=1
*3 Stargz Snapshotter ghcr.io/stargz-containers/estargz-kind-node lazy pulling

20. Copyright(c)2021 NTT Corp. All Rights Reserved
https://github.com/opencontainers/image-spec/issues/815
l OCI Image Specification
• gzip 2 application/vnd.oci.image.layer.v1.tar+gz
• 0 Annotation org.opencontainers.image.toc.digest

21. Copyright(c)2021 NTT Corp. All Rights Reserved
Join our discussion!
l Subproject (non-core) of CNCF graduated containerd
l github https://github.com/containerd/stargz-snapshotter
l slack https://slack.containerd.io/ (#containerd-dev channel at CNCF slack)
l Pre-converted images are available on ghcr.io/stargz-containers
l Pull 2
l eStargz 2 lazy pulling2 pull 1
• https://github.com/containerd/stargz-snapshotter
l 1 eStargz Lazy pulling
• Kubernetes k3s KinD containerd nerdctl Podman CRI-O BuildKit Kaniko
go-containerregistry ko buildpacks.io
l eStargz OCI Image Specification 1