Presented by Gareth Rushgrove, Sr. Software Engineer, Puppet Labs The shipping container metaphor for Docker points to many of the advantages of building and running software using containers. But what about other essential parts of the shipping container ecosystem like the shipping manifest and bill of lading? Many of the most powerful features of traditional package management tools like apt or yum are based on metadata associated with the packages. You can find out who created a package and when, check where a particular file came from, whether the package has a known vulnerability and more. What would this capability look like for Docker containers? This talk will look at the power of metadata for containers, in particular: * Docker provides labels for associating metadata with images and containers but how best to use them?* What problems can be solved by agreeing on standards for container metadata?* Exposing standard commands and endpoints to expose metadata about what is inside a container* Demo some open source toolings and also look at the sort of tools we might build atop those standards and low-level tools.

1. Shipping Manifests, Bill of Lading and Docker
Metadata and Container
Gareth Rushgrove
Senior Software Engineer, Puppet Labs

2. @garethr

3. Senior Engineer at Puppet Labs

4. Creator of the Puppet Docker module

5. This Talk
The introduction

6. Shipping containers are cool

7. But nothing without all the
paper work

8. A manifest or ship's manifest
is a document listing the
cargo, passengers, and crew
of a ship, aircraft, or vehicle,
for the use of customs and
other officials.

9. A bill of lading is a
document issued by a carrier
which details a shipment of
merchandise and gives title
of that shipment to a
specified party.

10. State of the Software Supply Chain

11. —State of the Software
Supply Chain 2015
A once safe component
may be found to be
vulnerable at any time”
“

12. Vulnerable or defective
components… end up in
our software largely
unnoticed”
—State of the Software
Supply Chain 2015
“

13. Lets apply the same
principles to a different type
of container

14. Docker Labels
Docker builtin metadata capabilities

15. Added in 1.6
One meta data to rule them all

16. Labels on Docker
Engines

17. Provide information about the host
$ docker daemon
--label com.example.environment="production"
--label com.example.storage="ssd"

18. Labels to guide Swarm scheduling
$ docker run -d -P
-e constraint:storage==ssd --name db mysql

19. Labels on Docker
images

20. LABEL [<namespace>.]<key>[=<value>] ...
Dockerfile Label instruction

21. Don’t do this - new layer per label
LABEL vendor=ACME Incorporated
LABEL com.example.version.is-beta
LABEL com.example.version="0.0.1-beta"
LABEL com.example.release-date="2015-02-12"

22. Better - only one layer
LABEL vendor="ACME Incorporated"
com.example.is-beta
com.example.version="0.0.1-beta"
com.example.release-date="2015-02-12"

23. $ docker inspect 4fa6e0f0c678
...
"Labels": {
"vendor": "ACME Incorporated",
"com.example.is-beta": "",
"com.example.version": "0.0.1-beta",
"com.example.release-date": "2015-02-12"
}
...
Access labels via inspect

24. Containers inherit
labels from the image

25. Add additional labels to containers
$ docker run
-d
--label com.example.group="webservers"
--label com.example.environment="production"
busybox
top

26. Query based on labels
with filters

27. Filter images by label
$ docker images --filter "label=com.example.is-beta"

28. Filter containers by label
$ docker ps --filter "label=com.example.is-beta"

29. Added in v1.17
Lots of label support in the API

30. But what to store in
Labels?

31. A quick aside
Package Managers

32. Post from 2011
I like system packages

33. The power of system
packages lies not in the
file format but in the
metadata

34. DPKG and RPM

35. Debian New Maintainers’ Guide

36. Fedora packaging guidelines

37. Summary: A CD player app that rocks!
Name: cdplayer
Version: 1.0
Release: 1
Copyright: GPL
Group: Applications/Sound
Source: ftp://ftp.gnomovision.com/pub/cdplayer/cdplayer
URL: http://www.gnomovision.com/cdplayer/cdplayer.html
Distribution: WSS Linux
Vendor: White Socks Software, Inc.
Packager: Santa Claus <sclaus@northpole.com>
%description
It slices! It dices! It's a CD player app that
can't be beat. By using the resonant frequency
of the CD itself, it is able to simulate 20X
Example RPM spec file

38. Given standard
metadata what
can we do?

39. $ dpkg -L lynx
/.
/usr
/usr/share
/usr/share/doc
/usr/share/doc/lynx
/usr/share/doc/lynx/copyright
/usr/share/doc/lynx/changelog.gz
/usr/share/doc/lynx/changelog.Debian.gz
List files from package

40. What installed that file?
$ rpm -qf /usr/bin/mysqlaccess
MySQL-client-3.23.57-1

41. Find unmet dependencies
$ apt-cache unmet
Package libdataobjects-sqlite3-ruby1.9.1 version
0.10.1.1-1 has an unmet dep:
Depends: libdataobjects-ruby1.9

42. $ rpm -qdf /usr/bin/mysqlaccess
/usr/share/man/man1/mysql.1.gz
/usr/share/man/man1/mysqlaccess.1.gz
/usr/share/man/man1/mysqladmin.1.gz
/usr/share/man/man1/mysqldump.1.gz
/usr/share/man/man1/mysqlshow.1.gz
Find documentation

43. $ apticron
The following packages are currently pending an
upgrade:
xfree86-common 4.3.0.dfsg.1-14sarge3
libice6 4.3.0.dfsg.1-14sarge3
libsm6 4.3.0.dfsg.1-14sarge3
xlibs-data 4.3.0.dfsg.1-14sarge3
libx11-6 4.3.0.dfsg.1-14sarge3
libxext6 4.3.0.dfsg.1-14sarge3
libxpm4 4.3.0.dfsg.1-14sarge3Find outdated packages

44. Standards
The power of agreement

45. Docker official label guidance

46. All (third-party) tools should
prefix their keys with the
reverse DNS notation of a
domain controlled by the
author. For example,
com.example.some-label.
1

47. The com.docker.*, io.docker.*
and org.dockerproject.*
namespaces are reserved for
Docker’s internal use. 2

48. Keys should only consist of
lower-cased alphanumeric
characters, dots and dashes
(for example, [a-z0-9-.]). 3

49. Keys should start and end
with an alpha numeric
character.
4

50. Keys may not contain
consecutive dots or dashes.
5

51. Keys without namespace
(dots) are reserved for CLI use.
6

52. How widely adhered to?

53. < 20% from a small sample

54. But some folks care

55. Unify the labels for Docker images

56. Merged

57. The problem with
inconsistent metadata

58. Without complete
metadata we can’t trust
the tools built on top

59. Docker Label Inspector

60. Check against Docker guidelines
$ dli lint
========> Check all labels have namespaces
[WARN] Label 'vendor' should use a namespace based
on reverse DNS notation
========> Check labels don't use reserved namespaces
========> Check labels only use valid characters
========> Check labels start and end with alpanumeric
characters
========> Check labels for double dots and dashes

61. $ dli validate
========> Check labels based on schema in 'schema.json'
[ERROR] u'com.example.is-beta' is a required property
Check against a schema

62. {
"title": "Dockerfile schema",
"type": "object",
"properties": {
"com.example.release-date": {
"type": "string"
},
"com.example.is-beta": {
"type": "string"
},
"com.example.version": {
"description": "Version",
"type": "integer",
"minimum": 0
}
},
"required": ["com.example.is-beta",
"com.example.version"]
}
Define labels in JSON schema

63. Stand well back
LIVE DEMO

64. Runtime Metadata
A missing piece, and some ideas

65. What temperature is a
refrigerated shipping
containers at?
*Not all shipping containers are refrigerated

66. The need for runtime
metadata

67. docker exec as an API

68. Dockerfile example
FROM alpine
LABEL net.morethanseven.dockerfile="/Dockerfile"
net.morethanseven.exec.packages="apk info -vv"
RUN apk add --update bash && rm -rf /var/cache/apk/*
COPY Dockerfile /

69. Discover out API
$ docker inspect -f "{{json .Config.Labels }}"
garethr/alpine
| jq
{
"net.morethanseven.dockerfile": "/Dockerfile",
"net.morethanseven.exec.packages": "apk info -vv"
}

70. Read the Dockerfile
$ docker run -i -t garethr/alpine cat /Dockerfile
FROM alpine
LABEL net.morethanseven.dockerfile="/Dockerfile"
net.morethanseven.exec.packages="apk info -vv"
RUN apk add --update bash && rm -rf /var/cache/apk/*
COPY Dockerfile /

71. $ docker run -i -t garethr/alpine apk info -vv
musl-1.1.11-r2 - the musl c library (libc) implementati
busybox-1.23.2-r0 - Size optimized toolbox of many comm
alpine-baselayout-2.3.2-r0 - Alpine base dir structure
openrc-0.15.1-r3 - OpenRC manages the services, startup
alpine-conf-3.2.1-r6 - Alpine configuration management
List installed packages

72. Second time lucky
LIVE DEMO

73. R.I.Pienaar

74. Tooling
What could we build atop our metadata?

75. Documentation
discovery

76. License verification

77. Package search

78. Tempting fate
LIVE DEMO

79. Conclusions
If all you remember is…

80. Step 1
Step 2
Step 3
Metadata!
Something…
Profit

81. Share schemas and
namespaces

82. Build agreement

83. Build tooling

84. Extract standards

85. Thank you!
Gareth Rushgrove
@garethr
gareth@puppetlabs.com