Presentation - Brandon Mitchell

name: empty
layout: true
---
name: base
layout: true
template: empty
background-image: none
<div class="slide-footer">@sudo_bmitch</div>
---
name: ttitle
layout: true
template: empty
class: center, middle
background-image: url(img/containers_bg.png)
background-size: cover
---
name: inverse
layout: true
template: base
class: center, middle, inverse
background-image: none
---
name: impact
layout: true
template: base
class: middle, impact
background-image: url(img/containers_bg.png)
background-size: cover
---
name: picture
layout: true
template: base
class: center, middle
background-image: none
---
name: code
layout: true
template: base
class: terminal
background-image: none
---
name: terminal
layout: true
template: base
class: center, middle, terminal
background-image: none
---
name: default
layout: true
template: base
class: bg-blur
background-image: url(img/containers_bg.png)
background-size: cover
---
layout: false
template: default
name: agenda

# Agenda

.left-column[
- [Docker Intro](#docker-intro)
- [Build w/o Docker](#build-wo-docker)
- [Debian Base](#debian)
- [Alpine Base](#alpine)
- [Multi-stage](#multi-stage)
- [Dev Workflow](#dev-workflow)
]
.right-column[
- [buildkit](#buildkit)
- [Experimental](#experimental)
- [Cross Platform](#cross-platform)
- [buildx](#buildx)
]

---
layout: false
template: ttitle
name: title

# Building Docker Images

.left-column[
.pic-circle-70[![Brandon Mitchell](img/bmitch.jpg)]
]
.right-column[.align-right[.no-bullets[
<br><br>
- Brandon Mitchell
- Twitter: @sudo_bmitch
- GitHub: sudo-bmitch
]]]

???

- My twitter and github handles are what any self respecting sysadmin does
  when you get a permission denied error on your favorite username.
- This presentation is on github and I'll have a link to it at the end.

---

```no-highlight
$ whoami
- Solutions Architect @ BoxBoat
- Docker Captain
- Frequenter of StackOverflow
```

.align-center[
.pic-30[![BoxBoat](img/boxboat-logo-color.png)]
.pic-30[![Docker Captain](img/docker-captain.png)]
.pic-30[![StackOverflow](img/stackoverflow-logo.png)]
]

???

---

# What is Docker?

???

- First a quick level set for anyone that doesn't already know Docker
- Who's already using Docker?

---

# What is Docker?

- App vs OS isolation
  - Shared kernel vs shared hardware
- Kernel namespaces
  - Mount, PID, Network, User, UTS, IPC
- Package and Ship Images
  - Filesystem + metadata
- Run Images as Containers
  - Running process with some settings

???

- Most namespaces (except for filesystem) can be adjusted
- Mount for filesystems
- UTS is for the hostname
- IPC for inter process comm

---

# Image Filesystem Layers

- Filesystems are packaged in layers, diff on previous filesystem state
- Layers are read-only once created
- Images may share layers
- Containers merge the read-only image layers plus a read-write container layer
- Write to a container filesystem performs a copy-on-write to read-write layer

---

# Dockerfile

- List of instructions to create an image
- Each instruction adds a new layer or sets some metadata
- Start from a known initial state, base image

???

- Starting from a known state is why I like containers vs other CM tools
- Avoids state drift when you always begin at a known state

---

# Where's the Go Code?

???

- This is a Go meetup, let's talk Go

---

# Hello World

```go
package main

import (
	"flag"
	"fmt"
	"html/template"
	log "github.com/sirupsen/logrus"
	"net/http"
	"os"
	"runtime"
)

var (
	httpAddr   *string
	serverName *string
)
```
???
- Sadly, I'm here to learn Go more than to teach it, so you get a hello world

---

# Hello World

```go
func init() {
	hostname, _ := os.Hostname()
	httpAddr = flag.String("http", ":8080", "Listen address")
	serverName = flag.String("server", hostname, "Server Name")
}

func main() {
	flag.Parse()
	http.HandleFunc("/", root)
	fmt.Print("Ready to receive requests on port 8080\n")
	log.Fatal(http.ListenAndServe(*httpAddr, nil))
}

func root(w http.ResponseWriter, r *http.Request) {
	path := "unknown"
	if (len(r.URL.Path) > 1) {
		path = r.URL.Path[1:]
	}

```
???
- I've got an app listening on port 8080 for http requests
- I take the URL from those requests
---

# Hello World

```go
	data := struct {
		ServerName, Path, OS, Arch string
	}{ *serverName, path, runtime.GOOS, runtime.GOARCH, }
	w.Header().Set("Server", "Hello Server")
	w.WriteHeader(200)
	err := tmpl.Execute(w, data)
	if (err != nil) {
		log.Print(err)
	}
}

var tmpl = template.Must(template.New("tmpl").Parse(`
<!DOCTYPE html><html><body><center>
	<h2 style="color:blue">Hello {{.Path}} from {{.ServerName}}</h2>
	<h3>Running on OS: {{.OS}}, Arch: {{.Arch}}</h3>
</center></body></html>
`))
```

???
And respond with a template that includes
- URL
- server name (hostname)
- OS
- Arch

---

# Lets Package This In Docker

???
- So that was a huge let down to the Go devs
- Maybe I can show you some cool docker stuff instead
---

# Dockerfile: External Build

```no-highlight
FROM debian:latest
COPY app /app
RUN chmod 755 /app
CMD /app
```
???
- Here's a first attempt at a Dockerfile
- It assumes we build the app outside of docker
- The instructions are fairly easy to read
  - FROM defines a parent image, all images need a starting point
  - COPY takes a file/dir from the context (build input) and adds it to our image
  - RUN executes a command inside of a temporary container and saves the changes to the filesystem
  - CMD is metadata saying how to run this image as a container (default)
---

<asciinema-player src="demo-external.cast" cols=100 rows=26 preload=true font-size=16></asciinema-player>

---

# Compile with Docker

???
- If docker does the compile, we can avoid dependencies on developer machines
- This also means one app could use a newer version of Go than another, on the same machine

---

<asciinema-player src="demo-byhand.cast" cols=100 rows=26 preload=true font-size=16></asciinema-player>

???
- Before I create an image, I like to run commands by hand interactively
- Makes for faster debugging

---

# Compile During Build

???
- Lets get to the real world, no one is compiling by hand on the CI server for every check-in
- Lets take our steps from the by-hand install and make a Dockerfile that does the compile

---

# Dockerfile: Debian v1

```no-highlight
FROM golang:1.12
RUN adduser --disabled-password --gecos appuser appuser
WORKDIR /src
COPY . /src/
RUN go build -o app .
WORKDIR /
RUN cp /src/app /app
RUN chown appuser /app
RUN chmod 755 /app
RUN rm -r /src
USER appuser
CMD /app
```
???
- Here's almost a literal translation to our by-hand steps
- Ops jumped in and required that we not run stuff as root
- But before you take notes on this, realize it's version 1 and a horrible example

---

<asciinema-player src="demo-debian1.cast" cols=100 rows=26 preload=true font-size=16></asciinema-player>
???
- Before I get to bad mouthing it too much, lets see that it works

---

# Dockerfile: Debian v1

```no-highlight
FROM golang:1.12
RUN adduser --disabled-password --gecos appuser appuser
WORKDIR /src
COPY . /src/
RUN go build -o app .
WORKDIR /
RUN cp /src/app /app
RUN chown appuser /app
RUN chmod 755 /app
RUN rm -r /src
USER appuser
CMD /app
```
???
- What's wrong with it?
- We don't cleanup all the go cache
- Once RUN finishes, those files are in the image and shipped

---

# Dockerfile: Debian v2

```no-highlight
FROM golang:1.12
RUN adduser --disabled-password --gecos appuser appuser
COPY . /src/
*RUN  cd /src \
* && go build -o app . \
* && cd / \
* && cp /src/app /app \
* && chown appuser /app \
* && chmod 755 /app \
* && rm -r /go/pkg /root/.cache/go-build /src
USER appuser
*CMD [ "/app" ]
```
???
- So the fix is we chain our RUN commands, 
  delete files in the same step where they are created
- This doesn't help with the COPY of /src
- I also switched CMD to have json syntax, with a string there's an extra shell
  run that we don't need, it would intercept and block SIGTERM

---

<asciinema-player src="demo-debian2.cast" cols=100 rows=26 preload=true font-size=16></asciinema-player>
???
- So that works
- And we shrink our image
- We aren't anywhere near the size of external image, because now we ship a compiler
- But first, there's an alpine image that's smaller, what if we used that?

---

# What About Alpine

---

# Dockerfile: Alpine v1

```no-highlight
*FROM golang:1.12-alpine
*RUN adduser -D appuser
COPY . /src/
RUN cd /src \
 && go build -o app . \
 && cd / \
 && cp /src/app /app \
 && chown appuser /app \
 && chmod 755 /app \
 && rm -r /go/pkg /root/.cache/go-build /src
USER appuser
CMD [ "/app" ]
```
???
- Not much to change with Alpine, just a different FROM,
  and the adduser syntax changes
---

<asciinema-player src="demo-alpine1.cast" cols=100 rows=26 preload=true font-size=16></asciinema-player>

???
- But when we run that, we don't have git
- Alpine is smaller because it doesn't include lots of stuff

---

# Dockerfile: Alpine v2

```no-highlight
FROM golang:1.12-alpine
*RUN apk add --no-cache git ca-certificates
RUN adduser -D appuser
COPY . /src/
RUN cd /src \
 && go build -o app . \
 && cd / \
 && cp /src/app /app \
 && chown appuser /app \
 && chmod 755 /app \
 && rm -r /go/pkg /root/.cache/go-build /src
USER appuser
CMD /app
```
???
- So lets install git, and you may also need other packages
  like certificates if you pull from HTTPS repos

---

<asciinema-player src="demo-alpine2.cast" cols=100 rows=26 preload=true font-size=16></asciinema-player>

???
- So that works
- And again, image size goes down, almost all from the smaller base.
- If you do the math, our code in debian2 (on top of 1.12) is more than alpine2
  (on top of 1.12-alpine) because we needed to install git
- But we are still shipping the compiler
---

# Do we need to ship a compiler with our app?

---

# No

???
- I wouldn't ask the question if the answer wasn't "No"
- Even with languages like Java, we could change from a JDK to a JRE

---

# Dockerfile: Multi-stage v1

```no-highlight
*FROM golang:1.12-alpine as build
RUN apk add --no-cache git ca-certificates
RUN adduser -D appuser
WORKDIR /src
COPY . /src/
RUN go build -o app .
USER appuser
CMD [ "/src/app" ]

*FROM scratch as release
*COPY --from=build /src/app /app
USER appuser
CMD [ "/app" ]
```
???
- Multi-stage images have multiple "FROM" lines
- This builds multiple images, and tags one of them (by default, the last one)
  - It does not merge multiple images together
- The key is we can copy files from one stage to another
- Our CI server now just needs the docker command, devs select their compiler
  image in the first stage, and minimal runtime in the last stage
- And for my runtime filesystem, I picked scratch, scrach is what you get when
  `rm -rf /` finishes
- There's no shell or libraries to run, so I need that CMD in json, and I need
  a language like Go that can statically compile the binary
---

<asciinema-player src="demo-multi-stage1.cast" cols=100 rows=26 preload=true font-size=16></asciinema-player>
???
- So lets run that... and it fails because the user doesn't exist in scratch
- We only created the user in our build stage, lets fix that in a sysadmin way

---

# Dockerfile: Multi-stage v2

```no-highlight
FROM golang:1.12-alpine as build
RUN apk add --no-cache git ca-certificates
RUN adduser -D appuser
WORKDIR /src
COPY . /src/
RUN go build -o app .
USER appuser
CMD [ "/src/app" ]

FROM scratch as release
*COPY --from=build /etc/passwd /etc/group /etc/
COPY --from=build /src/app /app
USER appuser
CMD [ "/app" ]
```
???
- Copy the /etc/passwd and /etc/group files, which contain all the
  user and group mappings to UID/GID's
- I could also copy those Certificates from /etc/ssl if they were needed
---

<asciinema-player src="demo-multi-stage2.cast" cols=100 rows=26 preload=true font-size=16></asciinema-player>
???
- Lets run that... and we get fail number 2
- Lets check the logs... wait, where's the container?
- We ran it with `--rm -d` so the container deletes in the background on exit, lets foreground it
- No such file or directory? Lets check the container and see why it's missing.
- And we don't have a /bin/sh in scratch, lets build a new image to debug

---

# Dockerfile: Multi-stage v3

```no-highlight
*FROM golang:1.12-alpine as dev
RUN apk add --no-cache git ca-certificates
RUN adduser -D appuser
WORKDIR /src
COPY . /src/
CMD go build -o app . && ./app

*FROM dev as build
RUN go build -o app .
USER appuser
CMD [ "./app" ]

...
```
???
- So first, this will be relevant later, I'm splitting the developer
  and build stages, dev has the source, build generates the binary
---

# Dockerfile: Multi-stage v3

```no-highlight
...

FROM scratch as release
COPY --from=build /etc/passwd /etc/group /etc/
COPY --from=build /src/app /app
USER appuser
CMD [ "/app" ]

*FROM debian as debug
*COPY --from=build /src/app /app
*CMD [ "/app" ]

FROM release
```
???
- More importantly, I'm adding a debug stage, using debian instead of scratch
- And I copy the /app just like I do in release
---

<asciinema-player src="demo-multi-stage3.cast" cols=100 rows=26 preload=true font-size=16></asciinema-player>
???
- Lets run that... and woops, a shell needs input or it will exit...
- See the "no such file or directory" here too, but the file exists, and is executable
- If you google this, you'll probably find a SO post that I've answered saying
  to check the linked libraries, and we see the broken link to libc.musl
- Note, had I picked alpine for my debug image, app would have run just fine

---

# Dockerfile: Multi-stage v4

```no-highlight
FROM golang:1.12-alpine as dev
RUN apk add --no-cache git ca-certificates
RUN adduser -D appuser
WORKDIR /src
COPY . /src/
*CMD CGO_ENABLED=0 go build -ldflags '-s -w -extldflags -static' -o app . && ./app

FROM dev as build
*RUN CGO_ENABLED=0 go build -ldflags '-s -w -extldflags -static' -o app .
USER appuser
CMD [ "./app" ]

...
```
???
- The fix for this, that you'd see from the same SO post, is to turn off CGO
  when you use networking libraries
---

<asciinema-player src="demo-multi-stage4.cast" cols=100 rows=26 preload=true font-size=16></asciinema-player>
???
- Lets run that, and this time it works
- So that was a pain in the butt...
- But our resulting image is a lot smaller, down to 10M, that will speed up the deploys
---

# Developer Workflow

???
- Remember me splitting the dev and build stages, lets use that to make a better
  developer workflow, because these builds take time downloading modules
---

<asciinema-player src="demo-dev-workflow.cast" cols=100 rows=26 preload=true font-size=16></asciinema-player>
???
- The dev can run build the dev target to have the full compiler
- But this time, when running it, I add a bunch of options
- Volumes for /src and HOME map those directories into the container
- I'm setting the GOPATH and HOME variables so the go cache gets saved somewhere persistent
- And I adjust the UID/GID to match my user outside of the container
- This runs a build every time we start up
- And to rerun a build, we can just restart the container
---

# How I Run Go

```shell
$ cat ~/bin/go
#!/bin/sh

GO_VER=${GO_VER:-1.12}
GO_ORIG=/usr/local/go/bin/go

docker run -it --rm --net host \
  -u "$(id -u):$(id -g)" -v "$HOME:$HOME" \
  -e "GOPATH=${HOME}/data/golang" \
  -e CGO_ENABLED \
  -e HOME -w "$(pwd)" \
  golang:${GO_VER} ${GO_ORIG} "$@"
```
???
- All of those variables come from a script that I use for `go`
- I don't even have go installed directly on my laptop, everything is in containers

---

# Users Want More

---

# Classic Builder Problems

- Users want new features
- Performance
- Cache Pruning
- Kubernetes support

???

- Multi-stage opened the flood gates on feature requests
- Changing the Dockerfile is a slow/painful process
  - All build engines need to be updated, long release cycles
- Performance issues are an issue with CI
  - Every build with a minor dep change downloads all deps again
- Cache pruning is all or nothing or based on build date
- K8s users want something k8s native without docker engine

---

![XKCD - Compiling](img/xkcd.303.compiling.png)
???
- Sure slow builds are good for office shenanigans
- But we can speed them up

---

# buildkit

???

- Standalone "kit"
- Uses containerd or runc as backend instead of docker engine
- Integrated back into docker similar to swarmkit

---

# buildkit Design

- LLB parser and dependency graph
- Efficient build context
- Parallel build
- Garbage collection
- Multiple Output Types

???

- Dockerfile parsed into Low Level Builder 
- Parser is swappable with a parser directive on the first line
- Target stage is selected and deps are selected based on a dep graph
- Bi-drectional comm with build client to only pull parts of context
- Previous context is cached and rdiff used to pull in changes
- Each stage is built in parallel
- GC can remove unused layers and preserve based on time and size
- Output can be tar, registry push, write to local dir

---

# Standalone Deployment

- Run `buildkitd` as root
- Rootless
- Container
- Kubernetes
- Client is `buildctl`

???

- Listens on a local socket by default
- Can also be configured to connect over TCP with mTLS
- Rootless requires:
  - sysctl: `kernel.unprivileged_userns_clone=1`
  - `modprobe overlay permit_mounts_in_userns=1`
- Env var BUILDKIT_HOST is set to a file, tls, docker-container, or kube-pod

---

<asciinema-player src="demo-buildkit1.cast" cols=100 rows=26 preload=true font-size=16></asciinema-player>

???

- Download the buildkit standalone files from github and extract for buildctl
- Run buildkitd as a container and set the env var to point to the container
- Run buildctl to access buildkitd in the container with lots of args
- Standalone buildctl doesn't output to docker (possible there isn't docker
  here), so the output is a tar that we load with `docker load`
- The result runs as expected

---

# Enable buildkit in Docker

In a shell:

```no-highlight
$ export DOCKER_BUILDKIT=1
```

Host default in `/etc/docker/daemon.json`:

```json
{ "features": {"buildkit": true} }
```

Compose:

```shell
$ export COMPOSE_DOCKER_CLI_BUILD=1
```

???
- Integrated into docker engine + CLI 18.09 or newer
- One user can update their shell to test
- Once you like buildkit, change the host default for all users
- Compose 1.25.0 or newer supports buildkit by calling cli builder

---

<asciinema-player src="demo-buildkit2.cast" cols=100 rows=26 preload=true font-size=16></asciinema-player>

???

- Lets run the same Dockerfile previously used in multi-stage examples
- All we do is set an env var to enable buildkit
- Output is same as buildctl, diff from classic docker build
- Result creates a docker image by default, and runs as expected
- Key here is setting a variable is all that's needed to transition

---

# Garbage Collection

In `/etc/docker/daemon.json`:

```json
  "builder": {
    "gc": {
      "enabled": true,
      "policy": [
*       {"keepStorage": "512MB", "filter": ["unused-for=1400h"]]},
*       {"keepStorage": "30GB", "all": true}
      ]
    }
  }
```

???

- You can specify one or more policies
- These two keep 512MB after 58 days and 30GB before that
- GC runs after each build completes (with a 1/sec? limit)

---

# Try This At Home

???

- Lets talk about some of the experimental features

---

# buildkit Experimental Features

- Cache folders
- Bind mount from context or image
- Tmpfs
- Secrets
- SSH creds

???

- Each of these creates a bind mount into a specific build step
- The contents of that bind mount are not included in the resulting image
- Cache: debian/ubuntu packages, go modules, etc
- Bind from context: processing a large data file
- Bind from image: running aqua microscanner
- Tmpfs: faster tmp files
- Secrets: AWS or SSH creds
- SSH: when your SSH creds are password protected

---

# Dockerfile: buildkit v3 (partial)

```no-highlight
*# syntax=docker/dockerfile:experimental

FROM golang:1.12-alpine as dev
COPY . /src/

FROM dev as build
*RUN --mount=type=cache,sharing=locked,id=gomod,target=/go/pkg/mod/cache \
*   --mount=type=cache,sharing=locked,id=goroot,target=/root/.cache/go-build \
*   CGO_ENABLED=0 go build -ldflags '-s -w -extldflags -static' -o app .
CMD ./app

FROM scratch as release
COPY --from=build /etc/passwd /etc/group /etc/
COPY --from=build /src/app /app
CMD [ "/app" ]
...
FROM release
```
???
- Here's a buildkit modified Dockerfile for speeding up a go build
- First line changes the frontend parser to enable experimental features
- The `--mount` option mounts effectively a named volume for that RUN line
- Directory persists between builds, does not get added to the image
---

<asciinema-player src="demo-buildkit3.cast" cols=100 rows=26 preload=true font-size=16></asciinema-player>

???
- Important note, that `--mount` option is not backwards compatible
- buildkit skips the debug stage, not needed for release stage
- Good for efficiency, bad for testing, need to inject a hard dependency
  or manually build specific stages
- Still works, but we need a comparison to see if it's better

---

<asciinema-player src="demo-buildkit4.cast" cols=100 rows=26 preload=true font-size=16></asciinema-player>

???

- I've populated the cache in both multi-stage and buildkit
- Change a single version number in the go.mod
- Rerunning both shows buildkit skips the download on most modules
  - Barely see the one change getting downloaded with a decent network

---

# buildkit Output

- Docker image
- Registry image
- Tar
- Local directory

???

- By default, we almost always create a docker image
- buildkit can push directly to a registry (may need creds)
- If we send to a tar, we can directly output OCI images
- We can also output the image filesystem to a local directory
  - extract artifacts
  - extract testing reports/logs
  - generate grpc proto bufs
  - attach debugger from host to extracted binary

---

# Dockerfile: buildkit v5 (partial)

```no-highlight
...
FROM scratch as release
COPY --from=build /etc/passwd /etc/group /etc/
COPY --from=build /src/app /app
USER appuser
CMD [ "/app" ]

FROM debian as debug
COPY --from=build /src/app /app
CMD [ "/app" ]

*FROM scratch as artifact
*COPY --from=build /src/app /app

FROM release
```
???
- Use scratch for my artifact layer so only the specifically copied files are extracted

---

<asciinema-player src="demo-buildkit5.cast" cols=100 rows=26 preload=true font-size=16></asciinema-player>

???

- Now if I start without an "app" binary in the directory
- Run a build with the `--output` option
- Possible use cases
  - extract artifacts
  - extract testing reports/logs
  - generate grpc proto bufs
  - attach debugger from host to extracted binary

---

# Cross Platform Builds

---

# Cross Platform Build Options

Three options:

1. Qemu with binfmt_misc
2. Cross compiling
3. Multiple build nodes

???

- Qemu is virtualization loaded into the kernel to run binaries for other platforms
  - Requires binfmt_misc module
- Cross compiling works great with Go
- Multiple build nodes available with buildx

---

# Setup binfmt_misc

- User mode emulation binaries
- Static version
- Installed with `--fix-binary`
- Verify `F` flag in `/proc/sys/fs/binfmt_misc/*`

???

- Part of qemu's processor emulation
- Allows you to run binaries for other CPU's
  - x86, ARM, MIPS, SPARC, PowerPC, etc
- Static version means it's a single binary
- The `--fix-binary` flag pins to the mount namespace so it works in containers
- This is already done in Docker Desktop
- This is done by unstable qemu-user-static package in Debian

---

# Easy button for binfmt_misc

```no-highlight
docker run --rm --privileged multiarch/qemu-user-static:register
```

---

<asciinema-player src="demo-buildkit6.cast" cols=100 rows=26 preload=true font-size=16></asciinema-player>

???

- On a host with binfmt_misc setup correctly, note the `F` flag
- Replacing my golang image with one for a different platform
- Running the build for that platform
- Inspect shows the selected platform
- And the container still runs, not the different arch output
- Lets prove we are really running a diff binary using the debug image
- From inside the debug image we need to install `file`
- And from `file` we can see AMD vs x86
  - The debug base image wasn't pulld again so it's still x86
- Before I forget, lets pull golang again with the default platform

---

# Cross Compiling

- Compiler support required
- Cannot use RUN step in final image
- buildkit includes platform ARGS

???

- I'm building with Go which includes cross compiling
- The final image will be for a diff platform, so RUN without qemu would fail
- You can still do everyting else, FROM, COPY, ENV, CMD, etc
- One more detail needed, variables to control our build

---

# buildkit ARGS

Image Target:

- `TARGETPLATFORM`: Eg linux/amd64, linux/arm/v7, windows/amd64.
- `TARGETOS`: OS component of TARGETPLATFORM
- `TARGETARCH`: architecture component of TARGETPLATFORM
- `TARGETVARIANT`: variant component of TARGETPLATFORM

Build Node:

- `BUILDPLATFORM`: Eg linux/amd64, linux/arm/v7, windows/amd64.
- `BUILDOS`: OS component of BUILDPLATFORM
- `BUILDARCH`: architecture component of BUILDPLATFORM
- `BUILDVARIANT`: variant component of BUILDPLATFORM

???

- Same options, just Target vs Build
- Platform is full string, e.g. linux/arm/v7
- OS e.g. linux
- Arch e.g. arm
- Variant e.g. v7

---

# Dockerfile: buildkit v7 (partial)

```no-highlight
# syntax=docker/dockerfile:experimental

*FROM --platform=$BUILDPLATFORM golang:1.12-alpine as dev
RUN apk add --no-cache git ca-certificates
RUN adduser -D appuser
WORKDIR /src
COPY . /src/
CMD CGO_ENABLED=0 go build -ldflags '-s -w -extldflags -static' -o app . && ./app

*FROM --platform=$BUILDPLATFORM dev as build
*ARG TARGETOS
*ARG TARGETARCH
RUN --mount=type=cache,sharing=locked,id=gomod,target=/go/pkg/mod/cache \
    --mount=type=cache,sharing=locked,id=goroot,target=/root/.cache/go-build \
*   CGO_ENABLED=0 GOOS=${TARGETOS} GOARCH=${TARGETARCH} \
*   go build -ldflags '-s -w -extldflags -static' -o app .
CMD ./app
...
```

???

- The `FROM --platform` flags are needed to pull an image for the build host
  rather than target platform
- These TARGET and BUILD ARGS line up nicely with GOOS and GOARCH
- I added even more flags to ensure a static compile
- Only use COPY steps to create target platform image as release, that stage is
  unchanged

---

<asciinema-player src="demo-buildkit7.cast" cols=100 rows=26 preload=true font-size=16></asciinema-player>

???

- Despite my typo, this is running on a node without qemu setup (ssh'd into a VM)
- The v7 image contains the `--platform` and cross compile vars
- First, lets try the v6 we ran before with qemu, that fails without qemu
- Next, lets try the v7 letting golang do the cross compile
- That builds, inspect shows the platform, but can we run it? Nope, as expected.
- We can build and run our debug container, and see the binary is indeed for ARM
- This can be pushed to a registry, but it would be nice if we pushed all the
  platforms under a single tag

---

# buildx

---

# buildx

- CLI Plugin for buildkit
- Multiple builders
- Multi-platform images

???

- `docker build` is a layer removed from buildkit, this gives all the options
  - All the cool options I've picked have been since added into `docker build`
- Remote builders are supported, uses `docker context`
- Multiple nodes may be added to a single builder definition for multiple platforms
- When pushing to a registry, you may create a multi-platform image manifest (x86, ARM, etc) for a single tag
- Multiple builds run for different platforms from a single command

---

# Enable Experimental CLI

```shell
$ export DOCKER_CLI_EXPERIMENTAL=enabled
```

```no-highlight
$ echo '{ "experimental": "enabled" }' >~/.docker/config.json
```

???

- buildx and other CLI plugins are experimental in 19.03
- CLI experimental is separate from Engine experimental
- The first option would need to be added to .bashrc for persistence
- The second option should probably be done with `vim` to avoid deleting existing logins

---

# Multi-Platform Images

```no-highlight
$ docker buildx create --driver docker-container \
    --name container default

$ docker buildx use container

$ docker buildx build \
  --platform linux/amd64,linux/arm64,linux/arm/v7 \
  -t sudobmitch/golang-hello:buildx \
  --output type=registry .

$ docker manifest inspect sudobmitch/golang-hello:buildx
```

---

<asciinema-player src="demo-buildx1.cast" cols=100 rows=26 preload=true font-size=16></asciinema-player>

???

- buildx works on top of docker context, there's the default, and I'll add one
  for a Pi
- Creating two buildx instances, local and Pi. Multi-node builders would have
  both in one instance.
- Inspect with a bootstrap deploys the docker-container driver deploys that
  container.
- buildx build now runs with all the platforms, pointed to a single reg tag
- Manifest list shows each manifest within tag, one per platform
- Pull will retrieve manifest for our local platform
- Override pull + qemu allows testing of other platforms
- Running container with rpi0 context pulls appropriate image for that arch
- Curl to different hostname shows it running

---

# Smaller Images

```shell
$ docker image ls golang-hello
REPOSITORY     TAG                 IMAGE ID       CREATED             SIZE
golang-hello   buildkit7           10e725ee3bd6   20 hours ago        6.99MB
golang-hello   buildkit6           10e725ee3bd6   20 hours ago        6.99MB
golang-hello   buildkit4           f5156b5aff72   20 hours ago        7.31MB
golang-hello   buildkit3           f5156b5aff72   20 hours ago        7.31MB
golang-hello   buildkit2           f5156b5aff72   20 hours ago        7.31MB
golang-hello   buildkit1           0f3a077975bd   3 hours ago         7.31MB
golang-hello   multi-stage4        69a3b1720f26   21 hours ago        7.31MB
golang-hello   multi-stage4-dev    9c0c8680f67c   21 hours ago        362MB
golang-hello   multi-stage3-debug  916e687a5071   21 hours ago        124MB
golang-hello   multi-stage2        15cfdffa0714   21 hours ago        10.1MB
golang-hello   multi-stage1        ce0eea9b7f73   21 hours ago        10.1MB
golang-hello   alpine2             c65efaacedef   21 hours ago        372MB
golang-hello   debian2             fec60bbc7c50   21 hours ago        820MB
golang-hello   debian1             38870c90431c   21 hours ago        886MB
golang-hello   external            ca2d1b1a8a8a   21 hours ago        134MB
```
???
- Some of those flags for static compiling or specifying the architecture
  managed to reduce my image size even more
- Importantly those smaller images also came with:
  - repeatable builds not dependent on developer machines
  - faster builds
  - portability to other platforms
---

# Are You Done Yet?

---

# Pretty Much

---

# Recap

---

# Recap

- Consolidate layers to reduce image size
- Select a minimal base image
- Don't ship a compiler if you don't need it
- Use volumes for a faster developer workflow
- buildkit can speed up builds
- buildkit can reuse the go cache between builds
- buildkit enables multi-platform building
- buildx makes multi-platform manifests easy

---

# Thank You

github.com/sudo-bmitch/presentations

.content[
.left-column[
.pic-80[![Slides QR](img/github-qr.png)]
]
.right-column[.align-right[.no-bullets[
<br><br>
- Brandon Mitchell
- Twitter: @sudo_bmitch
- GitHub: sudo-bmitch
]]]
]
???
- This presentation is online, you can watch it again