Using Go for Multi Arch Kubernetes - Part 2

Posted on by

Part 2 - Building CGO for Multi Arch

This is Part 2 of a series on building out multi-arch services in Kubernetes using Golang. For a guide on how to build plain Go services in multiple architectures see Part 1 of this series. In this post we’ll be building containers similar to the previous example with the added feature of supporting CGO.

If you’ve read any of Rob Pike’s writings on the design of Go, watched one of his talks online, or attended a conference where he’s spoken you’ll know that Rob firmly believes that CGO is not Go. Considering he’s one of the primary designers and contributors to Go I believe him when he says this. However, sometimes it’s impossible to escape and you have to include a C library in a Go program. Often when working in non-containerized environments it’s suffcient to state the library is a necessary dependency to install prior to running a particular Go program. But what if you want to use a scratch container as in the previous example? Or run all of your build steps on an x86_64 machine?

As a refresher the benefits of using scratch containers are that they have no runtime. Meaning they are designed to run your application and your application only. Look for another blog post at a later date detailing the benefits and the drawbacks of using scratch containers.

Now, if you’ve been developing software for a while you’re likely familiar with the process of building C libraries and programs into your projects. However, you may be less familiar with building them in statically so they have no external dependencies and even less familiar with building them across CPU architectures. For this example we’re going to be pulling from my network scanning project, Scandalorian, which uses gopacket under the hood. This is one of the cases where there are not really a lot of good options or alternatives to using CGO.

One caveat is that this is heavily centered around Linux. Considering that the vast majority of container workloads run on Linux other operating systems such as Windows, OSX, and BSD are out of scope for this article. It seems like it should be possible to do but your mileage may vary. Ok, let’s dive right in to some code.

The first thing we need to be able to do is identify the C library that we’re going to load. For this example that’s going to be libpcap v1.9.1. Retrieve and unpack the library with the following commands:

> cd ~/Development
> wget http://www.tcpdump.org/release/libpcap-1.9.1.tar.gz
> tar xvf libpcap-1.9.1.tar.gz

Now that we have the library we’re going to work with we need to configure the library with the appropriate setup that Go can understand. We’re going to start with amd64 for simplicity. Before configuring you’ll need the following tools:

  • byacc
  • flex
  • libpcap-dev
  • unzip

On a Debian based distribution you can install those tools with the following command:

> apt update && apt install -y build-essential byacc flex libpcap-dev file unzip

Now we can setup libpcap for compiling into our Go binary:

> cd libpcap-1.9.1
> ./configure --with-pcap=linux
> make

Keep in mind that these steps may vary depending on the C library you want to use, this may or may not work for every one that you encounter so it might take some experimentation to get it right. At this point you’re ready to build your Go binary, let’s write a small amount of code to use libpcap.

> cd ~/Development
> mkdir pcap-test
> cd pcap-test
> touch main.go
> go get github.com/google/gopacket/pcap

Now add the following code in the main.go file:

package main

import (
    "fmt"
    "log"
    "github.com/google/gopacket/pcap"
)

func main() {
    devices, err := pcap.FindAllDevs()
    if err != nil {
        log.Fatal(err)
    }

    // Print device information
    fmt.Println("Devices found:")
    for _, device := range devices {
        fmt.Println("\nName: ", device.Name)
        fmt.Println("Description: ", device.Description)
        fmt.Println("Addresses: ", device.Description)
        for _, address := range device.Addresses {
            fmt.Println("IP address: ", address.IP)
            fmt.Println("Subnet mask: ", address.Netmask)
        }
    }
}

Finally, let’s build the Go binary. We’re going to start with the command first and then we’ll cover what each new component of the command is doing:

> CGO_ENABLED=1 GOOS=linux GOARCH=amd64 CGO_LDFLAGS="-L/home/charles/Development/libpcap-1.9.1" go build -a -ldflags '-w -extldflags "-static"' -o pcap-test  .

Here’s a breakdown of each component of the command:

  • CGO_ENABLED=1 (this tells the Go compiler to use CGO)
  • GOOS=linux (building for Linux)
  • GOARCH=amd64 (the cpu arch we’re buidling for)
  • CGO_LDFLAGS=”-L/home/charles/Development/libpcap-1.9.1” (location of the built library)
  • -ldflags ‘-w -extldflags “-static”’ (build the library is statically linked)

Now that you know how to build this on an x86_64 machine, let’s do the same process to generate an arm64 binary. There are only a couple of steps added to accomplish this, paying careful attention to the environment variables. Assuming you’re going to build this on a Debian based distro we need to install the following extra dependencies for the C cross compiler toolchain:

> export CC=aarch64-linux-gnu-gcc
> export CFLAGS='-Os'
> apt install -y gcc-aarch64-linux-gnu libc6-dev-arm64-cross

Remove all references to the PCAP download we ran earlier and download a fresh copy, this makes sure that no configuration persists that could tie back to the x64 based system. Once that’s done enter the directory and build the library with the following commands:

> ./configure --host=aarch64-unknown-linux-gnu --with-pcap=linux
> make

Once your dependency was successfully built we’ll need to re-compile the Go program for arm64 as well:

> cd pcap-test
> export CC=aarch64-linux-gnu-gcc
> export CFLAGS='-Os'
> CGO_ENABLED=1 GOOS=linux GOARCH=arm64 CGO_LDFLAGS="-L/app/libpcap-1.9.1" go build -a -ldflags '-w -extldflags "-static"' -o pcap-test .
> file pcap-test
pcap-test: ELF 64-bit LSB executable, ARM aarch64, version 1 (GNU/Linux), statically linked, BuildID[sha1]=ac4e20d6404725b29719dd98497a6c98e11bf331, for GNU/Linux 3.7.0, not stripped

At this point all that’s left is to package these steps into a Docker file. Here’s a complete example that will build all three architectures:

FROM golang:latest as build-arm

RUN apt update && apt install -y \
		gcc-arm* \
		libc6-dev-armhf-cross\
        byacc flex \
        libpcap-dev \
        file unzip
RUN mkdir /app
WORKDIR /app
COPY ./ .
RUN ls -al
ENV PCAPV=1.9.1
ENV CC=arm-linux-gnueabi-gcc
RUN wget http://www.tcpdump.org/release/libpcap-$PCAPV.tar.gz && \
    tar xvf libpcap-$PCAPV.tar.gz && \
    cd libpcap-$PCAPV && \
    ./configure --host=arm-linux --with-pcap=linux && \
    make
RUN cd /app && \
    CGO_ENABLED=1 GOOS=linux GOARCH=arm CGO_LDFLAGS="-L/app/libpcap-$PCAPV" go build -a -ldflags '-w -extldflags "-static"' -o pcap-test . && \
    file pcap-test

FROM golang:latest as build-arm64
RUN apt update && apt install -y \
        gcc-aarch64* \
        libc6-dev-arm64-cross\
        byacc flex \
        libpcap-dev \
        file unzip
RUN mkdir /app
WORKDIR /app
COPY ./ .
RUN ls -al
ENV PCAPV=1.9.1
ENV CC=aarch64-linux-gnu-gcc
ENV CFLAGS='-Os'
RUN wget http://www.tcpdump.org/release/libpcap-$PCAPV.tar.gz && \
    tar xvf libpcap-$PCAPV.tar.gz && \
    cd libpcap-$PCAPV && \
    ./configure --host=aarch64-unknown-linux-gnu --with-pcap=linux && \
    make
RUN cd /app && \
    CGO_ENABLED=1 GOOS=linux GOARCH=arm64 CGO_LDFLAGS="-L/app/libpcap-$PCAPV" go build -a -ldflags '-w -extldflags "-static"' -o pcap-test . && \
    file pcap-test


FROM golang:latest as build-amd64
RUN apt update && apt install -y \
        build-essential \
        byacc flex \
        libpcap-dev \
        file unzip
RUN mkdir /app
WORKDIR /app
COPY ./ .
RUN ls -al
ENV PCAPV=1.9.1
ENV CC=gcc
ENV CFLAGS='-Os'
RUN wget http://www.tcpdump.org/release/libpcap-$PCAPV.tar.gz && \
    tar xvf libpcap-$PCAPV.tar.gz && \
    cd libpcap-$PCAPV && \
    ./configure --with-pcap=linux && \
    make
RUN cd /app && \
    CGO_ENABLED=1 GOOS=linux GOARCH=amd64 CGO_LDFLAGS="-L/app/libpcap-$PCAPV" go build -a -ldflags '-w -extldflags "-static"' -o pcap-test . && \
    file pcap-test

FROM scratch as arm
COPY --from=build-arm /app/pcap-test /go/bin/pcap-test
ENTRYPOINT [ "/go/bin/pcap-test" ]

FROM scratch as arm64
COPY --from=build-arm64 /app/pcap-test /go/bin/pcap-test
ENTRYPOINT ["/go/bin/pcap-test"]

FROM scratch as amd64
COPY --from=build-amd64 /app/pcap-test /go/bin/pcap-test
ENTRYPOINT ["/go/bin/pcap-test"]

In Part 3 we’ll be using this project to build the project in CI using Github Actions. It’ll cover setting up your workflow files and how we use manifest files in Docker to tag an image with the appropriate CPU architecture.

Charles Burton

Denver, United States
Email me

It's much better in my opinion to make things for the joy of making it rather than buying it for the joy of having it.