The motivation

When deploying applications and dependencies to customer VMs, it is useful to reduce the number of files and the total size you need to upload.

While we could use something like gzip for this, it would be better to use zstd because it manages to compress better and faster. And even after deployment, we can use it in our tools and scripts.

As the result of this project, I wanted to have a single executable file that I can copy to a VM to compress and decompress data. I don’t want to deal with glibc version incompatibilities.

And I don’t want to need root access to install any dependencies, this should work on any VM we are given, regardless of permissions.

The solution

A good candidate for doing this is musl libc, which is designed to be statically compiled.Alpine Linux uses musl libc by default, and makes this really easy to do. We can just use a Podman container and build a static zstd binary inside it.

Aside from working on zstd-compressed files, the zstd tool can also use the LZ4, gzip and xz/lzma formats. This makes it a pretty good “universal” compression tool, and having all that in a single binary is very convenient.

Below is the script I used to compile a static zstd binary on Alpine Linux.

apk update
apk add wget
apk add make
apk add gcc
apk add musl-dev

apk add lz4-dev
apk add lz4-static
apk add zlib-dev
apk add zlib-static
apk add xz-dev
apk add xz-static

ZSTD_VERSION=1.5.4
URL="https://github.com/facebook/zstd/releases/download"
URL="${URL}/v${ZSTD_VERSION}/zstd-${ZSTD_VERSION}.tar.gz"

wget -O zstd.tar.gz "${URL}"
tar -xzf zstd.tar.gz
cd zstd-${ZSTD_VERSION}

export CFLAGS="-flto"
export LDLAGS="-flto -static -static-libgcc -static-libstdc++"
export HAVE_LZ4=1
export HAVE_ZLIB=1
export HAVE_LZMA=1
make zstd-release

cp programs/zstd /output/path/zstd

The results

And the result? A single 2.6 MB file that can compress/decompress any format we care about using.

$ file ./zstd
./zstd: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically
linked, with debug_info, not stripped

$ ls -lah ./zstd
-rwxrwxrwx 1 root root 2.6M Mar 21 13:28 ./zstd

$ ./zstd --help

  --format=zstd          Compress files to the `.zst` format. [Default]
  --format=gzip          Compress files to the `.gz` format.
  --format=xz            Compress files to the `.xz` format.
  --format=lzma          Compress files to the `.lzma` format.
  --format=lz4           Compress files to the `.lz4` format.