pull fio/iperf3 binaries from github releases instead of repo

bin/README.md

@@ -1,28 +1,49 @@
 ## YABS Pre-Compiled Binaries
 
-This directory contains all of the binaries required to run the benchmarking tests. Naturally, there is a security risk to your machine and its contents by running this script since, after all, this is just a script on the internet. You'll simply have to have confidence that I don't have malicious intent and am semi-competent at writing a bash script. The script is made public so you can look at the code yourself. The binaries were compiled using a [Holy Build Box](https://github.com/phusion/holy-build-box) compilation environment in order to ensure the most portability. The compiled binary version numbers and compilations steps are noted below. Please open an issue if the compiled version is out of date and lacking any security-related and/or performance updates.
+This directory contains the compilation scripts used to build the fio and iperf3
+binaries that YABS uses. **The binaries themselves are no longer committed to the
+repository** - they are built automatically by GitHub Actions workflows
+([build-fio.yml](../.github/workflows/build-fio.yml) and
+[build-iperf3.yml](../.github/workflows/build-iperf3.yml)) on GitHub-hosted
+runners and published as release assets on this repository:
 
-### Binaries
+* `fio-*` releases (e.g. `fio-3.42`) - contain `fio_x64`, `fio_x86`, `fio_aarch64`, `fio_arm`
+* `iperf3-*` releases (e.g. `iperf3-3.21`) - contain `iperf3_x64`, `iperf3_x86`, `iperf3_aarch64`, `iperf3_arm`
+* `toolchains` release - musl cross-compilation toolchains mirrored from
+  [musl.cc](https://musl.cc/) (which blocks GitHub Actions IP ranges); see the
+  NOTICE.txt asset for licensing/source info
 
-| Binary Name | Version | Compile Date | Architecture | OS | SHA-256 Hash<br>(VirusTotal Scan) |
-|:-:|:-:|:-:|:-:|:-:|:-:|
-| fio_x64 | 3.39 | 20-APR-2025 | x86_64 | 64-bit | [b511bda](https://www.virustotal.com/gui/file/b511bda3b26b6d840698f543d63e956d7466b8512c10ff0ada8292d556c33fb1) |
-| fio_x86 | 3.39 | 20-APR-2025 | i686 | 32-bit | [42e2e0b](https://www.virustotal.com/gui/file/42e2e0b0370faeb8e53dcf48dfff15daa9baaadfd196d9bdda57af196bedf0b3) |
-| fio_aarch64 | 3.39 | 20-APR-2025 | ARM (aarch64) | 64-bit | [e2942a2](https://www.virustotal.com/gui/file/e2942a26d4b249076486677c9c12cd7f1a572854a5e136597d1391e8ad75ffb0) |
-| fio_arm | 3.39 | 20-APR-2025 | ARM  | 32-bit | [3a96b1c](https://www.virustotal.com/gui/file/3a96b1cadfb51501b7fd54dc47a4dad666dc4382d4fe152116c62ae6a07485ea) |
-| iperf3_x64 | 3.18 | 14-DEC-2024 | x86_64 | 64-bit | [ef787ab](https://www.virustotal.com/gui/file/ef787abbe4b09c7958ed592df52dfe3a2848cbdee5b76738c757d7c51c348053) |
-| iperf3_x86 | 3.18 |  14-DEC-2024 | i686 | 32-bit | [655eb51](https://www.virustotal.com/gui/file/655eb51abc36ddaa624c1d0e98c6930e8b1e9d91c85e5a3443624355656be9b9) |
-| iperf3_aarch64 | 3.18 | 14-DEC-2024 | ARM (aarch64) | 64-bit | [92e5821](https://www.virustotal.com/gui/file/92e5821cfbaa1f8faf123b4d6773dc0f6efef221b9308668a21ddabc04a1de20) |
-| iperf3_arm | 3.15* | 20-OCT-2023 | ARM | 32-bit | [310e80f](https://www.virustotal.com/gui/file/310e80f442dda47fa0fe41225af85e8b91e75116dce5187f123380fd3c3c85a8) |
+Each binary release includes SHA-256 checksums (`*-checksums.sha256`), a
+VirusTotal scan summary in the release notes, and the exact upstream source
+tarball the binaries were built from. Binaries are cross-compiled with musl
+toolchains for maximum portability.
 
-Note: ARM compatibility is considered experimental. Static binaries for 32-bit and ARM-based machines are cross-compiled within a Holy Build Box container using the [musl toolchain](https://musl.cc/).
+At runtime, yabs.sh auto-detects the latest `fio-*`/`iperf3-*` release via the
+repository's releases feed and downloads the appropriate binary for the host
+architecture. If the lookup fails, it falls back to a pinned release tag.
 
-### Compile Notes
+Naturally, there is a security risk to your machine and its contents by running
+this script since, after all, this is just a script on the internet. You'll
+simply have to have confidence that I don't have malicious intent and am
+semi-competent at writing a bash script. The script and the workflows that build
+the binaries are public so you can look at the code yourself. The binaries are
+compiled using a [Holy Build Box](https://github.com/phusion/holy-build-box)
+compilation environment in order to ensure the most portability. Please open an
+issue if the binaries are out of date and lacking any security-related and/or
+performance updates - the daily scheduled workflows should pick up new upstream
+releases automatically.
+
+Note: ARM compatibility is considered experimental.
+
+### Compile Notes (manual builds)
+
+The workflows above are the canonical build path. To build manually, the same
+scripts can be run inside a Holy Build Box container:
 
 **Pre-reqs**:
   * Docker - https://www.docker.com/
 
-**Compiling 64-bit binaries**:
+**Compiling 64-bit (x86_64) binaries**:
 
 ```sh
 docker run -t -i --rm -v `pwd`:/io phusion/holy-build-box-64:latest bash /io/compile.sh
@@ -32,9 +53,11 @@ docker run -t -i --rm -v `pwd`:/io phusion/holy-build-box-64:latest bash /io/com
 
 ### Cross-compiling Notes
 
-Compilation of ARM-compatible binaries requires additional environment variables to identify the proper musl toolchain and architecture to target for cross-compilation.
+Compilation of 32-bit and ARM-compatible binaries requires additional
+environment variables to identify the proper musl toolchain and architecture to
+target for cross-compilation.
 
-**Compiling 32-bit binaries**:
+**Compiling 32-bit x86 binaries**:
 
 ```sh
 docker run -t -i --rm -v `pwd`:/io --env ARCH=x86 --env CROSS=i686-linux-musl --env HOST=i686-linux-musl phusion/holy-build-box-64:latest bash /io/cross-compile.sh
@@ -46,12 +69,8 @@ docker run -t -i --rm -v `pwd`:/io --env ARCH=x86 --env CROSS=i686-linux-musl --
 docker run -t -i --rm -v `pwd`:/io --env ARCH=aarch64 --env CROSS=aarch64-linux-musl --env HOST=aarch64-linux-gnu phusion/holy-build-box-64:latest bash /io/cross-compile.sh
 ```
 
-**Compiling ARM 32-bit binaries\***:
+**Compiling ARM 32-bit binaries**:
 
 ```sh
 docker run -t -i --rm -v `pwd`:/io --env ARCH=arm --env CROSS=arm-linux-musleabihf --env HOST=arm-linux-gnueabihf phusion/holy-build-box-64:latest bash /io/cross-compile.sh
 ```
-
-64-bit (aarch64) and 32-bit (x86, arm) binaries will be placed in the current directory.
-
-\* ARM 32-bit: Last sucessful compiliation of ARM 32-bit binary for iperf3 is v3.15. All later versions fail to compile.

bin/compile.sh

@@ -36,9 +36,12 @@ source /hbb_exe/activate
 
 # download and compile fio
 cd ~
-curl -L https://github.com/axboe/fio/archive/fio-3.39.tar.gz -o "fio.tar.gz"
+curl -L https://github.com/axboe/fio/archive/fio-3.42.tar.gz -o "fio.tar.gz"
 tar xf fio.tar.gz
-cd fio-fio*
+cd fio-fio-*
+# fio >= 3.42 includes both linux/prctl.h and sys/prctl.h in backend.c, which
+# conflicts under musl; sys/prctl.h alone is sufficient (harmless on glibc too)
+sed -i '/#include <linux\/prctl.h>/d' backend.c
 ./configure --disable-native
 make
 
@@ -49,9 +52,9 @@ cp fio "/io/fio_$ARCH"
 
 # download and compile iperf
 cd ~
-curl -L https://github.com/esnet/iperf/archive/3.18.tar.gz -o "iperf.tar.gz"
+curl -L https://github.com/esnet/iperf/archive/3.21.tar.gz -o "iperf.tar.gz"
 tar xf iperf.tar.gz
-cd iperf*
+cd iperf-*
 ./configure --disable-shared --disable-profiling
 make
 

bin/cross-compile.sh

@@ -8,14 +8,16 @@ set -x
 
 # remove obsolete CentOS repos
 cd /etc/yum.repos.d/
-rm CentOS-Base.repo CentOS-SCLo-scl-rh.repo CentOS-SCLo-scl.repo CentOS-fasttrack.repo CentOS-x86_64-kernel.repo
+rm -f CentOS-Base.repo CentOS-SCLo-scl-rh.repo CentOS-SCLo-scl.repo CentOS-fasttrack.repo CentOS-x86_64-kernel.repo
 
 yum install -y yum-plugin-ovl # fix for docker overlay fs
 yum install -y xz
 
-# download musl cross compilation toolchain
+# download musl cross compilation toolchain - mirrored as a release asset on the
+# repo since musl.cc blocks GitHub Actions IP ranges (musl.cc fallback kept)
 cd ~
-curl -L "https://musl.cc/${CROSS}-cross.tgz" -o "${CROSS}-cross.tgz"
+curl -fL -4 --retry 5 --retry-delay 2 --connect-timeout 15 "https://github.com/masonr/yet-another-bench-script/releases/download/toolchains/${CROSS}-cross.tgz" -o "${CROSS}-cross.tgz" \
+	|| curl -fL -4 --retry 5 --retry-delay 2 --connect-timeout 15 "https://musl.cc/${CROSS}-cross.tgz" -o "${CROSS}-cross.tgz"
 tar xf "${CROSS}-cross.tgz"
 
 # download, compile, and install libaio as static library
@@ -30,11 +32,15 @@ source /hbb_exe/activate
 
 # download and compile fio
 cd ~
-curl -L https://github.com/axboe/fio/archive/fio-3.39.tar.gz -o "fio.tar.gz"
+curl -L https://github.com/axboe/fio/archive/fio-3.42.tar.gz -o "fio.tar.gz"
 tar xf fio.tar.gz
-cd fio-fio*
+cd fio-fio-*
+# fio >= 3.42 includes both linux/prctl.h and sys/prctl.h in backend.c, which
+# conflicts under musl; sys/prctl.h alone is sufficient
+sed -i '/#include <linux\/prctl.h>/d' backend.c
 CC=/root/${CROSS}-cross/bin/${CROSS}-gcc ./configure --disable-native --build-static
-make
+# link against libatomic for 32-bit/ARM targets that lack native 64-bit atomics
+make EXTLIBS+=' -latomic'
 
 # verify no external shared library links
 libcheck fio
@@ -43,10 +49,13 @@ cp fio "/io/fio_$ARCH"
 
 # download and compile iperf
 cd ~
-curl -L https://github.com/esnet/iperf/archive/3.18.tar.gz -o "iperf.tar.gz"
+curl -L https://github.com/esnet/iperf/archive/3.21.tar.gz -o "iperf.tar.gz"
 tar xf iperf.tar.gz
-cd iperf*
+cd iperf-*
 CC=/root/${CROSS}-cross/bin/${CROSS}-gcc ./configure --disable-shared --disable-profiling --build x86_64-pc-linux-gnu --host "${HOST}" --with-openssl=no --enable-static-bin
+# remove libatomic.la so libtool links the static libatomic.a rather than
+# attempting to link libatomic.so into the static binary (breaks arm32 builds)
+rm -f "/root/${CROSS}-cross/${CROSS}/lib/libatomic.la"
 make
 
 # verify no external shared library links