diff --git a/docker/README.md b/docker/README.md new file mode 100644 index 000000000..5da0821ef --- /dev/null +++ b/docker/README.md @@ -0,0 +1,23 @@ +## directory layout + +### dist/base_image/ + +Base Docker images for building distributable binaries. Uploaded to +Docker Hub (we need them to reproduce officially released builds). + +### dist/ + +Dockerfiles used to build local Docker images based on the base images +described above. Only used for generating distributable binaries. Not uploaded +to Docker Hub. + +### dist/binaries/ + +Docker images for end-users, obtained by copying distributable binaries inside +official Debian images. Uploaded to Docker Hub as part of the CI release process. + +Also contains some example `docker-compose` configuration files. + +## more details + +See the ["Binary distribution internals"](https://nimbus.guide/distribution_internals.html) page of the Nimbus book. diff --git a/docker/dist/README-Windows.md b/docker/dist/README-Windows.md.tpl similarity index 100% rename from docker/dist/README-Windows.md rename to docker/dist/README-Windows.md.tpl diff --git a/docker/dist/README.md b/docker/dist/README.md.tpl similarity index 100% rename from docker/dist/README.md rename to docker/dist/README.md.tpl diff --git a/docker/dist/entry_point.sh b/docker/dist/entry_point.sh index 7ce85e81c..727b8a9a2 100755 --- a/docker/dist/entry_point.sh +++ b/docker/dist/entry_point.sh @@ -192,7 +192,7 @@ for BINARY in ${BINARIES}; do fi cd - >/dev/null done -sed -e "s/GIT_COMMIT/${GIT_COMMIT}/" docker/dist/README.md > "${DIST_PATH}/README.md" +sed -e "s/GIT_COMMIT/${GIT_COMMIT}/" docker/dist/README.md.tpl > "${DIST_PATH}/README.md" if [[ "${PLATFORM}" == "Linux_amd64" ]]; then sed -i -e 's/^make dist$/make dist-amd64/' "${DIST_PATH}/README.md" @@ -202,7 +202,7 @@ elif [[ "${PLATFORM}" == "Linux_arm64v8" ]]; then sed -i -e 's/^make dist$/make dist-arm64/' "${DIST_PATH}/README.md" elif [[ "${PLATFORM}" == "Windows_amd64" ]]; then sed -i -e 's/^make dist$/make dist-win64/' "${DIST_PATH}/README.md" - cp -a docker/dist/README-Windows.md "${DIST_PATH}/" + cp -a docker/dist/README-Windows.md.tpl "${DIST_PATH}/README-Windows.md" elif [[ "${PLATFORM}" == "macOS_amd64" ]]; then sed -i -e 's/^make dist$/make dist-macos/' "${DIST_PATH}/README.md" elif [[ "${PLATFORM}" == "macOS_arm64" ]]; then diff --git a/docs/README.md b/docs/README.md index 0eca42b5f..ae4a8337d 100644 --- a/docs/README.md +++ b/docs/README.md @@ -10,7 +10,7 @@ cargo install mdbook mdbook-toc mdbook-open-on-gh cd docs/the_nimbus_book mdbook serve -p 4000 -# Publish book using makefile (in top-level) +# Publish book using makefile (in the top-level dir) make publish-book ``` diff --git a/docs/the_nimbus_book/src/binaries.md b/docs/the_nimbus_book/src/binaries.md index c6dd03df3..0ea03b4f8 100644 --- a/docs/the_nimbus_book/src/binaries.md +++ b/docs/the_nimbus_book/src/binaries.md @@ -1,18 +1,21 @@ # Nimbus binaries -We currently have binaries available for Linux `AMD64`, `ARM32` and `ARM64`, Windows and MacOS (`AMD64` and `ARM64`). +We currently have binaries available for Linux `AMD64`, `ARM` and `ARM64`, Windows `AMD64` and macOS (`AMD64` and `ARM64`). You can find the latest release here: [https://github.com/status-im/nimbus-eth2/releases](https://github.com/status-im/nimbus-eth2/releases) -Scroll to the bottom of the first release you see, and click on `Assets`. You should see a list that looks like the following: +Scroll to the bottom of the first release you see (no, not the nightly pre-release), and click on `Assets`. You should see a list that looks like the following: -![](https://i.imgur.com/R33o4MG.png) +![](https://i.imgur.com/6wuvM2d.png) Click on the `tar.gz` file that corresponds to your OS and architecture, unpack the archive, read the README and run the binary directly (or through one of our provided wrapper scripts). We've designed the build process to be reproducible. In practice, this means that anyone can verify that these exact binaries were produced from the corresponding source code commits. For more about the philosophy and importance of this feature see [reproducible-builds.org](https://reproducible-builds.org/). -For instructions on how to reproduce the build, [see here](https://github.com/status-im/nimbus-eth2/blob/master/docker/dist/README.md#reproducing-the-build). - -> **N.B.** Binaries are built with Docker. As such they do not support the `log-file` [option](./options.md). We recommend capturing the console output with a [log rotation helper](./log-rotate.md) instead. +For instructions on how to reproduce those binaries, see "README.md" inside the archive. +> **N.B.** These binaries are meant to run as a service. As such, they do not +support the `log-file` [option](./options.md) - duplicating stdout to a +log file would complicate things. We recommend capturing +their output (stdout and stderr) with a [log rotation +helper](./log-rotate.md) instead. diff --git a/docs/the_nimbus_book/src/distribution_internals.md b/docs/the_nimbus_book/src/distribution_internals.md index 443dff334..f9b3683f9 100644 --- a/docs/the_nimbus_book/src/distribution_internals.md +++ b/docs/the_nimbus_book/src/distribution_internals.md @@ -11,16 +11,17 @@ for software that deals with digital tokens of significant value. ## Docker containers for internal use -The easiest way to guarantee that users are able to replicate +The easiest way to guarantee that users are able to replicate our binaries for themselves is to give them the same software environment we used in CI. Docker containers fit the bill, so everything starts with the architecture- and -OS-specific containers in `docker/dist/base\_image/`. +OS-specific containers in `docker/dist/base_image/`. These images contain all the packages we need, are built and published once (to Docker Hub), and are then reused as the basis for temporary Docker images where the `nimbus-eth2` build is carried out. -These temporary images are controlled by Dockerfiles in `docker/dist/`. Since we're not publishing them anywhere, we can customize them to the system +These temporary images are controlled by Dockerfiles in `docker/dist/`. Since +we're not publishing them anywhere, we can customize them to the system they run on (we ensure they use the host's UID/GID, the host's QEMU static binaries, etc); they get access to the source code through the use of external volumes. @@ -30,25 +31,25 @@ It all starts from the GitHub actions in `.github/workflows/release.yml`. There is a different job for each supported OS-architecture combination and they all run in parallel (ideally). -The `build-amd64` CI job is special, because it creates a new -GitHub release draft, as soon as possible. All the other jobs will upload their -binary distribution tarballs to this draft release, but, since it's not feasible -to communicate between CI jobs, they simply use GitHub APIs to find out what -the latest release is, check that it has the right Git tag, and use that as their -last step. +Once all those CI jobs complete successfully, a GitHub release draft is created +and all the distributable archives are uploaded to it. A list of checksums for +the main binaries is inserted in the release description. That draft needs to +be manually published. -The build itself is triggered by a Make target: `make dist-amd64`. This invokes -`scripts/make\_dist.sh` which builds the corresponding Docker container from +The build itself is triggered by a Make target. E.g.: `make dist-amd64`. This invokes +`scripts/make_dist.sh` which builds the corresponding Docker container from `docker/dist/` and runs it with the Git repository's top directory as an external volume. -The entry point for that container is `docker/dist/entry\_point.sh` and that's +The entry point for that container is `docker/dist/entry_point.sh` and that's where you'll find the Make invocations needed to finally build the software and create distributable tarballs. ## Docker images for end users -Configured in `.github/workflows/release.yml` (exclusively for the `build-amd64` job): we unpack the distribution tarball and copy its content into a third type of Docker image - this one meant for end users and defined by `docker/dist/binaries/Dockerfile.amd64`. +Configured in `.github/workflows/release.yml` (only for Linux AMD64, ARM and +ARM64): we unpack the distribution tarball and copy its content into a third +type of Docker image - meant for end users and defined by +`docker/dist/binaries/Dockerfile.amd64` (and related). We then publish that to [Docker Hub](https://hub.docker.com/r/statusim/nimbus-eth2). -