SRTOOL & Deterministic builds are back to Kusama & Polkadot
1. Introduction
In previous articles, I introduced srtool and announced its support for proposal hashes.
srtool allows making deterministic builds for Substrate based chains such as Kusama and Polkadot.
srtool has been extensively used for many of the Kusama runtime upgrades until april 2020 when we observed that builds were no longer deterministic.
srtool, that used to produce deterministic hashes for various users, using various Operating Systems on various machines, suddendly would not even produce the same binary for 2 subsequent runs on the same machine. So much for determinism!
1.1. Save your time
All I want is for you to save time!
This is why I suggest you jump to Run srtool, copy/paste the few commands and come back here to keep reading while srtool builds your runtime .
2. The issue
2.1. Finding the source of the issue
Finding the source of an issue usually helps greatly in solving it quickly.
When I saw that srtool was no longer producing similar binaries, I started investigating potential issues on srtool itself and ended concluding that the problem was not related to srtool.
My first attempt consisted in going back to a working state and changing one variant at a time to figure out what would cause the issue. We will see that this angle did not lead very far.
Going back to working state meant:
-
going back to the last known working version of
srtool -
going back to the last known working version of Kusama
I then experimented with a newer version of Kusama. In order to build Kusama and its runtime, you must use a nightly version the Rust compiler.
Nightlies are versions of the compiler including cutting edge features as well as features that are not mature/tested enough to make it into the stable version of the compiler.
You get with nightlies more goodies at the price of things breaking sometimes. As a matter of fact, some nightlies are not even usable at all.
Kusama (and Substrate in general) are on the cutting edge and do use newer features: they rely heavily on nightlies. Compiling a recent version of the Kusama runtime was unfortunately not possible with the latest "knonw working version" of srtool.
My conclusions about the issue at that point were:
-
does not come from
srtool -
may come from the code base
-
may come from the rustc compiler
-
may actually come from both the code base and the rutc compiler
Knowing where an issue is not, is a great start but this is not enough!
Benjamin Kampmann and his colleagues from Paritytech started looking deeper into the issue and they discovered several problems resulting in breaking the build determinism.
I highly recommend reading the article about Hunting down a non-determinism-bug in our Rust Wasm build from Benjamin Kampmann. It guides the reader through their findings, their reasoning and the discoveries.
Yesterday, with the release of Polkadot v0.8.16, the first version where deterministic builds were working again was available and it was time to test with srtool.
2.2. Upgrading srtool
Older versions of srtool would not work on the newer code base. A newer version of the rustc compiler (nightlies) is required, which mewns a new image version for srtool.
The problem was not over yet on the srtool side:
-
nightly-2020-06-30would build the runtime but fail the determinism test -
nightly-2020-07-20would fix the determinism issue but the runtime would not build at all
It did not leave many options but to make a new image for srtool and find a way to make it work.
The new image for srtool can be found here: chevdor/srtool:nightly-0.7.20 but as mentioned above, this version may not build by default currently due to a cargo issue.
There is however a workaround for the time being. The solution to this issue was brought up by pierre-tomaka on Riot. You can follow Tomaka on Twitter if you are interested in crazy cool projects and Rust topics.
3. Run srtool
Here are the steps required to successfully use srtool at the moment.
|
Note
|
As the cargo issue gets resolved, the step marked EXTRA below will no longer be required.
|
Before we jump into it, let’s prepare and run a few checks:
-
docker pull chevdor/srtool:nightly-2020-07-20 -
docker images --digests chevdor/srtool | grep "2020-07-20"
The digest should match:
sha256:e69123e5779e71e56960a958deeac6be2fc78f8a149a3c4a1e3c0a90130cc77a
|
Note
|
The instructions below are a little more complex than they have to be. This is to ensure that you are building the same runtime with the right srtool version. If you followed the steps in the first article, you merely need to patch your .bashrc / .bash_profile and make sure you source it or re-open your shell.
|
The following commands are all one-liners. You may copy the entire highlighted command, paste in your terminal and press Enter.
-
cd into the folder of your repo
-
use a supported version of Polkadot / Kusama :
git checkout -b v0.8.16 v0.8.16 -
ensure you are building the polkadot runtime:
export PACKAGE=polkadot-runtime -
EXTRA 🧙♂️:
rm -rf target/srtool; export RUSTC_VERSION=nightly-2020-07-20; srtool cargo update -p libp2p-core --precise 0.20.1 -
regular
srtoolbuild:
rm -rf target/srtool; export RUSTC_VERSION=nightly-2020-07-20; srtool build
|
Note
|
The last command will take a while, if you need a coffee, that’s a good time. If you jumped here to Save your time, you may now keep reading. Throw a 6-dice, if the result is less than 8, jump back to Finding the source of the issue |
You should get an output similar to this:
🧰 Substrate Runtime Toolbox - srtool v0.9.5 🧰
- by Chevdor -
🏗 Building polkadot-runtime as release using rustc 1.47.0-nightly (d7f945163 2020-07-19)
⏳ That can take a little while, be patient... subsequent builds will be faster.
Since you have to wait a little, you may want to learn more about Substrate runtimes:
https://substrate.dev/docs/en/runtime/architecture-of-a-runtime
Compiling libc v0.2.72
Compiling proc-macro2 v1.0.18
Compiling unicode-xid v0.2.1
Compiling syn v1.0.33
Compiling crossbeam-queue v0.2.3
...
Building [==========================> ] 325/641: syn, gimli, crossbeam-epoch, ...
About 30 minutes later depending on your machine…
The output will look like:
✨ Your Substrate WASM Runtime is ready! ✨ Summary: Generator : srtool v0.9.5 GIT commit : 4a997abed01651da38dd454efb380ec6fa5ec2b0 GIT tag : v0.8.16 GIT branch : heads/v0.8.16 Time : 2020-07-21T13:28:43Z Rustc : rustc 1.47.0-nightly (d7f945163 2020-07-19) Size : 2130 KB (2181633 bytes) Content : 0x0061736d0100000001c0033b60037f7f...3531363320323032302d30372d313929 Package : polkadot-runtime Proposal : 0xbb15e679b58eec2a888994d2178ef2dccb6d95434183daa3659257136092b64c SHA256 : 04d45b8b2f10a5317d32f047b5fad9e42ee992fb970b41475e83e54be2f20565 Wasm : ./target/srtool/release/wbuild/polkadot-runtime/polkadot_runtime.compact.wasm
Beside showing you lots of information relevant to this build, the most interesting values srtool is producing are:
GIT commit : 4a997abed01651da38dd454efb380ec6fa5ec2b0 (1) Rustc : rustc 1.47.0-nightly (d7f945163 2020-07-19) (2) Package : polkadot-runtime (3) Proposal : 0xbb15e679b58eec2a888994d2178ef2dccb6d95434183daa3659257136092b64c (4)
-
This shows the exact version of the code
-
This shows the exact version of the Rustc compile that was used
-
This shows the runtime we built. We could have built
kusama-runtimeas well for instance -
This is THE interesting value and should match what others find, as well as the hash of the proposal you see onchain.
Wilfried Kopp aka. Chevdor
Building Blockchains & Decentralized Solutions
I build decentralized solutions and tooling to support them. I am developing Smart Contracts and dApps on Ethereum and developing tooling for Substrate (Polkadot & Kusama). I love Rust! I am using Docker extensively and above all I like efficiency. GPG Fingerprint 15AF C574 D3F9 F1C3 CCDD E31E 2DCE C4DC 506E 6475.