Cargo Remote: Speed up your Rust builds
There is no question that Rust is one of the, if not THE, best language of the decade.
Rust allows us writting safe code while doing most of the safety checks at compile time. That means that we no longer need to bloat the binary that is produced with extra checks to safeguard against developer mistakes such as null and dangling pointers or overflows to mention just a few.
This comes at a price: regular size Rust projects take a while to build. The Rust magic needs to happen somehow!
Not only that takes time but it also draws the precious energy our batteries tries saving for us!
1. Polkadot / Substrate
Polkadot became a very big project based on Rust.
You can get a feeling about activity on the project with the following video.
I am building the projects on a MacbookPro from 2017. This is admitely not the fastest machine for the job but I get a good level of productivity with it. The compile time increased with the size of the project, ranging from some 18 minutes at the beginning to more than an hour nowadays.
The problem became even more obvious when I started using a Docker container to build Polkadot. You may allow Docker using more or less resources such as RAM and CPU from your machine. Since I allowed Docker to acess limited resources on my local system, the build time increased again a little more.
It was definitely time to find a better solution and what a better time than holidays to tackle such a problem :)
2. distcc
C/C++ users may know solutions such as distcc allowing your machine to distribute the compiling of files to as many machines in your network that are available with distcc ready to chew on some compile work.
There is however no such a solution available for Rust.
3. Solution: cargo remote
I want to thank (again) Tomasz Drwięga for pointing me into the direction of the cargo plugin called cargo-remote.
Cargo-remote is rather a hack and is not distributed at all but the author can be thanked for putting this solution together as it does the trick and will save you tons of precious minutes if not hours every day.
Unlike distcc, cargo remote will not distribute the building of your project accros many machines in your network. Instead, it will send the job to ONE other machine and let it do the work. Once the remote machine is done, all the produced artifacts may be returned to the local machine so you can enjoy your freshly built binaries locally.
Not only it can reduce your Rust build time but it can also save those little laptop that are, for most, not really built to spend the day building Rust binaries. Instead, you may code on your favorite low specs laptop and use cargo remote to let this beefy cloud server with 16 cores chew your Rust source in no time.
This is also a great solution in a team where you want to boost productivity by investing into a beefy build server and save precious hours every day.
4. Installation
Before you can enjoy cargo remote, you will need to installa few things locally and on the remote server you intend to use.
4.1. Local install
If you are using a Mac, you first will need to install a fresher version of rsync as the version that comes with your Mac seems to have issues:
brew install rsync
|
Note
|
On other OS, you will need rsync as well. |
You will also need to install cargo-remote locally:
cargo install --git https://github.com/sgeisler/cargo-remote
Finally, you will need an ssh key and I will assume that you already have that done. If this is not the case, gitlab and github have great tutorials.
4.2. Remote install
The first thing to be done on your remote is to add your ssh public key in $HOME/.ssh/authorized_keys in order to be able to ssh from your local system to the remote with:
ssh <user>@<remote>
|
Note
|
You may need to run ssh-add first to load your key once and no longer be bothered. |
We then need to install the whole rust toolchain and some extra goodies if we plan on hacking on Polkadot. I planned on providing a one-liner to do all of that but I quickly realized that there is not much needed on top of what ParityTech already provides:
curl https://getsubstrate.io -sSf | bash -s -- --fast
Running this script only takes a few minutes and your system will be ready to accept remote builds for any Rust project but also for Polkadot and Substrate based projects.
5. Enjoy the powaaa!
In the following samples, we connect to a machine I named cargoremote (in my /etc/hosts file) and as user will.
|
Note
|
All of the following command are ran from the project’s folder on the local machine. |
cargo remote -c -r will@cargoremote -e .profile -- build --release
-h flag:cargo remote -c -h -r will@cargoremote -e .profile -- contract build
-c:cargo remote -h -r will@cargoremote -e .profile -d nightly -- test
|
Note
|
I found the -e flag to be required for everything to work |
6. How does cargo remote work?
Cargo remote uses rsync to first copy your code base to the remote server. The first time it runs, rsync makes a full copy of your source. Later on, only the files you have changed will be Remotely SYNCed with with the remote.
The output in your local terminal is very similar to what you would see if you would build locally so if you run tests or benchmarks for instance, everything will look 'local', only much faster.
cargo remote also offer the possibility to rsync back the artifacts produced by the build.
7. Caching
Using cargo remote does not prevent you from using caching. The remote cache will be based on the user and the cargo project name. So your first build will be as fast as your remote server can handle but the next build should benefit from the local cache and be much faster.
|
Note
|
If you update your toolchain with rustup update, your cache will be voided and the next build will be a full one again. |
8. Results
Using cargo remote significantly reduced my build time. I tested using a 16 cores clould machine with fast networking and by build time for the Polkadot project went down from over 1 hour to around 8 minutes!!!
If you observe your CPU usage during the build, you will notice that all your precious cores are not always put at work during the build.
At first, rsync does not benefit from any extra core. The beginning of the cargo build also seems to be running on a small number of cores/cpus.
After that, all your cores will be used and that when the benefit of using cargo remote kicks in.
At the end of the build process, the linking does not benefit from many cores/spus so you will likely see less and less cores used toward the end of the build.
On a freshly installed system, I got the HEAD of Polkadot (release) to build in just about 10 min as shown below.
...
Compiling polkadot-validation v0.7.10 (/home/will/remote-builds/polkadot/validation)
Compiling polkadot-network v0.7.10 (/home/will/remote-builds/polkadot/network)
Compiling polkadot-service v0.7.10 (/home/will/remote-builds/polkadot/service)
Compiling polkadot-cli v0.7.10 (/home/will/remote-builds/polkadot/cli)
Finished release [optimized] target(s) in 10m 19s
It also takes some time to fetch all the artifacts the first time but this process is also greatly reduced for subsequent builds.
This site is best viewed using the Brave browser.
Not only it will help you block most trackers, keep you safer on the internet but it also allows you supporting websites such as this one without having to spend a dime.
Using Brave allows me bringing this content to you, without any ad, subscription and other annoyance (I am looking at you Medium…). Using Brave gives you the opportunity to show your appreciation for that in a few clicks!
You can read more about Brave in this article.
Wilfried Kopp aka. Chevdor
Building Blockchains & Decentralized Solutions
I build decentralized solutions and tooling to support them. I am developing Smart Contracts and dApps on Ethereum and developing tooling for Substrate (Polkadot & Kusama). I love Rust! I am using Docker extensively and above all I like efficiency. GPG Fingerprint 15AF C574 D3F9 F1C3 CCDD E31E 2DCE C4DC 506E 6475.