{"id":5166,"date":"2024-07-01T13:50:08","date_gmt":"2024-07-01T18:50:08","guid":{"rendered":"https:\/\/neosmart.net\/blog\/?p=5166"},"modified":"2024-07-10T15:13:45","modified_gmt":"2024-07-10T20:13:45","slug":"benchmarking-rust-compilation-speedups-and-slowdowns-from-sccache-and-zthreads","status":"publish","type":"post","link":"https:\/\/neosmart.net\/blog\/benchmarking-rust-compilation-speedups-and-slowdowns-from-sccache-and-zthreads\/","title":{"rendered":"Benchmarking rust compilation speedups and slowdowns from sccache<\/code> and -Zthreads<\/code>"},"content":{"rendered":"

Just a PSA from one rust developer to another: if you use sccache<\/code>, take a moment to benchmark a clean build1<\/a><\/sup> of your favorite or current project and verify whether or not having RUSTC_WRAPPER=sccache<\/code> is doing you any favors.<\/p>\n

I’ve been an sccache user almost from the very start, when the Mozilla team first introduced it in the rust discussions on GitHub maybe seven years back or so, probably because of my die-hard belief in the one-and-only ccache<\/code> earned over long years of saving my considerable time and effort on C++ development projects (life pro-tip: git bisect<\/code> on large C or C++ projects is a night-and-day difference with versus without ccache<\/code>). At the same time, I was always painfully aware of just how little sccache<\/code> actually cached compared to its C++ progenitor, and I was left feeling perpetually discontented ever since learning to query its hit rate stats with sccache -s<\/code> (something I never needed to do for ccache<\/code>).<\/p>\n

But my blind belief in the value of build accelerators led me to complacency, and I confess that with sccache<\/code> mostly<\/em> chugging away without issue in the background, I kind of forgot that I had RUSTC_WRAPPER<\/code> set at all. But I recently remembered it and in a bout of procrastination, decided to benchmark how much time sccache<\/code> was actually<\/em> saving me… and the results were decidedly not great.<\/p>\n

<\/p>\n

The bulk of my rust use at $work is done on a workstation under WSLv1, while a lot of of my open source rust work is done on a “mobile workstation” running a Debian derivative. I began my investigation on the WSL desktop, originally spurred by a need<\/del> desire to see what kind of speedups different values for the ~recently added -Z threads<\/code> flag<\/a> to parallelize the rustc<\/code> frontend would net me, then remembered that I had sccache<\/code> enabled and should disable that for deterministic results… leading me to include the impact of sccache<\/code> in my benchmarks.<\/p>\n

On a (well-cooled, not thermally throttled) 16-core (32-thread) AMD Ryzen ThreadRipper 1950X machine with 64 GB of DDR4 RAM and NVMe boot and (code) storage disks, using rustc 1.81.0-nightly to compile a relatively modest-sized rust project from scratch in debug mode (followed by exactly one clean rebuild in the case of sccache to see how much of a speedup it gives), I obtained the following results:<\/p>\n\n\n\n\n\n\n\n\n\n\n\n\n
-Zthreads<\/th>\nsccache<\/th>\nTime<\/th>\nTime (2nd run)<\/th>\n<\/tr>\n<\/thead>\n
not set<\/em><\/td>\nno<\/td>\n33.08s<\/td>\n<\/td>\n<\/tr>\n
not set<\/em><\/td>\nyes<\/td>\n1m 18s<\/td>\n56.20s<\/td>\n<\/tr>\n
8<\/td>\nno<\/td>\n33.27s<\/td>\n<\/td>\n<\/tr>\n
8<\/td>\nyes<\/td>\n1m 32s<\/td>\n1m 00s<\/td>\n<\/tr>\n
16<\/td>\nno<\/td>\n34.43s<\/td>\n<\/td>\n<\/tr>\n
16<\/td>\nyes<\/td>\n40.78s<\/td>\n56.06s<\/td>\n<\/tr>\n
32<\/td>\nno<\/td>\n37.25s<\/td>\n<\/td>\n<\/tr>\n
32<\/td>\nyes<\/td>\n1m 14s<\/td>\n52.99s<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

Shockingly enough, there was not a single configuration where the usage of sccache<\/code> ended up speeding up compilation over the baseline without it \u2014 not even on a subsequent clean build of the same code with the same RUSTFLAGS<\/code> value.<\/p>\n

After inspecting the results of sccache -s<\/code> and with some experimentation, it turned out that this pathological performance appeared to be caused \u2013 in part \u2013 by having a full sccache cache folder.2<\/a><\/sup> Running rm -rf ~\/.cache\/sccache\/*<\/code> then re-running the benchmark (a few project commits later, so not to be directly compared to the above) revealed a significant improvement to the baseline build times with sccache\u2026 but probably not enough to justify its use:<\/p>\n

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
-Zthreads<\/th>\nsccache<\/th>\nTime<\/th>\nTime
\n(2nd Run)<\/th>\n		-ZTHREADS<\/th>\nsccache
\n(initial)<\/th>\n		sccache
\n(subsequent)<\/th>\n<\/tr>\n<\/thead>\n
not set<\/em><\/td>\nno<\/td>\n31.41s<\/td>\n<\/td>\n1.000<\/td>\n<\/td>\n<\/td>\n<\/tr>\n
not set<\/em><\/td>\nyes<\/td>\n42.57s<\/td>\n29.47s<\/td>\n<\/td>\n1.355<\/td>\n0.938<\/td>\n<\/tr>\n
0<\/td>\nno<\/td>\n35.94s<\/td>\n<\/td>\n1.144<\/td>\n<\/td>\n<\/td>\n<\/tr>\n
0<\/td>\nyes<\/td>\n44.26s<\/td>\n28.95s<\/td>\n<\/td>\n1.231<\/td>\n0.806<\/td>\n<\/tr>\n
2<\/td>\nno<\/td>\n30.53s<\/td>\n<\/td>\n0.972<\/td>\n<\/td>\n<\/td>\n<\/tr>\n
2<\/td>\nyes<\/td>\n58.94s<\/td>\n38.36s<\/td>\n<\/td>\n1.931<\/td>\n1.256<\/td>\n<\/tr>\n
4<\/td>\nno<\/td>\n29.69s<\/td>\n<\/td>\n0.945<\/td>\n<\/td>\n<\/td>\n<\/tr>\n
4<\/td>\nyes<\/td>\n1m 10s<\/td>\n43.57s<\/td>\n<\/td>\n2.358<\/td>\n1.467<\/td>\n<\/tr>\n
8<\/td>\nno<\/td>\n30.43s<\/td>\n<\/td>\n0.969<\/td>\n<\/td>\n<\/td>\n<\/tr>\n
8<\/td>\nyes<\/td>\n1m 17s<\/td>\n47.90s<\/td>\n<\/td>\n2.530<\/td>\n1.574<\/td>\n<\/tr>\n
16<\/td>\nno<\/td>\n32.39s<\/td>\n<\/td>\n1.031<\/td>\n<\/td>\n<\/td>\n<\/tr>\n
16<\/td>\nyes<\/td>\n1m 22s<\/td>\n52.67s<\/td>\n<\/td>\n2.532<\/td>\n1.626<\/td>\n<\/tr>\n
32<\/td>\nno<\/td>\n35.17s<\/td>\n<\/td>\n1.120<\/td>\n<\/td>\n<\/td>\n<\/tr>\n
32<\/td>\nyes<\/td>\n1m 26s<\/td>\n53.27s<\/td>\n<\/td>\n2.445<\/td>\n1.515<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n

Looking at the chart above, using sccache slowed down the clean build by anywhere from 23% (in the case of no -Zthreads<\/code>) to 153% (in the case of -Zthreads<\/code> set to 8 or 16), while providing a speed up to subsequent builds with identical flags and unchanged code\/toolchain only in the case of no -Zthreads<\/code> (6% speedup) and by 19% in the case of -Zthreads<\/code> set to 0<\/code> (its default value),\u00a0but it still managed to slow down even subsequent, clean builds with a fully primed cache as compared to the no-sccache baseline <\/strong>by from 25% (-Zthreads=2<\/code>) to 51% (-Zthreads=32<\/code>).<\/p>\n

Analyzing the benefits of -Zthreads<\/code> is much harder. Looking at the cases with -Zthreads<\/code> but no sccache<\/code>, it appears that with a heavily saturated pipeline (building all project dependencies from scratch in non-incremental mode affords a lot of opportunities for parallelized codegen), the use of -Zthreads<\/code> can provide at best<\/em> a very modest 5% speed up to build times (in the case of -Zthreads=4<\/code>) while actually slowing down\u00a0<\/em>compile times by 14% (the soon-to-be-default -Zthreads=0<\/code>) and 12% (with -Zthreads=16<\/code>).3<\/a><\/sup><\/p>\n

It’s interesting to note that the rust developers were rather more ebullient when introducing this feature<\/a> to the world, claiming wins of up to 50% with -Zthreads=8<\/code> and suggesting that lower levels of parallelism would see lower speedups (the opposite of what I saw, where using 8 threads provided about half the benefit of using 4, and going any higher caused slow-downs rather than speed-ups). Note that I was compiling in the default dev\/debug profile above, so maybe I should try and see what happens in release mode, though I would think the architectural limitations would persist.<\/p>\n

Back to sccache, though.<\/p>\n

One of the open source projects I contribute to most is fish-shell<\/a>, which recently underwent a complete port from C++ to Rust (piece-by-piece while still passing all tests every step of the way). Some day I want to write at length about that experience, but the reason I’m bringing this up is because my fish experience has taught me that some things that are normally very fast under Linux can run much slower than expected under WSL, primarily due to I\/O constraints caused by the filesystem virtualization layer. I haven’t dug into it yet, but going with the working theory that reading\/writing some 550-800 MiB4<\/a><\/sup> was slowing down my builds (even with the code and the cache located on two separate NVMe drives), I moved on to my other machine (where I’m running Linux natively).<\/p>\n

Running the same benchmark with the same versions of rustc<\/code> and sccache<\/code> on the 4-core\/8-thread Intel Xeon E3-1545M v5 (Skylake) with 32GiB of DDR3 RAM gave the following results, which were much<\/em> more in line with my expectations for sccache (though even more disappointing when it came to the frontend parallelization flag):<\/p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
-Zthreads<\/th>\nsccache<\/th>\nTime<\/th>\nTime (2nd Run)<\/th>\n<\/tr>\n<\/thead>\n
unset<\/td>\nno<\/td>\n1m 10s<\/td>\n<\/td>\n<\/tr>\n
unset<\/td>\nyes<\/td>\n1m 13s<\/td>\n14.12s<\/td>\n<\/tr>\n
0<\/td>\nno<\/td>\n1m 12s<\/td>\n<\/td>\n<\/tr>\n
0<\/td>\nyes<\/td>\n1m 18s<\/td>\n14.26s<\/td>\n<\/tr>\n
2<\/td>\nno<\/td>\n1m 10s<\/td>\n<\/td>\n<\/tr>\n
2<\/td>\nyes<\/td>\n1m 17s<\/td>\n14.20s<\/td>\n<\/tr>\n
4<\/td>\nno<\/td>\n1m 11s<\/td>\n<\/td>\n<\/tr>\n
4<\/td>\nyes<\/td>\n1m 15s<\/td>\n14.44s<\/td>\n<\/tr>\n
6<\/td>\nno<\/td>\n1m 12s<\/td>\n<\/td>\n<\/tr>\n
6<\/td>\nyes<\/td>\n1m 16s<\/td>\n14.90s<\/td>\n<\/tr>\n
8<\/td>\nno<\/td>\n1m 14s<\/td>\n<\/td>\n<\/tr>\n
8<\/td>\nyes<\/td>\n1m 20s<\/td>\n15.04s<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

Here at last are the sccache<\/code> results I was expecting! A maximum slowdown of about 8% for uncached builds and speedups of about 80% across the board for a (clean) rebuild of the same project immediately after caching the artifacts.5<\/a><\/sup><\/p>\n

As for -Zthreads<\/code>, the results here are consistent with what we saw above, at least once you take into account the fact that there are significantly fewer cores\/threads to distribute the parallelized frontend work across. But we’re left with the same conclusion that at times when the CPU is already handling a high degree of concurrency with jobserver already saturated with work from the compilation pipeline across multiple compilation units from independent crates, adding further threads to the mix ends up hurting overall performance (to the tune of 14% in the worst case, when -Zthreads<\/code> is set to total number of cores). The good news is that adding just a slight degree of parallelization with -Zthreads=2<\/code> doesn’t hurt build times in this worst-case scenario and likely helps when the available threads aren’t already saturated with more work than they can handle, so that at least seems to be a safe value for the option for now.<\/p>\n

I would have expected<\/em> that -Zthreads<\/code> wouldn’t “unilaterally” dictate the number of chunks frontend work was being broken up into. While I’m sure it integrates nicely with jobserver to prevent an insane number of threads from being spawned and overwhelming the machine, it would seem that dividing the frontend work into n<\/em> chunks when there aren’t n<\/em> threads immediately available ends up hurting overall build performance. So in that sense, I suppose it would be better if -Zthreads<\/code> were a hint of sorts, treated as a “max chunks” limit, and if introspection of available threads happened before<\/em> the decision was made to chunk the available work (and to what extent) so that the behavior of -Zthreads<\/code>, even with a hard-coded number, would hopefully only ever improve build times and never hurt.<\/p>\n

\n

If you would like to receive a notification the next time we release a rust library, publish a crate, or post some rust-related developer articles, you can subscribe below. Note that you'll only get notifications relevant to rust programming and development by NeoSmart Technologies. If you want to receive email updates for all NeoSmart Technologies posts and releases, please sign up in the sidebar to the right instead.<\/em><\/p>\n