Syncing Rust GCC backend or how to test Murphy's law

This blog post is about the Rust GCC backend (not to be confused with gccrs which is a Rust front-end for the GCC compiler), how we synchronize its repository with Rust's and how everything went so wrong that it took us 2 months to be able to finally make it. A good illustration of Murphy's law:

Anything that can go wrong will go wrong.

How the GCC backend is developped

The Rust GCC backend is developped in its own repository: https://github.com/rust-lang/rustc_codegen_gcc/. It allows us to experiment things without having to worry about breaking Rust's CI. Meaning these changes are for now only present in the GCC backend repository.

Since it's a code generator backend (I'll abbreviate it as "codegen" from now on) of the Rust compiler, its source code is also present in the Rust compiler repository: https://github.com/rust-lang/rust/tree/main/compiler/rustc_codegen_gcc. This folder contains the exact same content as our repository... most of the time. When the compiler adds a new feature, clean things up, etc, it sometimes need to be reflected in the codegens: LLVM, cranelift and GCC. And so we end up with changes only present in the Rust repository.

So both repositories have unique changes, and we need to merge them at some point so both repositories are in sync with each other.

How syncing work

Now comes the "fun" part: syncing changes between these two repositories. We use the git subtree feature. Never heard about it? No surprise, it's an incomplete feature (doesn't work well with big repositories). You can see the pull request fixing it on the git repository here. So you need to clone the git repository, checkout on this branch, build this git version and have it somewhere locally to use it when needed. We have instructions on how to do all that here.

So once you have the patched git ready, next step is to merge the Rust repository changes into the rustc_codegen_gcc repository:

# Current folder is rust's repository.
git-subtree push -P compiler/rustc_codegen_gcc/ ../rustc_codegen_gcc/ sync_branch_name
cd ../rustc_codegen_gcc
git checkout master
git pull
git checkout sync_branch_name
git merge master

Once Rust's changes are merged into the GCC backend, we can send rustc_codegen_gcc changes into Rust's:

# Current folder is rust's repository.
git pull origin master
git checkout -b subtree-update_cg_gcc_YYYY-MM-DD
git-subtree pull --prefix=compiler/rustc_codegen_gcc/ https://github.com/rust-lang/rustc_codegen_gcc.git master
git push

Since rustc_codegen_gcc targets a specific GCC version (our own fork which includes unmerged changes from upstream GCC), we also need to update the Rust's GCC submodule:

# Current folder is rust's repository.
cd src/gcc
git fetch
git checkout $(cat compiler/rustc_codegen_gcc/libgccjit.version)

We commit the change, we check it compiles fine and then we can push.

One last thing to do: once the pull request on the Rust repository is merged, we merge the merge commit back into the rustc_codegen_repository:

# Current folder is rust's repository.
git-subtree push -P compiler/rustc_codegen_gcc/ ../rustc_codegen_gcc/ sync_branch_name

Finally done!

Now time to talk about the last sync and why I mentioned Murphy's law.

So far so good

So, 24th of July 2026, I opened #159844. First issue we encountered: license issue. We copied a file used to know which features some CPUs had. After (legal) discussions, this was solved on the 31st of July.

In parallel we had some CI mirroring issues because the new GCC version we used needed some "more up-to-date" packages. To reduce CI flaky error for network, Rust infra provides a mirror which is often much more reliable than directly downloading from an external source (doesn't change location and a lot less traffic too).

When we update the GCC version, the Rust's CI re-build it completely so it can be downloaded by Rust developers who work on the GCC backend. And the new GCC version needed a newer make than the one available in the docker image we used. So: we add the new make source code to our mirror, we build it into our docker image and then we build GCC as we did before.

With this, all fixed! It got merged on the 3rd of August. 3 issues so far, not great but not the worst. So far so good.

Now as you might have guessed, things didn't stop there.

Ever heard of side effects?

As soon as it was merged, I got pinged on zulip on the infra team channel because the CI was completely broken and error messages seemed related to GCC. So we decided the revert the sync pull request. Of course, when trying to revert something using the github UI:

Revert failed: Repository rule violations found

Cannot create ref due to creations being restricted.

So after doing it by hand: #160468, CI was fixed, everything went back to normal. Except the sync is still not done and we still had no clue why the CI succeeded for the sync pull request. After some investigation, we realized that the built GCC didn't have some features enabled, making some tests fail. Big question was: why did the sync CI succeeded then? Well, that allowed us to discover a bug in the CI caching. So once the caching issue was fixed, only one bug remains: the GCC build was done incorrectly.

Interestingly enough, when you build GCC, it configures itself depending on the tools available on your system. If your linker doesn't support retain, then this feature is disabled. Recently we added support for "externally implementable items" in the GCC backend... which uses retain (if you're interested about this, the pull requests for this feature in GCC backend are gcc#88, gcc#89, cg_gcc#921 and cg_gcc#962). So the related tests were unignored for the GCC backend in the Rust's repository, and failed.

So the first thing to do was to fix the build of GCC in our CI by installing a newer version of binutils, which I did in #161006 on the 12th of August. After a lot of tries, we were able to make it work and it was merged on the 17h of August.

Last blocker removed! Oh wait. It broke miri build. Reverting!

Instead of installing it in replacement of the installed binutils in docker, why not only using it for the GCC build to limit its scope? Let's try that! Done in #161243 and merged on the 21st of August.

... But again miri seems broken! Luckily this time it was only bad timing: it was actually another unrelated pull request updating the cc crate which happened to be merged just after ours. Big relief.

So much time has passed since then that we needed to sync again changes from the Rust repository into rustc_codegen_gcc repository. So we opened cg_gcc#959. Aaaaand... it failed. A change in Rust's bootstrap broke the handling of the sysroot (in short: where rustc is supposed to look for libraries) and could not find GCC's library anymore. It was fixed in #162152 on the 3rd of September.

Time to try for a new Rust to rustc_codegen_gcc sync: cg_gcc#972, which failed too. Problem this time is that we need to use the same LLVM version in rustc_codegen_gcc as in Rust's because we use LLVM tools for some tests (FileCheck for assembly tests for example).

So after an update of LLVM and of rust version we use to build rustc_codegen_gcc, we opened a new sync: cg_gcc#974. This time it worked! \o/ It was merged on the 8th of September.

So time to try again the subtree sync (rustc_codegen_gcc to Rust) with #162499. CI failed because of some run-make tests (a testsuite in Rust). Ok, that one is on me. Between the last fix and this sync, I fixed a bug in Rust's bootstrap which was preventing other codegen backends than LLVM from running the run-make tests in #162482.

So taking that into account, added a commit to ignore these tests for the GCC backend until it's fixed on our side.

Now let's run the CI once again... Failed again! This time it's because in the meantime, we updated the GCC version in rustc_codegen_gcc and it requires a newer make version. So be it! To make sure it didn't break things to update make version, I opened #163031 with just this change.

No problem apparently. Oh well, time to try to merge the sync then!

On the 22nd of September, #162499 was merged and the sync was finally finished!

What's next

This has been quite the unlucky stroke! But along the way we improved Rust's CI, uncovered caching issues, fixed some testsuites bugs/limitations. So all in all, even if exhausting, it was quite rewarding.

However, this sync underlined that this process has too many issues and that we should move to something else. Luckily, a few subtrees have already migrated to josh so we're planning to do the same for rustc_codegen_gcc soon.

Words of the end

Cat was too asleep to provide some luck.