flutter_flutter/docs/engine/rbe/rbe.md

RBE for Flutter Engine Developers

Overview

This is documentation on setting up RBE for building the Flutter engine. It is expected to work for Googlers only on corp Linux, macOS, and Windows hosts, including cloudtop instances.

Getting started

The first step is ensure RBE is enabled in .gclient file. Add the entry "use_rbe": True in the custom_vars section if it is not already present.

Tip

: If your .gclient file was copied from engine/scripts/rbe.gclient, the entry will already be present.

.gclient should look like this:

solutions = [
  {
    # ...
    "custom_vars": {
      "use_rbe": True,
    },
  },
]

After making this edit, you must be authenticated as a Googler by CIPD so that the RBE configurations can be downloaded from the flutter_internal CIPD bucket:

cipd auth-login

After authentication successfully, run gclient sync -D.

gcloud

Before running an RBE build, you must be authenticated with the Google cloud project that owns the RBE worker pool. You'll need the gcloud SDK, which you can install by following the instructions at https://cloud.google.com/sdk/docs/install.

On macOS, before running the gcloud command ensure that python3 is on your path, and does not come from e.g. homebrew. The command which python3 should return /usr/bin/python3.

gcloud init --project flutter-rbe-prod

Execute the following to create application default credentials:

gcloud auth application-default login

Already using another cloud project or haven't refreshed in a while? Try:

gcloud config set project flutter-rbe-prod
gcloud auth application-default login

Running an RBE build

In the engine repo, all RBE builds must be initiated through the et tool whose entrypoint is the script //flutter/bin/et. This is so that the local RBE proxy is correctly initialized and shut down around invocations of ninja.

Listing builds

The builds available to the et tool are those specified by the build configuration json files under //flutter/ci/builders. A list of builds suitable for local development can also be printed by running et help build. The list of all builds can be printed by running et help build --verbose. Builds in the verbose list prefixed with ci/ or ci\ are exactly the builds run on CI with the same GN flags and Ninja targets.

Running builds

To run a build, pass the name of the configuration to the et build command:

et build -c host_debug

If RBE is working correctly, you should see logs like the following:

[2024-04-22T09:58:48.643][windows/host_debug: GN]: OK
[2024-04-22T09:58:59.361][windows/host_debug: RBE startup]: OK

To disable RBE in a build, pass the flag --no-rbe to the build command.

et build -c host_debug --no-rbe

Since LTO is slow and rarely useful in local builds, et disables it by default in all builds, even when it is specified by a build configuration. To enable it, pass the --lto flag to the build command.

et build -c host_debug --lto

Customizing builds

Beyond enabling/disabling RBE and LTO, the build command does not currently support customizing builds at the command line.

If you need custom GN flags or Ninja targets, then this can be achieved by making local edits to the build configuration json files in your checkout. In particular, the file //flutter/ci/builders/local_engine.json is intended to contain builds that are commonly used for local engine development.

If a configuration does not exist there, and will be needed by multiple people over a long period of time, consider checking in the new configuration so that it will be built on CI, which will keep the RBE cache warm for fast local builds.

On the other hand, if a build is intended to produce artifacts or run tests on CI, then it should not go in local_engine.json. Instead, it should go in the json file appropriate for its purpose. There are many examples available, so following suit is likely your best bet, but if you are unsure, ask in the hackers-engine Discord.

Maintaining Compatibility with RBE

RBE is sensitive to what paths in compile commands look like. In particular, all paths in compile commands must be relative paths, and those relative paths must all resolve to files within the engine source tree. In practice, this means that the GN function rebase_path() should only lack a second parameter when its result won’t be used in a compile or link command. There are unfortunately many examples in the tree where rebase_path() lacks a second parameter. However to be on the safe side a good rule of thumb is: do not add new usages of the single parameter rebase_path().

Troubleshooting

Error obtaining credentials

If you get the following error while running et build:

E0815 09:30:03.169505 1413247 main.go:147] Error obtaining credentials: application default credentials were invalid: could not get valid Application Default Credentials token: oauth2: cannot fetch token: 400 Bad Request

Check your ${HOME}/.config/gcloud/application_default_credentials.json to see if you are signed into another project / account. Run the following to reset the default credentials:

gcloud auth application-default login

Too many open files

For developers on a macOS device, if you get the following error while running et build:

ninja: fatal: pipe: Too many open files

Increase the maximum number of open files on your machine with the instructions here.

Slow builds

RBE builds can be slow for a few different reasons. The most common reason is likely to be that the remote caches are cold. When the caches are warm, a compile step consists only of downloading the compiled TU from the cache. When the caches are cold, the remote workers must run the compilation commands, which takes more time. If the worker pool is overloaded, compile commands may run locally instead, which will also be slower.

RBE builds can also be slow if your network connection is bandwidth constrained. Anecdotally, even with a warm cache, I have noticed slow builds from home due to RBE saturating my low-tier Comcast Business connection.

For developers on a macOS host device, ensure that you're using the same version of Xcode as is in use on the bots. The value of "Build version" returned by xcodebuild -version should match the sdk_version value set in ci/builders/local_engine.json for the build you're running.

For Googlers on a corp macOS device, both RBE and non-RBE builds can be slow due to various background and monitoring processes running. See here for how to disable some of them. You should also disable Spotlight scanning of the engine source directory as described here.

When RBE builds are slow, non-RBE builds may be faster, especially incremental builds. You can disable remote builds without invalidating your existing build by setting the environment variable RBE_exec_strategy=local.

Proxy status and debug logs

Warning

Since et will start and stop the local RBE proxy while performing a build, the following command will only work when a build is running.

The status of the local RBE proxy can be queried with the following command

buildtools/mac-arm64/reclient/reproxystatus

It will give output describing the number of actions completed and in progress, and the number of remote executions, local executions, and remote cache hits.

For example:

$ reproxystatus
Reproxy(unix:///tmp/reproxy.sock) is OK
Actions completed: 4405 (750 cache hits, 3075 racing locals, 580 racing remotes)
Actions in progress: 11
QPS: 12

The logs for RBE live under the system /tmp folder in the files /tmp/reproxy. {INFO,WARNING,ERROR} and /tmp/bootstrap.{INFO,WARNING,ERROR}.

In CI runs, the RBE logs are also available by following the links as in the screenshot below under teardown remote execution > collect rbe logs:

Dependency analysis failures

These logs can be useful to highlight failures in RBE’s dependency analysis. The dependency analysis can fail for a variety of reasons, but a common one during development is likely to be that the source file is really malformed somehow. This can be debugged by doing a local build with RBE turned off.

References

• Code for RBE (i.e. reproxy, rewrapper, bootstrap, etc.) lives in this GitHub repository. The tools are not well-documented, so the source code is the source of truth for the command line flags that they accept, for example.
• Internal-facing RBE migration guide is here. (Mostly focused on Chrome and Android, so not all parts are relevant to Flutter.)
• The version of RBE for local development is set in the DEPS file here. It needs to be manually rolled occasionally.
• The version of RBE used by CI is set in a LUCI recipe here. It also needs to be manually rolled occasionally.
• Googler-only RBE configuration files live in the CIPD bucket here. They need to be updated when we roll clang to a new version as described here.
• Flutter’s RBE worker pool is defined here.
• Using RBE for Engine clang-tidy is blocked on b/326591374.