Since gen_snapshot is lipo'd everywhere now, it should be safe to both
build with either arm64 or intel macs, and to use the arm64 native clang
toolchain on arm64 macs instead of the intel toolchain under Rosetta.
Related to https://github.com/flutter/flutter/issues/103386
Part of https://github.com/flutter/flutter/issues/147883.
Excluding Fuchsia and the Web SDK, the remaining scripts are:
```sh
$ ./tools/find_pubspecs_to_workspacify.sh
/Users/matanl/Developer/engine/src/flutter/impeller/tessellator/dart/pubspec.yaml
/Users/matanl/Developer/engine/src/flutter/sky/packages/sky_engine/pubspec.yaml
/Users/matanl/Developer/engine/src/flutter/shell/vmservice/pubspec.yaml
/Users/matanl/Developer/engine/src/flutter/lib/snapshot/pubspec.yaml
/Users/matanl/Developer/engine/src/flutter/lib/gpu/pubspec.yaml
/Users/matanl/Developer/engine/src/flutter/lib/web_ui/pubspec.yaml
/Users/matanl/Developer/engine/src/flutter/flutter_frontend_server/pubspec.yaml
```
These could be the trickiest, so I'm doing them separately.
This PR does a few things. Mainly it makes the builds in
mac_host_engine.json each build fewer targets to increase parallelism
for post-submit and release builds. To ensure that increasing
parallelism doesn't lead to capacity issues, this change also allows the
mac_host_engine.json builds to run on either intel or arm64 macOS hosts.
Finally, when building on an arm64 macOS host to target macOS, this PR
changes the `gn` script to ensure that the arm64 native clang toolchain
will be used.
To keep mac_host_engine.json focused on builds that produce artifacts,
this PR also moves tests from that file into the ill-named
mac_unopt.json. In a subsequent PR, I'll rename all the *_unopt.json
files to *_tests.json or something similar.
The artifacts produced by these builds are passing framework presubmit
checks in https://github.com/flutter/flutter/pull/152345.
Replaces and closes https://github.com/flutter/engine/pull/53997.
This PR migrates the following packages to the pub workspace:
- testing/benchmark
- testing/dart
- testing/litetest
- testing/pkg_test_demo
- testing/scenario_app
- testing/skia_gold_client
- testing/symbols
- tools/golden_tests_harvester
- tools/pkg/process_fakes
It also makes minor changes to the Dart build rules in order to resolve the root package_config, instead of per-package.
I am _not_ totally confident of the `_embedder.yaml` change, but I also can't explain what is needed to continue analyzing `lib/ui` without analysis failures that all of the symbols in `dart:nativewrappers` are missing.
Previously we were appending Flutter's fork of third_party/pyyaml to the Python sys.path. In the case where a bot image or local install already has pyyaml installed elsewhere on the path, ours will fail to be picked up. Instead prepend to the path.
Issue spotted by jsimmons@.
Related: https://github.com/flutter/engine/pull/54001
[C++, Objective-C, Java style guides]: https://github.com/flutter/engine/blob/main/CONTRIBUTING.md#style
We were using a very old version of this library that broke with
Python3.12. This updates to a Python 3.12 compatible SHA.
The specific breakage at the former commit was that
`collections.Hashable` was deprecated and moved to
`collections.abc.Hashable` in Python 3.7. It was removed in Python 3.8.
For anyone on newer versions of Python3, such as Python 3.9.6 which
ships with macOS 15 Sequoia, this is broken.
The error on versions of Python newer than 3.8 is:
```
________ running 'python3 src/flutter/tools/pub_get_offline.py' in '/Users/chris/Developer/flutter/engine'
Traceback (most recent call last):
File "/Users/chris/Developer/flutter/engine/src/flutter/tools/pub_get_offline.py", line 167, in <module>
sys.exit(main())
^^^^^^
File "/Users/chris/Developer/flutter/engine/src/flutter/tools/pub_get_offline.py", line 157, in main
if not package_uses_workspace_resolution(package):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/chris/Developer/flutter/engine/src/flutter/tools/pub_get_offline.py", line 76, in package_uses_workspace_resolution
return yaml.safe_load(pubspec_file).get('resolution') == 'workspace'
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/chris/Developer/flutter/engine/src/flutter/tools/../third_party/pyyaml/lib3/yaml/__init__.py", line 75, in load
return loader.get_single_data()
^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/chris/Developer/flutter/engine/src/flutter/tools/../third_party/pyyaml/lib3/yaml/constructor.py", line 37, in get_single_data
return self.construct_document(node)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/chris/Developer/flutter/engine/src/flutter/tools/../third_party/pyyaml/lib3/yaml/constructor.py", line 46, in construct_document
for dummy in generator:
File "/Users/chris/Developer/flutter/engine/src/flutter/tools/../third_party/pyyaml/lib3/yaml/constructor.py", line 398, in construct_yaml_map
value = self.construct_mapping(node)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/chris/Developer/flutter/engine/src/flutter/tools/../third_party/pyyaml/lib3/yaml/constructor.py", line 204, in construct_mapping
return super().construct_mapping(node, deep=deep)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/chris/Developer/flutter/engine/src/flutter/tools/../third_party/pyyaml/lib3/yaml/constructor.py", line 126, in construct_mapping
if not isinstance(key, collections.Hashable):
^^^^^^^^^^^^^^^^^^^^
AttributeError: module 'collections' has no attribute 'Hashable'
Error: Command 'python3 src/flutter/tools/pub_get_offline.py' returned non-zero exit status 1 in /Users/chris/Developer/flutter/engine
```
## Pre-launch Checklist
- [X] I read the [Contributor Guide] and followed the process outlined
there for submitting PRs.
- [X] I read the [Tree Hygiene] wiki page, which explains my
responsibilities.
- [X] I read and followed the [Flutter Style Guide] and the [C++,
Objective-C, Java style guides].
- [ ] I listed at least one issue that this PR fixes in the description
above.
- [X] I added new tests to check the change I am making or feature I am
adding, or the PR is [test-exempt]. See [testing the engine] for
instructions on writing and running engine tests.
- [X] I updated/added relevant documentation (doc comments with `///`).
- [X] I signed the [CLA].
- [X] All existing and new tests are passing.
If you need help, consider asking for advice on the #hackers-new channel
on [Discord].
<!-- Links -->
[Contributor Guide]:
https://github.com/flutter/flutter/wiki/Tree-hygiene#overview
[Tree Hygiene]: https://github.com/flutter/flutter/wiki/Tree-hygiene
[test-exempt]:
https://github.com/flutter/flutter/wiki/Tree-hygiene#tests
[Flutter Style Guide]:
https://github.com/flutter/flutter/wiki/Style-guide-for-Flutter-repo
[C++, Objective-C, Java style guides]:
https://github.com/flutter/engine/blob/main/CONTRIBUTING.md#style
[testing the engine]:
https://github.com/flutter/flutter/wiki/Testing-the-engine
[CLA]: https://cla.developers.google.com/
[flutter/tests]: https://github.com/flutter/tests
[breaking change policy]:
https://github.com/flutter/flutter/wiki/Tree-hygiene#handling-breaking-changes
[Discord]: https://github.com/flutter/flutter/wiki/Chat
This directory consistes of patches that Chromium's maintainers of the Chromium ICU fork have already applied to the ICU source code.
Usually these are fix-forwards that have been applied to the ICU library after it has been released.
The patch files are kept to ensure that the patches are applied if we ever need to create a new branch, or update the major ICU version.
The patch files specifically do *not* get used during the build process.
Issues: https://github.com/flutter/flutter/issues/151116
[C++, Objective-C, Java style guides]: https://github.com/flutter/engine/blob/main/CONTRIBUTING.md#style
flatland_connection.cc used to allow an arbitrary number of acquire and release fences to be scheduled for each frame.
Sadly, Fuchsia has a limitation of (1) the number of total handles that can be sent per a FIDL call, but also (2) the Flatland protocol only supports sending up to 16 fences per each fence type.
Now, normally there should be very few scheduled fences per frame. But if frames get skipped, we could amass many fences which would then crash our attempts to send all of them to the Flatland `Present` endpoint.
This change introduces two fence multiplexer, which allow us to signal more than 16 fences per type, at a performance penalty. We expect to be able *not* to crash the FIDL subsystem using this approach, and may even be able to hobble along for a bit, until the fences issue is hopefully self-resolved.
That said, this issue seems to indicate there are frame scheduling problems elsewhere. But this is a fairly straightforward change to make without affecting the rest of the flatland code or integration, so we opt to do that first.
Issues: [#150136](https://github.com/flutter/engine/pull/53366)
- [] I updated/added relevant documentation (doc comments with `///`).
Removes reported errors in VSCode when using `clangd`
* Parsing a 21MB json string is not performant
* RegExp replacing all occurrences in said 21MB string is (~61ms)
* Fixes#147767
Revert label failed due to conflicts
`FlutterFragmentTest.java` was the only file that had merge issues, everything else is the output of `git revert 802e5d2cd3c9e73f336e3fe43487b64a5fdf98d8`
[C++, Objective-C, Java style guides]: https://github.com/flutter/engine/blob/main/CONTRIBUTING.md#style
We don't support 32-bit arm iOS builds anymore. But adding the --ios-cpu=arm64 appends the "_arm" to the out subdirectory. This just causes confusion since the ios_debug_unopt is still arm64.
Just remove the flag that does nothing.
Upgrades every `androidx` dependency in the `src/third_party/android_embedding_dependencies` bundle to the latest version, except the `lifecycle` group*. Tested running a couple of apps as well because when updating these dependencies in the past I've been able to build the engine but then flutter run fail when trying to run an app.
Fixes https://github.com/flutter/flutter/issues/129307, also unblocks a feature that will eventually be needed for Scribe.
[*]`2.8.0` is the latest there, but I ran into an issue with dexing when I tried to upgrade, due to b/336164417, an AGP bug that had its fix backported to all >`8.0.0` versions, but we still support less than that so we will have to wait on that upgrade.
[C++, Objective-C, Java style guides]: https://github.com/flutter/engine/blob/main/CONTRIBUTING.md#style
Android only for now but this avoids rebuilding a bunch of test targets that aren't necessary to do a `flutter run` with a custom engine.
I need to fill in some blanks for other platforms before landing.
This flag as originally introduced to see if it made the GN step slower. It did not do so in any measurable manner and the default was flipped to true. AFAICT, no one disables and its good to have tracelogs to debug GN slowdowns. Remove the flag.
Removes impeller_capture, impeller_use_prebuilt_impellerc, impeller_use_prebuilt_scenec, and impeller_trace_canvas.
Also removes setting these flags from ./flutter/tools/gn.
Closes https://github.com/flutter/flutter/issues/147666.
This is a large change, I'd be happy to either review synchronously or
change commands one at a time, but I think this is overall the right
approach. I also didn't see any reason to reuse the `BuildRunner` code
for these commands, the flow is basically:
```mermaid
graph LR
A[et targetsOrPatterns] --> B[gn desc --format=json]
B --> C[existing code that runs ninja/workers]
```
Quick summary of changes:
- Introduced a [`Label` and `TargetPattern`][1] type to avoid awkward
string parsing/manipulation all over
- Replaced `gn_utils` (i.e. `targetsFromCommandLine` and friends) with
[an invocation of `gn desc`][2]
- Some tests were brittle in terms of expected output, I mostly left
them alone and wrote TODOs where applicable
Here is the fun part, results (and some manual integration tests):
```sh
# build, previously was 10-15s, is now ~3s
flutter % time et build -c host_debug_unopt_arm64 //flutter/impeller:impeller_unittests
[2024-05-14T19:05:55.163][macos/host_debug_unopt_arm64: GN]: OK
[2024-05-14T19:05:56.119][macos/host_debug_unopt_arm64: RBE startup]: Proxy started successfully.
[macos/host_debug_unopt_arm64: ninja] 0.0% (0/1) Regenerating ninja files[2024-05-14T19:05:57.309][macos/host_debug_unopt_arm64: ninja]: OK
[2024-05-14T19:05:57.573][macos/host_debug_unopt_arm64: RBE shutdown]: Actions completed: 0
$ENGINE/flutter/bin/et build -c host_debug_unopt_arm64 3.20s user 0.87s system 107% cpu 3.776 total
# test, also benefits from speedup, but I didn't time it
flutter % et test -c host_debug_unopt_arm64 //flutter/fml:fml_unittests
[2024-05-14T19:07:01.843][macos/host_debug_unopt_arm64: GN]: OK
[2024-05-14T19:07:02.707][macos/host_debug_unopt_arm64: RBE startup]: Proxy started successfully.
[2024-05-14T19:07:07.400][macos/host_debug_unopt_arm64: ninja]: 100.0% (3/3) LINK ./fml_unittests
[2024-05-14T19:07:07.404][macos/host_debug_unopt_arm64: ninja]: OK
[2024-05-14T19:07:07.748][macos/host_debug_unopt_arm64: RBE shutdown]: Actions completed: 1 (1 racing local)
OKAY: 7s.95ms //flutter/fml:fml_unittests
# query, also benefits from speedup
flutter % time et query targets -c host_debug_unopt_arm64
//flutter/display_list:display_list_benchmarks
//flutter/display_list:display_list_builder_benchmarks
# ... many targets omitted ...
$ENGINE/flutter/bin/et query targets -c host_debug_unopt_arm64 1.27s user 0.18s system 147% cpu 0.978 total
```
In other words, ~5x improvement on `et build` and `et query` is _much_
faster as well.
[1]:
https://github.com/flutter/engine/compare/main...matanlurey:engine:gn-label?expand=1#diff-ac008475ce9e209f4ecf6594b0cd48d12d86a1ffa0aa7d555375ff43b983eb2b
[2]:
https://github.com/flutter/engine/compare/main...matanlurey:engine:gn-label?expand=1#diff-ad116b8504dd0a500bf758a327c7052e5733a2b465d3295e35b74458d483276f
I started on a refactoring of
https://github.com/flutter/flutter/issues/147666, and was frustrated
with the JSON decoding of `gn desc --format=json`, so I used a similar
pattern from my micro-json library (https://pub.dev/packages/jsonut) and
encapsulated it as an extension type.
My favorite is the `JsonObject.map` function which replaces:
```dart
/// Construct a RunTarget from a JSON map.
factory RunTarget.fromJson(Map<String, Object> map) {
final List<String> errors = <String>[];
final String name = stringOfJson(map, _nameKey, errors)!;
final String id = stringOfJson(map, _idKey, errors)!;
final String targetPlatform =
stringOfJson(map, _targetPlatformKey, errors)!;
if (errors.isNotEmpty) {
throw FormatException('Failed to parse RunTarget: ${errors.join('\n')}');
}
return RunTarget._(name, id, targetPlatform);
}
```
... with:
```dart
/// Construct a RunTarget from a JSON map.
factory RunTarget.fromJson(Map<String, Object> map) {
return JsonObject(map).map((JsonObject json) => RunTarget._(
json.string(_nameKey),
json.string(_idKey),
json.string(_targetPlatformKey),
));
}
```
The size of the LTO build of the engine with the dylib compressed is as follows:
```sh
$ du -k libFlutter*
5236 libFlutter.dylib.tar.gz
4324 libFlutterSlimpeller.dylib.tar.gz
```
Sizes are in KiB. This represents a binary size reduction of 17.41% of the compressed artifacts. The compression ratios will likely differ based on the compression scheme.
Uncompressed, the sizes are:
```sh
$ du -k libFlutter*
16920 libFlutter.dylib
14044 libFlutterSlimpeller.dylib
```
This represents a binary size reduction of 16.99% which is in the same ballpark.
The really mucky bit was backing out the raster cache and persistent cache. I want to clean that up in a later patch so that those TUs are part of a separate submodule.
Opting out of Impeller will lead to a fatal log at startup saying the opt-out is disallowed.
Fixes https://github.com/flutter/flutter/issues/126606
This PR changes from using a different GN pool for each non-compiler tool, to using one GN pool for everything.
Additionally, since we are no longer linking remotely in any configuration, this PR uses the pool for linking.
ObjC and impellerc tasks also do not (yet) run on RBE, so they are also now governed by the pool.
Needs https://github.com/flutter/buildroot/pull/856
For illustrative purposes:
```sh
$ et build | grep '.*'
```
... should still get line-per-line status updates, but it does not without this patch.
It's hard to write tests because of global state, so I've declined to do so at the moment.
Closes https://github.com/flutter/flutter/issues/147903.
