content aware hash was using --format which isn't present in older, but
still supported operating systems. this pr removes the format string
which basically reduces the output before hasing to:
```shell
100644 blob 198d80926b6e873c327f71350a0cdefee6a8402f DEPS
040000 tree 139c1f10f92e4b9d4ac3ec7d4d27b2aa9775c5cd engine
```
this format is still stable across all platforms and passed into `git
hash-object` - which produces the actual hash fingerprint of the engine.
safety: this is the only scripts that produce this hash, so all
downstream consumers keep consuming a sha1 output. Since this changes
the sha, an engine version shouldn't exist for it and cocoon will build
the artifacts for it.
fixes: #175265
The `content_aware_hash.sh` script determines which version of the
engine to use. For local development, it uses the merge-base with the
remote tracking branch to avoid unnecessary rebuilds.
However, when using `jj`, the underlying git repository is in a detached
HEAD state. The script was incorrectly interpreting this as a CI
environment and was not calculating the hash based on the merge-base,
leading to incorrect engine versions and failed Dart SDK downloads.
This change modifies the script to differentiate between a local
detached HEAD state (like with `jj`) and a CI environment by checking
for the `LUCI_CI` environment variable. This ensures the correct engine
hash is generated for both local `jj` users and CI builds.
Here is an example of failing to download the Dart SDK before:
```
Downloading Darwin arm64 Dart SDK from Flutter engine f6ea244d7b75547c2c1a4613299b24dcebe3ce5c...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 258 100 258 0 0 858 0 --:--:-- --:--:-- --:--:-- 857
[/Users/het/Projects/flutter/bin/cache/dart-sdk-darwin-arm64.zip]
End-of-central-directory signature not found. Either this file is not
a zipfile, or it constitutes one disk of a multi-part archive. In the
latter case the central directory and zipfile comment will be found on
the last disk(s) of this archive.
unzip: cannot find zipfile directory in one of /Users/het/Projects/flutter/bin/cache/dart-sdk-darwin-arm64.zip or
/Users/het/Projects/flutter/bin/cache/dart-sdk-darwin-arm64.zip.zip, and cannot find /Users/het/Projects/flutter/bin/cache/dart-sdk-darwin-arm64.zip.ZIP, period.
It appears that the downloaded file is corrupt; please try again.
If this problem persists, please report the problem at:
https://github.com/flutter/flutter/issues/new?template=01_activation.yml
```
## Pre-launch Checklist
- [x] I read the [Contributor Guide] and followed the process outlined
there for submitting PRs.
- [x] I read the [Tree Hygiene] wiki page, which explains my
responsibilities.
- [x] I read and followed the [Flutter Style Guide], including [Features
we expect every widget to implement].
- [x] I signed the [CLA].
- [ ] I listed at least one issue that this PR fixes in the description
above.
- [x] I updated/added relevant documentation (doc comments with `///`).
- [x] I added new tests to check the change I am making, or this PR is
[test-exempt].
- [x] I followed the [breaking change policy] and added [Data Driven
Fixes] where supported.
- [x] All existing and new tests are passing.
If you need help, consider asking for advice on the #hackers-new channel
on [Discord].
**Note**: The Flutter team is currently trialing the use of [Gemini Code
Assist for
GitHub](https://developers.google.com/gemini-code-assist/docs/review-github-code).
Comments from the `gemini-code-assist` bot should not be taken as
authoritative feedback from the Flutter team. If you find its comments
useful you can update your code accordingly, but if you are unsure or
disagree with the feedback, please feel free to wait for a Flutter team
member's review for guidance on which automated comments should be
addressed.
<!-- Links -->
[Contributor Guide]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#overview
[Tree Hygiene]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md
[test-exempt]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#tests
[Flutter Style Guide]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md
[Features we expect every widget to implement]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md#features-we-expect-every-widget-to-implement
[CLA]: https://cla.developers.google.com/
[flutter/tests]: https://github.com/flutter/tests
[breaking change policy]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#handling-breaking-changes
[Discord]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Chat.md
[Data Driven Fixes]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Data-driven-Fixes.md
The content hash doesn't exist for local engine changes, except for on
CI. If we detect we're on a branch with committed or uncommitted changes
to engine files; use "master".
towards #171790
re-land attempt for #173114 (try 2)
re-land attempt for #172792 (original)
The first commit in this PR is the previously LGTM'd changes for the
above; with tests.
The second commit is the critical change to make this work in post
submits (fixes#173143). It turns out that while LUCI reports the GitHub
private branches; our recipes directly checkout the git sha. This
matters because the content scripts couldn't determine the branch name
and the rev-parse was just HEAD. This lead the scripts down the
merge-base logic, which returns the previous commit.
A test was added specifically for this.
Alternatively to this change, we could have checked for LUCI_CONTEXT
being present in the environment. This is checked by Flutter tools in
some cases, but not by any other scripts in `bin/internal`. The downside
to checking HEAD: if you have a local branch with engine changes and you
move back a revision - `dart`/`flutter` invocations will generate the
hash for your local changes and fail.
<!-- start_original_pr_link -->
Reverts: flutter/flutter#173114
<!-- end_original_pr_link -->
<!-- start_initiating_author -->
Initiated by: jtmcdole
<!-- end_initiating_author -->
<!-- start_revert_reason -->
Reason for reverting: there is still another problem with the merge
queue causing the content hash to be different:
git revision: e13dd5384cc0db576291c2925852879f1111f284
actual hash: 4b6f7b0f9849efaa59f515c8e95f3f27a6eb2ffb
hash in the queue? 9e5b2eef4ba79b15b4f80dbba812d199d262366f
<!-- end_revert_reason -->
<!-- start_original_pr_author -->
Original PR Author: jtmcdole
<!-- end_original_pr_author -->
<!-- start_reviewers -->
Reviewed By: {matanlurey, chingjun}
<!-- end_reviewers -->
<!-- start_revert_body -->
This change reverts the following previous change:
The content hash doesn't exist for local engine changes, except for on
CI. If we detect we're on a branch with committed or uncommitted changes
to engine files; use "master".
towards #171790
re-land attempt for #172792 with the following changes:
1. content_aware_hash.(ps1|sh) now consider multiple branches to choose
between HEAD and merge-base.
2. content_aware_hash_test.dart updated for these new requirements
3. content_aware_hash_test.dart allows for forcing powershell on mac for
testing
4. updated docs/tool/Engine-artifacts.md documentation.
## Pre-launch Checklist
- [x] I read the [Contributor Guide] and followed the process outlined
there for submitting PRs.
- [x] I read the [Tree Hygiene] wiki page, which explains my
responsibilities.
- [x] I read and followed the [Flutter Style Guide], including [Features
we expect every widget to implement].
- [x] I signed the [CLA].
- [x] I listed at least one issue that this PR fixes in the description
above.
- [x] I updated/added relevant documentation (doc comments with `///`).
- [x] I added new tests to check the change I am making, or this PR is
[test-exempt].
- [ ] I followed the [breaking change policy] and added [Data Driven
Fixes] where supported.
- [x] All existing and new tests are passing.
<!-- end_revert_body -->
Co-authored-by: auto-submit[bot] <flutter-engprod-team@google.com>
The content hash doesn't exist for local engine changes, except for on
CI. If we detect we're on a branch with committed or uncommitted changes
to engine files; use "master".
towards #171790
re-land attempt for #172792 with the following changes:
1. content_aware_hash.(ps1|sh) now consider multiple branches to choose
between HEAD and merge-base.
2. content_aware_hash_test.dart updated for these new requirements
3. content_aware_hash_test.dart allows for forcing powershell on mac for
testing
4. updated docs/tool/Engine-artifacts.md documentation.
## Pre-launch Checklist
- [x] I read the [Contributor Guide] and followed the process outlined
there for submitting PRs.
- [x] I read the [Tree Hygiene] wiki page, which explains my
responsibilities.
- [x] I read and followed the [Flutter Style Guide], including [Features
we expect every widget to implement].
- [x] I signed the [CLA].
- [x] I listed at least one issue that this PR fixes in the description
above.
- [x] I updated/added relevant documentation (doc comments with `///`).
- [x] I added new tests to check the change I am making, or this PR is
[test-exempt].
- [ ] I followed the [breaking change policy] and added [Data Driven
Fixes] where supported.
- [x] All existing and new tests are passing.
<!-- start_original_pr_link -->
Reverts: flutter/flutter#172792
<!-- end_original_pr_link -->
<!-- start_initiating_author -->
Initiated by: jtmcdole
<!-- end_initiating_author -->
<!-- start_revert_reason -->
Reason for reverting: almost, but not quite right. android builders
generate the wrong hash because they are on custom branches.
<!-- end_revert_reason -->
<!-- start_original_pr_author -->
Original PR Author: jtmcdole
<!-- end_original_pr_author -->
<!-- start_reviewers -->
Reviewed By: {bdero, matanlurey}
<!-- end_reviewers -->
<!-- start_revert_body -->
This change reverts the following previous change:
The content hash doesn't exist for local engine changes, except for on
CI. If we detect we're on a branch with committed or uncommitted changes
to engine files; use "master".
towards #171790
<!-- end_revert_body -->
Co-authored-by: auto-submit[bot] <flutter-engprod-team@google.com>
The content hash doesn't exist for local engine changes, except for on
CI. If we detect we're on a branch with committed or uncommitted changes
to engine files; use "master".
towards #171790
reland of https://github.com/flutter/flutter/pull/171927
towards https://github.com/flutter/flutter/issues/171790
Use the content aware hash when downloading engine artifacts. These are
currently produced when changes to DEPs, engine/, or the release file
are changed in a not-seen-before way.
We can eventually remove engine.version being tracked in release
branches as an optimization.
FLUTTER_PREBUILT_ENGINE_VERSION will stay for overriding the
engine.version for testing. Though Cocoon does not need to actually set
it for framework only PRs anymore.
fxies post submit tests for content_hash download:
SHARD=framework_tests SUBSHARD=misc dart --enable-asserts
dev/bots/test.dart
SHARD=tool_host_cross_arch_tests dart --enable-asserts
dev/bots/test.dart
```
╔═╡ERROR https://github.com/flutter/flutter/pull/1╞════════════════════════════════════════════════════════════════════
║ Expected "Flutter Engine Version: 45b085ab2118f30441ae6e2daab4b8eb3e4a063c", but found "Flutter Engine Version: 407c051ec9".
╚═══════════════════════════════════════════════════════════════════════════════
```
```
Expected: contains '45b085ab2118f30441ae6e2daab4b8eb3e4a063c'
Which: does not contain '45b085ab2118f30441ae6e2daab4b8eb3e4a063c'
```
<!-- start_original_pr_link -->
Reverts: flutter/flutter#171927
<!-- end_original_pr_link -->
<!-- start_initiating_author -->
Initiated by: matanlurey
<!-- end_initiating_author -->
<!-- start_revert_reason -->
Reason for reverting: Broke the tree, due to naughty tests but too
intensive to fix forward.
<!-- end_revert_reason -->
<!-- start_original_pr_author -->
Original PR Author: jtmcdole
<!-- end_original_pr_author -->
<!-- start_reviewers -->
Reviewed By: {matanlurey, chingjun}
<!-- end_reviewers -->
<!-- start_revert_body -->
This change reverts the following previous change:
towards #171790
Use the content aware hash when downloading engine artifacts. These are
currently produced when changes to DEPs, engine/, or the release file
are changed in a not-seen-before way.
We can eventually remove `engine.version` being tracked in release
branches as an optimization.
`FLUTTER_PREBUILT_ENGINE_VERSION` will stay for overriding the
engine.version for testing. Though Cocoon does not need to actually set
it for framework only PRs anymore.
<!-- end_revert_body -->
Co-authored-by: auto-submit[bot] <flutter-engprod-team@google.com>
towards #171790
Use the content aware hash when downloading engine artifacts. These are
currently produced when changes to DEPs, engine/, or the release file
are changed in a not-seen-before way.
We can eventually remove `engine.version` being tracked in release
branches as an optimization.
`FLUTTER_PREBUILT_ENGINE_VERSION` will stay for overriding the
engine.version for testing. Though Cocoon does not need to actually set
it for framework only PRs anymore.
This fixes https://github.com/flutter/flutter/pull/172184 by considering
the last commit to `release-candidate-branch.version` as the significant
commit.
Otherwise, operationally, it works the same. I tested this on 3.35 and
it works as expected (where the current does not).
Rolls forward https://github.com/flutter/flutter/pull/166717.
Does not copy the `README` or `DEPS` files, and instead uses synthetic
scratch files.
These files can change, so we can't possibly know how to hash them
consistently.
<!-- start_original_pr_link -->
Reverts: flutter/flutter#166717
<!-- end_original_pr_link -->
<!-- start_initiating_author -->
Initiated by: matanlurey
<!-- end_initiating_author -->
<!-- start_revert_reason -->
Reason for reverting: Broke tree on
[postsubmit](https://logs.chromium.org/logs/flutter/buildbucket/cr-buildbucket/8718108088961866849/+/u/test:_run_suite_safari-dart2js-canvaskit-engine/stdout)
<!-- end_revert_reason -->
<!-- start_original_pr_author -->
Original PR Author: jtmcdole
<!-- end_original_pr_author -->
<!-- start_reviewers -->
Reviewed By: {matanlurey}
<!-- end_reviewers -->
<!-- start_revert_body -->
This change reverts the following previous change:
1. Calculate the hash in only two places: `content_aware_hash.{ps1|sh}`
2. Call this from the workflow
3. Eventually call this from `update_engine_version.{ps1|sh}`
The files of import:
* `DEPS`: tracks third party dependencies related to building the engine
* `engine`: all the code in the engine folder
* `bin/internal/content_aware_hash.ps1`: script for calculating the hash
on windows
* `bin/internal/content_aware_hash.sh`: script for calculating the hash
on mac/linux
* `.github/workflows/content-aware-hash.yml`: github action for CI/CD
hashing
Tested on windows and mac:
```shell
PS C:\src\flutter> C:\src\flutter\bin\internal\content_aware_hash.ps1
c24231e276e0719738e175e0622e040ad21a7012
```
```shell
❯ ~/src/flutter/bin/internal/content_aware_hash.sh
c24231e276e0719738e175e0622e040ad21a7012
```
<!-- end_revert_body -->
Co-authored-by: auto-submit[bot] <flutter-engprod-team@google.com>
1. Calculate the hash in only two places: `content_aware_hash.{ps1|sh}`
2. Call this from the workflow
3. Eventually call this from `update_engine_version.{ps1|sh}`
The files of import:
* `DEPS`: tracks third party dependencies related to building the engine
* `engine`: all the code in the engine folder
* `bin/internal/content_aware_hash.ps1`: script for calculating the hash
on windows
* `bin/internal/content_aware_hash.sh`: script for calculating the hash
on mac/linux
* `.github/workflows/content-aware-hash.yml`: github action for CI/CD
hashing
Tested on windows and mac:
```shell
PS C:\src\flutter> C:\src\flutter\bin\internal\content_aware_hash.ps1
c24231e276e0719738e175e0622e040ad21a7012
```
```shell
❯ ~/src/flutter/bin/internal/content_aware_hash.sh
c24231e276e0719738e175e0622e040ad21a7012
```
`update_engine_version.{ps1|sh}` needs to operate on a foreign
repository.
When flutter is run in a git-hook, these environment variables will
override our git calls location and corrupt the install.
fixes: #165390
<!-- start_original_pr_link -->
Reverts: flutter/flutter#164317
<!-- end_original_pr_link -->
<!-- start_initiating_author -->
Initiated by: matanlurey
<!-- end_initiating_author -->
<!-- start_revert_reason -->
Reason for reverting: `bin/cache` does not exist on a fresh checkout,
and `echo bin/cache/...` will fail as a result.
This blocked the google3 roll, but would also break new checkouts of
Flutter, for regular users/contributors.
<!-- end_revert_reason -->
<!-- start_original_pr_author -->
Original PR Author: matanlurey
<!-- end_original_pr_author -->
<!-- start_reviewers -->
Reviewed By: {jtmcdole}
<!-- end_reviewers -->
<!-- start_revert_body -->
This change reverts the following previous change:
Towards https://github.com/flutter/flutter/issues/164315.
This PR just writes `bin/cache/engine.stamp` identically to how
`bin/internal/engine.version` would otherwise be written, with a caveat
that _if_ `engine.version` is tracked, it is now _copied_ to
`bin/cache/engine.stamp`.
After this lands, I'll send PRs to update tooling that looks for
`engine.version` and give a heads up to the larger team (i.e. Dart HH
bot or whomever we will break by doing this).
<!-- end_revert_body -->
Co-authored-by: auto-submit[bot] <flutter-engprod-team@google.com>
Towards https://github.com/flutter/flutter/issues/164315.
This PR just writes `bin/cache/engine.stamp` identically to how
`bin/internal/engine.version` would otherwise be written, with a caveat
that _if_ `engine.version` is tracked, it is now _copied_ to
`bin/cache/engine.stamp`.
After this lands, I'll send PRs to update tooling that looks for
`engine.version` and give a heads up to the larger team (i.e. Dart HH
bot or whomever we will break by doing this).
Towards https://github.com/flutter/flutter/issues/162201.
**NOTE**: This renames the environment variable to
`FLUTTER_PREBUILT_ENGINE_VERSION`.
---
We occasionally break ourselves, our users, and the Dart up (or is it
down? side-ways) stream repos (i.e. HHH) when we change how the
undocumented
[`update_engine_version.sh`](https://github.com/flutter/flutter/blob/master/bin/internal/update_engine_version.sh)
script, and it's Windows counterpart
[`update_engine_version.ps1`](https://github.com/flutter/flutter/blob/master/bin/internal/update_engine_version.ps1)
work, but have no way of knowing until N days/weeks later when someone
tells us.
For example,
<https://flutter-review.googlesource.com/c/recipes/+/62400>.
This is my attempt to encode "this is what you can guarantee by calling
this script".
It _still_ will be an internal only API that we might rev at any time,
but at least we:
1. Can tell the Dart team "this is tested and how it works"
2. If we want to change it, the tests will keep us from changing it
without informing folks it changed
These tests should (in theory) cover both Linux/MacOS and Windows.
/cc @a-siva
This auto-formats all *.dart files in the repository outside of the
`engine` subdirectory and enforces that these files stay formatted with
a presubmit check.
**Reviewers:** Please carefully review all the commits except for the
one titled "formatted". The "formatted" commit was auto-generated by
running `dev/tools/format.sh -a -f`. The other commits were hand-crafted
to prepare the repo for the formatting change. I recommend reviewing the
commits one-by-one via the "Commits" tab and avoiding Github's "Files
changed" tab as it will likely slow down your browser because of the
size of this PR.
---------
Co-authored-by: Kate Lovett <katelovett@google.com>
Co-authored-by: LongCatIsLooong <31859944+LongCatIsLooong@users.noreply.github.com>
This applied the improvements from
https://github.com/flutter/engine/pull/57206 to the flutter/flutter
version of that script (hopefully, we can deduplicate this soon with the
repo merge).
The script is modeled after a similar script in the engine (see engine's
[format.dart](https://github.com/flutter/engine/blob/main/ci/bin/format.dart)).
It identifies the files that have been changed and checks if their
formatting is correct. It also offers an option to correct formatting
(`--fix`) and an option to check the formatting of all files in the
repro (not just changed ones, `--all-files`).
When we are enforcing dart format this script will be called as part of
presubmit.
Useful in tooling, releases, etc - where Dart is available
Example usages:
```bash
# Monorepo world
❯ dart ./dev/tools/bin/engine_hash.dart -s head
226e13826c7253c968d798666f323b1f207979f8
# Non-monorepo world
❯ dart ./dev/tools/bin/engine_hash.dart -s head
Error calculating engine hash: Not in a monorepo
```
## Description
This adds a fake dependency to the docs generation for the `flutter_gpu` package so that it will be included in the docs even though Flutter itself doesn't depend on it directly.
## Related Issues
- Fixes https://github.com/flutter/flutter/issues/153320
## Tests
- Adding a test to the canary files that the docs generation script looks for. (Not yet added: still draft)
Fix is in the second commit. The logic here went out of sync with the logic in the snippets generator from https://github.com/flutter/assets-for-api-docs, whose version was bumped as part of this change.
Reverts flutter/flutter#144329
Initiated by: goderbauer
Reason for reverting: broke postsubmit doc generation.
Original PR Author: goderbauer
Reviewed By: {devoncarew, HansMuller, gspencergoog}
This change reverts the following previous change:
Original Description:
Dartpad doesn't have a "master" channel anymore, it got renamed to "main". Sadly, specifying "master" is now falling back to "stable" which breaks some of our examples in the docs that require a more current Flutter version, e.g. https://main-api.flutter.dev/flutter/material/TextButton-class.html
Dartpad doesn't have a "master" channel anymore, it got renamed to "main". Sadly, specifying "master" is now falling back to "stable" which breaks some of our examples in the docs that require a more current Flutter version, e.g. https://main-api.flutter.dev/flutter/material/TextButton-class.html
Part of work on [#101077](https://github.com/flutter/flutter/pull/141194). This is done as a separate PR to avoid a massive diff.
## Context
1. The `FakeCommand` class accepts a list of patterns that's used to match a command given to its `FakeProcessManager`. Since `FakeCommand` can match a list of patterns, not just specifically strings, it can be used to match commands where the exact value of some arguments can't (easily) known ahead of time. For example, a part of the tool may invoke a command with an argument that is the path of a temporarily file that has a randomly-generated basename.
2. The `FakeCommand` class provides on `onRun` parameter, which is a callback that is run when the `FakeProcessManager` runs a command that matches the `FakeCommand` in question.
## Issue
In the event that a `FakeCommand` is constructed using patterns, the test code can't know the exact values used for arguments in the command. This PR proposes changing the type of `onRun` from `VoidCallback?` to `void Function(List<String>)?`. When run, the value `List<String>` parameter will be the full command that the `FakeCommand` matched.
Example:
```dart
FakeCommand(
command: <Pattern>[
artifacts.getArtifactPath(Artifact.engineDartBinary),
'run',
'vector_graphics_compiler',
RegExp(r'--input=/.*\.temp'),
RegExp(r'--output=/.*\.temp'),
],
onRun: (List<String> command) {
final outputPath = (() {
// code to parse `--output` from `command`
})();
testFileSystem.file(outputPath).createSync(recursive: true);
},
)
```
