content aware hash was using --format which isn't present in older, but
still supported operating systems. this pr removes the format string
which basically reduces the output before hasing to:
```shell
100644 blob 198d80926b6e873c327f71350a0cdefee6a8402f DEPS
040000 tree 139c1f10f92e4b9d4ac3ec7d4d27b2aa9775c5cd engine
```
this format is still stable across all platforms and passed into `git
hash-object` - which produces the actual hash fingerprint of the engine.
safety: this is the only scripts that produce this hash, so all
downstream consumers keep consuming a sha1 output. Since this changes
the sha, an engine version shouldn't exist for it and cocoon will build
the artifacts for it.
fixes: #175265
The `content_aware_hash.sh` script determines which version of the
engine to use. For local development, it uses the merge-base with the
remote tracking branch to avoid unnecessary rebuilds.
However, when using `jj`, the underlying git repository is in a detached
HEAD state. The script was incorrectly interpreting this as a CI
environment and was not calculating the hash based on the merge-base,
leading to incorrect engine versions and failed Dart SDK downloads.
This change modifies the script to differentiate between a local
detached HEAD state (like with `jj`) and a CI environment by checking
for the `LUCI_CI` environment variable. This ensures the correct engine
hash is generated for both local `jj` users and CI builds.
Here is an example of failing to download the Dart SDK before:
```
Downloading Darwin arm64 Dart SDK from Flutter engine f6ea244d7b75547c2c1a4613299b24dcebe3ce5c...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 258 100 258 0 0 858 0 --:--:-- --:--:-- --:--:-- 857
[/Users/het/Projects/flutter/bin/cache/dart-sdk-darwin-arm64.zip]
End-of-central-directory signature not found. Either this file is not
a zipfile, or it constitutes one disk of a multi-part archive. In the
latter case the central directory and zipfile comment will be found on
the last disk(s) of this archive.
unzip: cannot find zipfile directory in one of /Users/het/Projects/flutter/bin/cache/dart-sdk-darwin-arm64.zip or
/Users/het/Projects/flutter/bin/cache/dart-sdk-darwin-arm64.zip.zip, and cannot find /Users/het/Projects/flutter/bin/cache/dart-sdk-darwin-arm64.zip.ZIP, period.
It appears that the downloaded file is corrupt; please try again.
If this problem persists, please report the problem at:
https://github.com/flutter/flutter/issues/new?template=01_activation.yml
```
## Pre-launch Checklist
- [x] I read the [Contributor Guide] and followed the process outlined
there for submitting PRs.
- [x] I read the [Tree Hygiene] wiki page, which explains my
responsibilities.
- [x] I read and followed the [Flutter Style Guide], including [Features
we expect every widget to implement].
- [x] I signed the [CLA].
- [ ] I listed at least one issue that this PR fixes in the description
above.
- [x] I updated/added relevant documentation (doc comments with `///`).
- [x] I added new tests to check the change I am making, or this PR is
[test-exempt].
- [x] I followed the [breaking change policy] and added [Data Driven
Fixes] where supported.
- [x] All existing and new tests are passing.
If you need help, consider asking for advice on the #hackers-new channel
on [Discord].
**Note**: The Flutter team is currently trialing the use of [Gemini Code
Assist for
GitHub](https://developers.google.com/gemini-code-assist/docs/review-github-code).
Comments from the `gemini-code-assist` bot should not be taken as
authoritative feedback from the Flutter team. If you find its comments
useful you can update your code accordingly, but if you are unsure or
disagree with the feedback, please feel free to wait for a Flutter team
member's review for guidance on which automated comments should be
addressed.
<!-- Links -->
[Contributor Guide]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#overview
[Tree Hygiene]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md
[test-exempt]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#tests
[Flutter Style Guide]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md
[Features we expect every widget to implement]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md#features-we-expect-every-widget-to-implement
[CLA]: https://cla.developers.google.com/
[flutter/tests]: https://github.com/flutter/tests
[breaking change policy]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#handling-breaking-changes
[Discord]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Chat.md
[Data Driven Fixes]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Data-driven-Fixes.md
The content hash doesn't exist for local engine changes, except for on
CI. If we detect we're on a branch with committed or uncommitted changes
to engine files; use "master".
towards #171790
re-land attempt for #173114 (try 2)
re-land attempt for #172792 (original)
The first commit in this PR is the previously LGTM'd changes for the
above; with tests.
The second commit is the critical change to make this work in post
submits (fixes#173143). It turns out that while LUCI reports the GitHub
private branches; our recipes directly checkout the git sha. This
matters because the content scripts couldn't determine the branch name
and the rev-parse was just HEAD. This lead the scripts down the
merge-base logic, which returns the previous commit.
A test was added specifically for this.
Alternatively to this change, we could have checked for LUCI_CONTEXT
being present in the environment. This is checked by Flutter tools in
some cases, but not by any other scripts in `bin/internal`. The downside
to checking HEAD: if you have a local branch with engine changes and you
move back a revision - `dart`/`flutter` invocations will generate the
hash for your local changes and fail.
<!-- start_original_pr_link -->
Reverts: flutter/flutter#173114
<!-- end_original_pr_link -->
<!-- start_initiating_author -->
Initiated by: jtmcdole
<!-- end_initiating_author -->
<!-- start_revert_reason -->
Reason for reverting: there is still another problem with the merge
queue causing the content hash to be different:
git revision: e13dd5384cc0db576291c2925852879f1111f284
actual hash: 4b6f7b0f9849efaa59f515c8e95f3f27a6eb2ffb
hash in the queue? 9e5b2eef4ba79b15b4f80dbba812d199d262366f
<!-- end_revert_reason -->
<!-- start_original_pr_author -->
Original PR Author: jtmcdole
<!-- end_original_pr_author -->
<!-- start_reviewers -->
Reviewed By: {matanlurey, chingjun}
<!-- end_reviewers -->
<!-- start_revert_body -->
This change reverts the following previous change:
The content hash doesn't exist for local engine changes, except for on
CI. If we detect we're on a branch with committed or uncommitted changes
to engine files; use "master".
towards #171790
re-land attempt for #172792 with the following changes:
1. content_aware_hash.(ps1|sh) now consider multiple branches to choose
between HEAD and merge-base.
2. content_aware_hash_test.dart updated for these new requirements
3. content_aware_hash_test.dart allows for forcing powershell on mac for
testing
4. updated docs/tool/Engine-artifacts.md documentation.
## Pre-launch Checklist
- [x] I read the [Contributor Guide] and followed the process outlined
there for submitting PRs.
- [x] I read the [Tree Hygiene] wiki page, which explains my
responsibilities.
- [x] I read and followed the [Flutter Style Guide], including [Features
we expect every widget to implement].
- [x] I signed the [CLA].
- [x] I listed at least one issue that this PR fixes in the description
above.
- [x] I updated/added relevant documentation (doc comments with `///`).
- [x] I added new tests to check the change I am making, or this PR is
[test-exempt].
- [ ] I followed the [breaking change policy] and added [Data Driven
Fixes] where supported.
- [x] All existing and new tests are passing.
<!-- end_revert_body -->
Co-authored-by: auto-submit[bot] <flutter-engprod-team@google.com>
The content hash doesn't exist for local engine changes, except for on
CI. If we detect we're on a branch with committed or uncommitted changes
to engine files; use "master".
towards #171790
re-land attempt for #172792 with the following changes:
1. content_aware_hash.(ps1|sh) now consider multiple branches to choose
between HEAD and merge-base.
2. content_aware_hash_test.dart updated for these new requirements
3. content_aware_hash_test.dart allows for forcing powershell on mac for
testing
4. updated docs/tool/Engine-artifacts.md documentation.
## Pre-launch Checklist
- [x] I read the [Contributor Guide] and followed the process outlined
there for submitting PRs.
- [x] I read the [Tree Hygiene] wiki page, which explains my
responsibilities.
- [x] I read and followed the [Flutter Style Guide], including [Features
we expect every widget to implement].
- [x] I signed the [CLA].
- [x] I listed at least one issue that this PR fixes in the description
above.
- [x] I updated/added relevant documentation (doc comments with `///`).
- [x] I added new tests to check the change I am making, or this PR is
[test-exempt].
- [ ] I followed the [breaking change policy] and added [Data Driven
Fixes] where supported.
- [x] All existing and new tests are passing.
<!-- start_original_pr_link -->
Reverts: flutter/flutter#172792
<!-- end_original_pr_link -->
<!-- start_initiating_author -->
Initiated by: jtmcdole
<!-- end_initiating_author -->
<!-- start_revert_reason -->
Reason for reverting: almost, but not quite right. android builders
generate the wrong hash because they are on custom branches.
<!-- end_revert_reason -->
<!-- start_original_pr_author -->
Original PR Author: jtmcdole
<!-- end_original_pr_author -->
<!-- start_reviewers -->
Reviewed By: {bdero, matanlurey}
<!-- end_reviewers -->
<!-- start_revert_body -->
This change reverts the following previous change:
The content hash doesn't exist for local engine changes, except for on
CI. If we detect we're on a branch with committed or uncommitted changes
to engine files; use "master".
towards #171790
<!-- end_revert_body -->
Co-authored-by: auto-submit[bot] <flutter-engprod-team@google.com>
The content hash doesn't exist for local engine changes, except for on
CI. If we detect we're on a branch with committed or uncommitted changes
to engine files; use "master".
towards #171790
Rolls forward https://github.com/flutter/flutter/pull/166717.
Does not copy the `README` or `DEPS` files, and instead uses synthetic
scratch files.
These files can change, so we can't possibly know how to hash them
consistently.
<!-- start_original_pr_link -->
Reverts: flutter/flutter#166717
<!-- end_original_pr_link -->
<!-- start_initiating_author -->
Initiated by: matanlurey
<!-- end_initiating_author -->
<!-- start_revert_reason -->
Reason for reverting: Broke tree on
[postsubmit](https://logs.chromium.org/logs/flutter/buildbucket/cr-buildbucket/8718108088961866849/+/u/test:_run_suite_safari-dart2js-canvaskit-engine/stdout)
<!-- end_revert_reason -->
<!-- start_original_pr_author -->
Original PR Author: jtmcdole
<!-- end_original_pr_author -->
<!-- start_reviewers -->
Reviewed By: {matanlurey}
<!-- end_reviewers -->
<!-- start_revert_body -->
This change reverts the following previous change:
1. Calculate the hash in only two places: `content_aware_hash.{ps1|sh}`
2. Call this from the workflow
3. Eventually call this from `update_engine_version.{ps1|sh}`
The files of import:
* `DEPS`: tracks third party dependencies related to building the engine
* `engine`: all the code in the engine folder
* `bin/internal/content_aware_hash.ps1`: script for calculating the hash
on windows
* `bin/internal/content_aware_hash.sh`: script for calculating the hash
on mac/linux
* `.github/workflows/content-aware-hash.yml`: github action for CI/CD
hashing
Tested on windows and mac:
```shell
PS C:\src\flutter> C:\src\flutter\bin\internal\content_aware_hash.ps1
c24231e276e0719738e175e0622e040ad21a7012
```
```shell
❯ ~/src/flutter/bin/internal/content_aware_hash.sh
c24231e276e0719738e175e0622e040ad21a7012
```
<!-- end_revert_body -->
Co-authored-by: auto-submit[bot] <flutter-engprod-team@google.com>
1. Calculate the hash in only two places: `content_aware_hash.{ps1|sh}`
2. Call this from the workflow
3. Eventually call this from `update_engine_version.{ps1|sh}`
The files of import:
* `DEPS`: tracks third party dependencies related to building the engine
* `engine`: all the code in the engine folder
* `bin/internal/content_aware_hash.ps1`: script for calculating the hash
on windows
* `bin/internal/content_aware_hash.sh`: script for calculating the hash
on mac/linux
* `.github/workflows/content-aware-hash.yml`: github action for CI/CD
hashing
Tested on windows and mac:
```shell
PS C:\src\flutter> C:\src\flutter\bin\internal\content_aware_hash.ps1
c24231e276e0719738e175e0622e040ad21a7012
```
```shell
❯ ~/src/flutter/bin/internal/content_aware_hash.sh
c24231e276e0719738e175e0622e040ad21a7012
```
