e65d28319a
📜 [Doc on the Google testing Queue](https://docs.google.com/document/d/15gMCfOIUWlsRuLRMhMge-7odVHTzU8zibiVFDLdn3Pk/edit?usp=sharing) Project for tracking: https://github.com/orgs/flutter/projects/200 This updates the docs on Google testing and incorporates the proposed queue for managing blocked PRs better. Fixes https://github.com/flutter/flutter/issues/162832 ## Pre-launch Checklist - [x] I read the [Contributor Guide] and followed the process outlined there for submitting PRs. - [x] I read the [Tree Hygiene] wiki page, which explains my responsibilities. - [x] I read and followed the [Flutter Style Guide], including [Features we expect every widget to implement]. - [x] I signed the [CLA]. - [x] I listed at least one issue that this PR fixes in the description above. - [x] I updated/added relevant documentation (doc comments with `///`). - [ ] I added new tests to check the change I am making, or this PR is [test-exempt]. - [x] I followed the [breaking change policy] and added [Data Driven Fixes] where supported. - [x] All existing and new tests are passing. If you need help, consider asking for advice on the #hackers-new channel on [Discord]. <!-- Links --> [Contributor Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#overview [Tree Hygiene]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md [test-exempt]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#tests [Flutter Style Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md [Features we expect every widget to implement]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md#features-we-expect-every-widget-to-implement [CLA]: https://cla.developers.google.com/ [flutter/tests]: https://github.com/flutter/tests [breaking change policy]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#handling-breaking-changes [Discord]: https://github.com/flutter/flutter/blob/main/docs/contributing/Chat.md [Data Driven Fixes]: https://github.com/flutter/flutter/blob/main/docs/contributing/Data-driven-Fixes.md
3.7 KiB
"Google testing" is the test suite that Google runs to test a Flutter pull request against their internal code base. This check shows up as one of the many check runs at the bottom of an open pull request.
Validation Pipeline (60 minutes total)
-
Triggering google testing (<1 minute)
Google testing starts once an approval from a member of flutter-hackers is given. For Googlers, the check is run immediately. Google testing is triggered on GitHub webhooks, and uses a 30-minute cron job to backfill when webhooks are dropped.
-
Running google testing (30 minutes)
A subset of tests have been selected to run on presubmit as our smoke test suite. This gives quick, high coverage for PRs without running everything.
-
Propagating results back to GitHub (30 minutes)
Once Google Testing finishes, it takes up to 1 hour for the results to be propagated back to Github. Once the result is available on Github, it will show "Google Testing" as either "success" or "failure".
Common issues
My PR is blocked on Google testing
If your reviewer is a Googler, ping them on the PR to let them know the change is blocked. Reviewers will typically be notified already if, for example, the autosubmit label was removed by the bot.
If your reviewer is not a Googler, reach out in the #hackers channel on Discord for support.
Google employees can view the test output and provide feedback for next steps.
In order to track pull requests blocked on Google testing for resolution, have your reviewer add yours to the Github testing queue project.
Pull requests in the queue are addressed in a FIFO fashion. The project will display where your change is in the queue.
For full guidance on presubmit failures, see the Fix failing checks doc.
There's goldens failures on my PR, but those are expected
If a Googler has verified the goldens are expected, the Googler can internally update the check to passing. This will indicate that we should accept the scubas. However, once merged, this will go through a second round of review as our smoke test suite only runs a subset of the codebase.
My PR got reverted due to it breaking Google, but the check never ran on presubmit!
As of January 2024, Google testing is using webhooks to power most of the validation. In the rare case this happens, please file a bug and add "team-infra" as the label to it so we can see what went wrong.
My PR has an infra error
Use the GitHub check run UI to rerun. If it goes from failing -> passing, it will be marked as a flake. The team that owns Google testing monitors the flake rate, and is working to get it under <1%.
If the issue is not a flake, a Googler is necessary to investigate the infra error. As of Feb 2024, the most common infra error is related to race conditions when creating the internal CL for testing.
What if the failure is expected or unrelated to my change?
Googlers can go to http://frob and override the results.
Where can I get help?
Work with your reviewer to resolve blocked changes. If there is an infrastructure issue related to Google testing, an issue should be filed with the label "team-infra". Confirm with your reviewer before filing one.
GitHub issues with the label "team-infra" are triaged weekly.
For Googlers, you can use go/file-frob-bug for issues where confidential information is needed for debugging.
For live questions, you can ask in #hackers-infra.