mirror of
https://github.com/flutter/flutter.git
synced 2026-02-20 02:29:02 +08:00
Refactor, update, and move around testing/scenario_app/README.md (flutter/engine#50659)
I'm not sure about long-term direction re: separating the Android and iOS code, but other than `lib/*.dart`, all of the code in `testing/scenario_app/**` is platform-specific. Therefore, I've moved the documentation around, and added to it where it made sense, in order to make it easier for someone else to grok. Index: - [`testing/scenario_app/README.md`](5012e1fd5d/testing/scenario_app/README.md) - [`testing/scenario_app/bin/README.md`](5012e1fd5d/testing/scenario_app/bin/README.md) - [`testing/scenario_app/android/README.md`](5012e1fd5d/testing/scenario_app/android/README.md) - [`testing/scenario_app/ios/README.md`](5012e1fd5d/testing/scenario_app/ios/README.md) This also gives me an obvious place to further document how the Android tests work: <a href="5012e1fd5d/testing/scenario_app/android/README.md"><img width="880" alt="Screenshot 2024-02-14 at 12 35 39 PM" src="https://github.com/flutter/engine/assets/168174/d4663005-d770-4003-a3fa-74ae8bf635c4"></a> Closes https://github.com/flutter/flutter/issues/143350. --------- Co-authored-by: Jenn Magder <magder@google.com> Co-authored-by: Gray Mackall <34871572+gmackall@users.noreply.github.com>
This commit is contained in:
Matan Lurey
GitHub
parent
82d5e3784f
commit
f3342a014c
@ -1,85 +1,43 @@
|
||||
# Scenario App
|
||||
|
||||
This folder contains e2e integration tests for the engine in conjunction with a
|
||||
fake dart:ui framework running in JIT or AOT.
|
||||
[](https://github.com/flutter/flutter/issues?q=is%3Aopen+is%3Aissue+label%3A%22e%3A+scenario-app%22)
|
||||
|
||||
It intentionally has no dependencies on the Flutter framework or tooling, such
|
||||
that it should be buildable as a presubmit or postsubmit to the engine even in
|
||||
the face of changes to Dart or dart:ui that require upstream changes in the
|
||||
Flutter tooling.
|
||||
This package simulates a Flutter app that uses the engine (`dart:ui`) only,
|
||||
in conjunction with Android and iOS-specific embedding code that simulates the
|
||||
use of the engine in a real app (such as plugins and platform views).
|
||||
|
||||
The [`run_android_tests.sh`](run_android_tests.sh) and
|
||||
[`run_ios_tests.sh`](run_ios_tests.sh) are then used to run the tests on a
|
||||
connected device or emulator.
|
||||
|
||||
See also:
|
||||
|
||||
- [`bin/`](bin/), the entry point for running Android integration tests.
|
||||
- [`lib/`](lib/), the Dart code and instrumentation for the scenario app.
|
||||
- [`ios/`](ios/), the iOS-side native code and tests.
|
||||
- [`android/`](android/), the Android-side native code and tests.
|
||||
|
||||
## Running a smoke test on Firebase TestLab
|
||||
|
||||
To run the smoke test on Firebase TestLab test, build `android_profile_arm64`,
|
||||
and run [`./ci/firebase_testlab.py`](../../ci/firebase_testlab.py), or pass
|
||||
`--variant` to run a different configuration.
|
||||
|
||||
```sh
|
||||
# From the root of the engine repository
|
||||
$ ./ci/firebase_testlab.py --variant android_debug_arm64
|
||||
```
|
||||
|
||||
> [!NOTE]
|
||||
> These instructions were not verified at the time of writing/refactoring.
|
||||
|
||||
## Adding a New Scenario
|
||||
|
||||
Create a new subclass of [Scenario](https://github.com/flutter/engine/blob/5d9509ae056b04c30295df27f201f31af9777842/testing/scenario_app/lib/src/scenario.dart#L9)
|
||||
and add it to the map in [scenarios.dart](https://github.com/flutter/engine/blob/db4d423ad9c6dad373618712690acd06b0a385fd/testing/scenario_app/lib/src/scenarios.dart#L22).
|
||||
For an example, see [animated_color_square.dart](https://github.com/flutter/engine/blob/5d9509ae056b04c30295df27f201f31af9777842/testing/scenario_app/lib/src/animated_color_square.dart#L15),
|
||||
which draws a continuously animating colored square that bounces off the sides
|
||||
of the viewport.
|
||||
Create a new subclass of [Scenario](lib/src/scenario.dart) and add it to the map
|
||||
in [scenarios.dart](lib/src/scenarios.dart). For an example, see
|
||||
[animated_color_square.dart](lib/src/animated_color_square.dart), which draws a
|
||||
continuously animating colored square that bounces off the sides of the
|
||||
viewport.
|
||||
|
||||
Then set the scenario from the Android or iOS app by calling "set_scenario" on
|
||||
platform channel.
|
||||
|
||||
## Running for iOS
|
||||
|
||||
Build the `ios_debug_sim_unopt` engine variant, and run
|
||||
|
||||
```sh
|
||||
./run_ios_tests.sh
|
||||
```
|
||||
|
||||
in your shell.
|
||||
|
||||
To run or debug in Xcode, open the xcodeproj file located in
|
||||
`<engine_out_dir>/ios_debug_sim_unopt/scenario_app/Scenarios/Scenarios.xcodeproj`.
|
||||
|
||||
### iOS Platform View Tests
|
||||
|
||||
For PlatformView tests on iOS, you'll also have to edit the dictionaries in
|
||||
[AppDelegate.m](https://github.com/flutter/engine/blob/5d9509ae056b04c30295df27f201f31af9777842/testing/scenario_app/ios/Scenarios/Scenarios/AppDelegate.m#L29) and [GoldenTestManager.m](https://github.com/flutter/engine/blob/db4d423ad9c6dad373618712690acd06b0a385fd/testing/scenario_app/ios/Scenarios/ScenariosUITests/GoldenTestManager.m#L25) so that the correct golden image can be found. Also, you'll have to add a [GoldenPlatformViewTests](https://github.com/flutter/engine/blob/5d9509ae056b04c30295df27f201f31af9777842/testing/scenario_app/ios/Scenarios/ScenariosUITests/GoldenPlatformViewTests.h#L18) in [PlatformViewUITests.m](https://github.com/flutter/engine/blob/af2ffc02b72af2a89242ca3c89e18269b1584ce5/testing/scenario_app/ios/Scenarios/ScenariosUITests/PlatformViewUITests.m).
|
||||
|
||||
If `PlatformViewRotation` is failing, make sure Simulator app Device > Rotate Device Automatically
|
||||
is selected, or run:
|
||||
|
||||
```bash
|
||||
defaults write com.apple.iphonesimulator RotateWindowWhenSignaledByGuest -int 1
|
||||
```
|
||||
|
||||
### Generating Golden Images on iOS
|
||||
|
||||
Screenshots are saved as
|
||||
[XCTAttachment](https://developer.apple.com/documentation/xctest/activities_and_attachments/adding_attachments_to_tests_and_activities?language=objc)'s.
|
||||
If you look at the output from running the tests you'll find a path in the form:
|
||||
`/Users/$USER/Library/Developer/Xcode/DerivedData/Scenarios-$HASH`.
|
||||
Inside that directory you'll find
|
||||
`./Build/Products/Debug-iphonesimulator/ScenariosUITests-Runner.app/PlugIns/ScenariosUITests.xctest/` which is where all the images that were
|
||||
compared against golden reside.
|
||||
|
||||
## Running for Android
|
||||
|
||||
### Integration tests
|
||||
|
||||
For emulators running on a x64 host, build `android_debug_unopt_x64` using
|
||||
`./tools/gn --android --unoptimized --goma --android-cpu=x64`.
|
||||
|
||||
Then, launch the emulator, and run `./testing/scenario_app/run_android_tests.sh android_debug_unopt_x64`.
|
||||
|
||||
If you wish to build a different engine variant, make sure to pass that variant to the script `run_android_tests.sh`.
|
||||
|
||||
If you make a change to the source code, you would need to rebuild the same engine variant.
|
||||
|
||||
### Smoke test on FTL
|
||||
|
||||
To run the smoke test on Firebase TestLab test, build `android_profile_arm64`, and run
|
||||
`./flutter/ci/firebase_testlab.py`. If you wish to test a different variant, e.g.
|
||||
debug arm64, pass `--variant android_debug_arm64`.
|
||||
|
||||
### Updating Gradle dependencies
|
||||
|
||||
If a Gradle dependency is updated, lockfiles must be regenerated.
|
||||
|
||||
To generate new lockfiles, run:
|
||||
|
||||
```bash
|
||||
cd android/app
|
||||
../../../../../third_party/gradle/bin/gradle generateLockfiles
|
||||
```
|
||||
Then set the scenario from the Android or iOS app by calling `set_scenario` on
|
||||
platform channel `driver`.
|
||||
|
||||
42
engine/src/flutter/testing/scenario_app/android/README.md
Normal file
42
engine/src/flutter/testing/scenario_app/android/README.md
Normal file
@ -0,0 +1,42 @@
|
||||
# Scenario App: Android Tests
|
||||
|
||||
As mentioned in the [top-level README](../README.md), this directory contains
|
||||
the Android-specific native code and tests for the [scenario app](../lib). To
|
||||
run the tests, you will need to build the engine with the appropriate
|
||||
configuration.
|
||||
|
||||
For example, `android_debug_unopt` or `android_debug_unopt_arm64` was built,
|
||||
run:
|
||||
|
||||
```sh
|
||||
# From the root of the engine repository
|
||||
$ ./testing/run_android_tests.sh android_debug_unopt
|
||||
|
||||
# Or, for arm64
|
||||
$ ./testing/run_android_tests.sh android_debug_unopt_arm64
|
||||
```
|
||||
|
||||
## CI Configuration
|
||||
|
||||
See [`ci/builders/linux_android_emulator.json`](../../../ci/builders/linux_android_emulator.json)
|
||||
, and grep for `run_android_tests.sh`.
|
||||
|
||||
The following matrix of configurations is tested on the CI:
|
||||
|
||||
| API Version | Graphics Backend | Skia Gold | Rationale |
|
||||
| ----------- | ------------------- | ---------------------------------------------------------------- | ---------------------------------------------------------- |
|
||||
| 28 | Skia | [Android 28 + Skia][skia-gold-skia-28] | Older Android devices (without `ImageReader`) on Skia. |
|
||||
| 28 | Impeller (OpenGLES) | [Android 28 + Impeller OpenGLES][skia-gold-impeller-opengles-28] | Older Android devices (without `ImageReader`) on Impeller. |
|
||||
| 34 | Skia | [Android 34 + Skia][skia-gold-skia-34] | Newer Android devices on Skia. |
|
||||
| 34 | Impeller (OpenGLES) | [Android 34 + Impeller OpenGLES][skia-gold-impeller-opengles-34] | Newer Android devices on Impeller with OpenGLES. |
|
||||
| 34 | Impeller (Vulkan) | [Android 34 + Impeller Vulkan][skia-gold-impeller-vulkan-34] | Newer Android devices on Impeller. |
|
||||
|
||||
[skia-gold-skia-28]: https://flutter-engine-gold.skia.org/search?left_filter=AndroidAPILevel%3D28%26GraphicsBackend%3Dskia&negative=true&positive=true
|
||||
[skia-gold-impeller-opengles-28]: https://flutter-engine-gold.skia.org/search?left_filter=AndroidAPILevel%3D28%26GraphicsBackend%3Dimpeller-opengles&negative=true&positive=true
|
||||
[skia-gold-skia-34]: https://flutter-engine-gold.skia.org/search?left_filter=AndroidAPILevel%3D34%26GraphicsBackend%3Dskia&negative=true&positive=true
|
||||
[skia-gold-impeller-opengles-34]: https://flutter-engine-gold.skia.org/search?left_filter=AndroidAPILevel%3D34%26GraphicsBackend%3Dimpeller-opengles&negative=true&positive=true
|
||||
[skia-gold-impeller-vulkan-34]: https://flutter-engine-gold.skia.org/search?left_filter=AndroidAPILevel%3D34%26GraphicsBackend%3Dimpeller-vulkan&negative=true&positive=true
|
||||
|
||||
## Updating Gradle dependencies
|
||||
|
||||
See [Updating the Embedding Dependencies](../../../tools/cipd/android_embedding_bundle/README.md).
|
||||
40
engine/src/flutter/testing/scenario_app/bin/README.md
Normal file
40
engine/src/flutter/testing/scenario_app/bin/README.md
Normal file
@ -0,0 +1,40 @@
|
||||
# `android_integration_tests` runner
|
||||
|
||||
This directory contains code specific to running Android integration tests.
|
||||
|
||||
The tests are uploaded and run on the device using `adb`, and screenshots are
|
||||
captured and compared using Skia Gold (if available, for example on CI).
|
||||
|
||||
## Usage
|
||||
|
||||
```sh
|
||||
dart bin/android_integration_tests.dart \
|
||||
--adb ../third_party/android_tools/sdk/platform-tools/adb \
|
||||
--out-dir ../out/android_debug_unopt_arm64
|
||||
```
|
||||
|
||||
## Debugging
|
||||
|
||||
When debugging, you can use the `--smoke-test` argument to run a single test
|
||||
by class name, which can be useful to verify the setup.
|
||||
|
||||
For example, to run the `EngineLaunchE2ETest` test:
|
||||
|
||||
```sh
|
||||
dart bin/android_integration_tests.dart \
|
||||
--adb ../third_party/android_tools/sdk/platform-tools/adb \
|
||||
--out-dir ../out/android_debug_unopt_arm64 \
|
||||
--smoke-test dev.flutter.scenarios.EngineLaunchE2ETest
|
||||
```
|
||||
|
||||
## Additional arguments
|
||||
|
||||
- `--use-skia-gold`: Use Skia Gold to compare screenshots. Defaults to true
|
||||
when running on CI, and false otherwise (i.e. when running locally). If
|
||||
set to true, [isSkiaGoldClientAvailable] must be true.
|
||||
|
||||
- `--enable-impeller`: Enable Impeller for the Android app. Defaults to
|
||||
false, which means that the app will use Skia as the graphics backend.
|
||||
|
||||
- `--impeller-backend`: The Impeller backend to use for the Android app.
|
||||
Defaults to 'vulkan'. Only used when `--enable-impeller` is set to true.
|
||||
@ -16,8 +16,7 @@ import 'utils/logs.dart';
|
||||
import 'utils/process_manager_extension.dart';
|
||||
import 'utils/screenshot_transformer.dart';
|
||||
|
||||
const int tcpPort = 3001;
|
||||
|
||||
// If you update the arguments, update the documentation in the README.md file.
|
||||
void main(List<String> args) async {
|
||||
final ArgParser parser = ArgParser()
|
||||
..addOption(
|
||||
@ -82,6 +81,8 @@ void main(List<String> args) async {
|
||||
);
|
||||
}
|
||||
|
||||
const int _tcpPort = 3001;
|
||||
|
||||
enum _ImpellerBackend {
|
||||
vulkan,
|
||||
opengles;
|
||||
@ -137,7 +138,7 @@ Future<void> _run({
|
||||
late ServerSocket server;
|
||||
final List<Future<void>> pendingComparisons = <Future<void>>[];
|
||||
await step('Starting server...', () async {
|
||||
server = await ServerSocket.bind(InternetAddress.anyIPv4, tcpPort);
|
||||
server = await ServerSocket.bind(InternetAddress.anyIPv4, _tcpPort);
|
||||
stdout.writeln('listening on host ${server.address.address}:${server.port}');
|
||||
server.listen((Socket client) {
|
||||
stdout.writeln('client connected ${client.remoteAddress.address}:${client.remotePort}');
|
||||
@ -237,7 +238,7 @@ Future<void> _run({
|
||||
});
|
||||
|
||||
await step('Reverse port...', () async {
|
||||
final int exitCode = await pm.runAndForward(<String>[adb.path, 'reverse', 'tcp:3000', 'tcp:$tcpPort']);
|
||||
final int exitCode = await pm.runAndForward(<String>[adb.path, 'reverse', 'tcp:3000', 'tcp:$_tcpPort']);
|
||||
if (exitCode != 0) {
|
||||
panic(<String>['could not forward port']);
|
||||
}
|
||||
|
||||
55
engine/src/flutter/testing/scenario_app/ios/README.md
Normal file
55
engine/src/flutter/testing/scenario_app/ios/README.md
Normal file
@ -0,0 +1,55 @@
|
||||
# Scenario App: iOS Tests
|
||||
|
||||
As mentioned in the [top-level README](../README.md), this directory contains
|
||||
the iOS-specific native code and tests for the [scenario app](../lib). To run
|
||||
the tests, you will need to build the engine with the appropriate configuration.
|
||||
|
||||
For example, after building `ios_debug_sim_unopt` (to run on Intel Macs) or `ios_debug_sim_unopt_arm64` (to run on ARM Macs),
|
||||
run:
|
||||
|
||||
```sh
|
||||
# From the root of the engine repository
|
||||
$ ./testing/run_ios_tests.sh ios_debug_sim_unopt
|
||||
```
|
||||
or
|
||||
```sh
|
||||
# From the root of the engine repository
|
||||
$ ./testing/run_ios_tests.sh ios_debug_sim_unopt_arm64
|
||||
```
|
||||
|
||||
To run or debug in Xcode, open the xcodeproj file located in
|
||||
`<engine_out_dir>/ios_debug_sim_unopt/scenario_app/Scenarios/Scenarios.xcodeproj`.
|
||||
|
||||
## CI Configuration
|
||||
|
||||
See [`ci/builders/mac_unopt.json`](../../../../ci/builders/mac_unopt.json), and
|
||||
grep for `run_ios_tests.sh`.
|
||||
|
||||
## iOS Platform View Tests
|
||||
|
||||
For PlatformView tests on iOS, edit the dictionaries in
|
||||
[AppDelegate.m](Scenarios/Scenarios/AppDelegate.m) and
|
||||
[GoldenTestManager.m](Scenarios/ScenariosUITests/GoldenTestManager.m) so that
|
||||
the correct golden image can be found. Also, add a
|
||||
[GoldenPlatformViewTests](Scenarios/ScenariosUITests/GoldenPlatformViewTests.h)
|
||||
in [PlatformViewUITests.m](Scenarios/ScenariosUITests/PlatformViewUITests.m).
|
||||
|
||||
If `PlatformViewRotation` is failing, make sure
|
||||
`Simulator app Device > Rotate Device Automatically` is selected, or run:
|
||||
|
||||
```bash
|
||||
defaults write com.apple.iphonesimulator RotateWindowWhenSignaledByGuest -int 1
|
||||
```
|
||||
|
||||
## Generating Golden Images on iOS
|
||||
|
||||
Screenshots are saved as
|
||||
[XCTAttachment](https://developer.apple.com/documentation/xctest/activities_and_attachments/adding_attachments_to_tests_and_activities?language=objc)'s.
|
||||
|
||||
A path in the form of
|
||||
`/Users/$USER/Library/Developer/Xcode/DerivedData/Scenarios-$HASH` will be
|
||||
printed to the console.
|
||||
|
||||
Inside that directory there is a directory
|
||||
`./Build/Products/Debug-iphonesimulator/ScenariosUITests-Runner.app/PlugIns/ScenariosUITests.xctest/`
|
||||
which is where all the images that were compared against golden reside.
|
||||
Loading…
x
Reference in New Issue
Block a user