flutter_flutter/docs/engine/testing/Testing-the-engine.md

# Testing the engine

This guide describes how to write and run various types of tests in the engine.

## C++ - core engine

If you edit `.cc` files in [`engine`](../../),
you're working on the core, portable Flutter engine.

### Unit tests (C++)

C++ unit tests are co-located with their header and source files. For instance,
`fml/file.h` and `fml/file.cc` have a `fml/file_unittest.cc` in the same
directory. When editing C++ files, look for its `_unittest.cc` sibling or create
one if there isn't one present.

The engine repo has a unified build system to build C, C++, Objective-C,
Objective-C++, and Java files using [GN](https://gn.googlesource.com/gn/) and
[Ninja](https://ninja-build.org/). Individual `_unittest.cc` files are
referenced by the `BUILD.gn` build rule located in the folder or in an ancestor
folder.

You can run the C++ unit tests with:

```sh
testing/run_tests.py --type=engine
```

from the `flutter` directory, after building the engine variant to test
(by default `host_debug_unopt`). To use a different variant (e.g. if you use
an Apple Silicon Mac), run:

```sh
testing/run_tests.py --type=engine --variant=host_debug_unopt_arm64
```

Behind the scenes, those tests in the same directory are built together as a
testonly executable when you build the engine variant. The `run_tests.py` script
executes them one by one.

C++ unit tests are executed during pre-submit on our CI system when submitting
PRs to the `flutter/engine` repository.

#### Google Tests

C++ unit tests in the core engine uses the [Google Test](https://google.github.io/googletest/primer.html)
C++ testing framework to facilitate C++ test discovery, assertions, etc.

Since the engine is portable, these unit tests are compiled and run directly
on and for your host machine architecture.

It's best practice to test only one real production class per test and create
mocks for all other dependencies.

Each Google Test target produces an executable in the `out` directory with the
`_unittests` suffix. You use these executables to run Google Tests.

To run the shell's C++ unit tests:

```sh
../out/host_debug_unopt/shell_unittests
```

To run a single test, use [Google Test's filters][]:

```sh
../out/host_debug_unopt/shell_unittests --gtest_filter="ShellTest.WaitForFirstFrame"
```

[Google Test's filters]: https://google.github.io/googletest/advanced.html#running-a-subset-of-the-tests

You can use `*` wildcards to run tests whose name matches a pattern:

```sh
../out/host_debug_unopt/shell_unittests --gtest_filter="ShellTest.WaitFor*"
```

> [!TIP]
> Google Test supports other patterns, like `-` for exclusions and `:` for joins.
> Check [Google Test's documentation][] for details.

[Google Test's documentation]: https://google.github.io/googletest/advanced.html#running-a-subset-of-the-tests

To reproduce test flakes, you can run a test multiple times:

```sh
../out/host_debug_unopt/shell_unittests --gtest_filter="ShellTest.WaitForFirstFrame" --gtest_repeat=1000
```

## Java - Android embedding

If you edit `.java` files in the [`shell/platform/android`](../../shell/platform/android/)
directory, you're working on the Android embedding which connects the core C++
engine to the Android SDK APIs and runtime.

### Robolectric JUnit tests

For testing logic within a class at a unit level, create or add to a JUnit test.

Existing Java unit tests are located at [`shell/platform/android/test`](../../shell/platform/android/test)
and follow the Java package directory structure. Files in the [`shell/platform/android/io/flutter/`](../../shell/platform/android/io/flutter/)
package tree can have a parallel file in the [`shell/platform/android/test/io/flutter/`](../../shell/platform/android/test/io/flutter/)
package tree. Files in matching directories are considered [package visible](https://docs.oracle.com/javase/tutorial/java/javaOO/accesscontrol.html)
as is the case in standard Java.

When editing production files in `shell/platform/android/io/flutter/`,
the easiest step to add tests is to look for a matching `...Test.java` file in
`shell/platform/android/test/io/flutter/`.

See the [Java unit test README](../../shell/platform/android/test/README.md)
for details.

The engine repo has a unified build system to build C, C++, Objective-C,
Objective-C++, and Java files using [GN](https://gn.googlesource.com/gn/) and
[Ninja](https://ninja-build.org/). Because it doesn't use the more common
Gradle build system (which can't build C++ for instance), the tests and its
dependencies can't be directly built and run inside Android Studio like a
standard Android project.

Instead, the engine provides the script:

```sh
testing/run_tests.py --type=java
```

to easily build and run the JUnit tests.

This script only has a limited amount of smartness. If you've never built the engine before, it'll build the test and classes under test with a reasonable default configuration. If you've built the engine before, it'll re-build the engine with the same GN flags. You may want to double check your [GN flags](../contributing/Compiling-the-engine.md#compiling-for-android-from-macos-or-linux) if you haven't built the engine for a while.

Behind the scenes, it invokes GN and Ninja to build a single .jar file
containing the test runner and dependencies. Then it uses the system `java`
runtime to execute the .jar. JDK v8 must be set as your `$JAVA_HOME` to run
the Robolectric tests.

See [Setting-up-the-Engine-development-environment#using-vscode-as-an-ide-for-the-android-embedding-java](../dev/Setting-up-the-Engine-development-environment.md)
for tips on setting up Java code completion and syntax highlighting in Visual
Studio when working on the engine and tests.

JUnit tests are executed during pre-submit on our CI system when submitting
PRs to the `flutter/engine` repository.

#### Robolectric

[Robolectric](http://robolectric.org/) is a standard Android testing library to
mock the Android runtime. It allows tests to be executed on a lightweight Java
JVM without booting a heavy Android runtime in an emulator. This allows for
rapid test iterations and allows our tests to run better on CI systems.

All engine JUnit tests are Robolectric tests. This means all `android.*` imports
are mocked by Robolectric. If you need to modify how Android components (such as
an [android.view.View](https://developer.android.com/reference/android/view/View.html)
or an [android.app.Activity](https://developer.android.com/reference/android/app/Activity.html))
behave in the test, see other tests for examples or see docs at
http://robolectric.org/ on how to interact with shadows.

#### Mockito

[Mockito](https://site.mockito.org/) is also a standard Android testing library
used to mock non-Android dependencies needed to construct and test interactions
with your your under-test production class.

It's best practice to test only one real production class per test and
mock all other dependencies with mockito.

The Mockito library is an available test dependency when writing Robolectric
tests.

### Component integration tests

Component tests test the interaction of multiple embedding Java classes together
but they don't test all production classes end-to-end. In the Android embedding
case, we test groups of Java classes by their function in `...ComponentTest.java`
files that are also in the `shell/platform/android/test/io/flutter/`
directory. C++ engine parts via JNI are not tested here.

Component tests are also Robolectric JUnit tests and are invoked together with
unit tests when running:

```
testing/run_tests.py --type=java
```

JUnit component tests are executed during pre-submit on our CI system when
submitting PRs to the `flutter/engine` repository.

### End-to-end tests

End-to-end tests for the Android embedder exist as part of the test suites
in the root of the monorepo. See
[`dev/integration_tests`](../../../../../dev/integration_tests/).

## Objective-C - iOS embedding

If you edit `.h` or `.mm` files in the [`shell/platform/darwin/ios`](../../shell/platform/darwin/ios)
directory, you're working on the iOS embedding which connects the core C++
engine to the iOS SDK APIs and runtime.

### XCTest unit tests

For testing logic within a class in isolation, create or add to a XCTestCase.

The iOS unit testing infrastructure is split in 2 different locations. The
`...Test.mm` files in [`shell/platform/darwin/ios`](../../shell/platform/darwin/ios)
contain the unit tests themselves. The [`testing/ios/IosUnitTests`](../../testing/ios/IosUnitTests/)
directory contains an Xcode container project to execute the test.

See [`testing/ios/IosUnitTests/README.md`](../../testing/ios/IosUnitTests/README.md)
for details on adding new test files.

The engine repo has a unified build system to build C, C++, Objective-C,
Objective-C++, and Java files using [GN](https://gn.googlesource.com/gn/) and
[Ninja](https://ninja-build.org/). Since GN and Ninja has to build the C++
dependencies that the Objective-C classes reference, the tests aren't built by
the Xcode project in [`testing/ios/IosUnitTests`](../../testing/ios/IosUnitTests/).

Instead, the engine provides the script:

```sh
testing/run_tests.py --type=objc
```

to easily build and run the XCTests.

- Add the `--variant host_debug_unopt_arm64` argument when using an arm64 Mac targeting either simulators or physical devices.
- Add the `--ios-variant ios_debug_sim_unopt_arm64` argument when using an arm64 Mac simulator (built with `--simulator-cpu=arm64`).

This script only has a limited amount of smartness. If you've never built the engine before, it'll build the test and classes under test with a reasonable default configuration. If you've built the engine before, it'll re-build the engine with the same GN flags. You may want to double check your GN flags ([See compiling for ios from macos](../contributing/Compiling-the-engine.md#compiling-for-ios-from-macos)) if you haven't built the engine for a while.

Behind the scenes, it invokes GN and Ninja to build the tests and dependencies
into a single `.dylib`. Then it uses Xcode and the Xcode project at
`testing/ios/IosUnitTests` to import and execute the XCTests in the `.dylib`.

If you get an `AssertionError: libios_test_flutter.dylib doesn't exist` error, you may need to manually run the ninja command that is printed to the terminal. e.g. `ninja -C $FLUTTER_ENGINE/out/ios_debug_sim_unopt_arm64 ios_test_flutter`

See [Setting-up-the-Engine-development-environment#editor-autocomplete-support](../dev/Setting-up-the-Engine-development-environment.md)
for tips on setting up C/C++/Objective-C code completion and syntax highlighting
when working on the engine and tests.

To debug the XCTests, you can open the Xcode project at `testing/ios/IosUnitTests/IosUnitTests.xcodeproj`
and run the tests (such as via ⌘U). Note you cannot modify the test source and
build the tests in Xcode for reasons mentioned above. If you modify the test,
you need to run `testing/run_tests.py` again.

XCTests are executed during pre-submit on our CI system when submitting PRs to
the `flutter/engine` repository.

#### XCTest

[XCTest](https://developer.apple.com/documentation/xctest) is the standard way
of creating unit tests in Xcode projects. Since iOS has x86 simulators and
since we can build x86 engines, we can execute the XCTests directly on macOS
in a headless simulator using the real iOS SDK.

#### OCMock

[OCMock](https://ocmock.org) is a standard iOS testing library used to mock
dependencies needed to construct and test interactions with your under-test
production class.

It's best practice to test only one real production class per test and
mock all other dependencies with OCMock.

The OCMock library is available as a test dependency when writing XCTests for
the engine.

### End-to-end tests (iOS Embedder)

End-to-end tests exercise the entire iOS embedding with the C++ engine on
a headless iOS simulator. It's an integration test ensuring that
the engine as a whole on iOS is functioning correctly.

The project containing the iOS end-to-end engine test is at [`testing/ios_scenario_app/ios`](../../testing/ios_scenario_app/ios/).

This test project is build similarly to a normal debug Flutter app. The Dart
code is bundled in JIT mode and is brought into Xcode with a `.framework`
dependency on the prebuilt local engine. It's then installed and executed on a
simulator via Xcode.

Unlike a normal Flutter app, the Flutter framework on the Dart side is a
lightweight fake at [`testing/ios_scenario_app/lib`](../../testing/ios_scenario_app/lib/).
that implements some of the basic functionalities of `dart:ui` Window rather
than using the real Flutter framework at `flutter/flutter`.

The end-to-end test can be executed by running:

```sh
testing/ios_scenario_app/run_ios_tests.sh
```

Additional end-to-end instrumented tests can be added to [`testing/ios_scenario_app/ios/Scenarios/ScenariosTests`](../../testing/ios_scenario_app/ios/Scenarios/ScenariosTests/).

If supporting logic is needed for the test case, it can be added to the
Android app under-test in [`testing/ios_scenario_app/ios/Scenarios/Scenarios`](../../testing/ios_scenario_app/ios/Scenarios/Scenarios/).
or to the fake Flutter framework under-test in [`testing/ios_scenario_app/lib`](../../testing/ios_scenario_app/lib/).

As best practice, favor adding unit tests if possible since end-to-end tests
are, by nature, non-hermetic, slow and flaky.

End-to-end tests on iOS are executed during pre-submit on our CI system when
submitting PRs.

## Dart - dart:ui

If you edit `.dart` files in [`lib/ui`](../../lib/ui/) you're working on the
`dart:ui` package which is the interface between the C++ engine and the Dart
Flutter framework.

### Unit tests (Dart)

Dart classes in [`lib/ui`](../../lib/ui/) have matching unit tests at
[`testing/dart`](../../testing/dart/).

When editing production files in the 'dart:ui' package, add to or create a
test file in `testing/dart`.

To run the Dart unit tests, use the script:

```sh
testing/run_tests.py --type=dart
```

Behind the scenes, it invokes the engine repo's unified [GN](https://gn.googlesource.com/gn/)
and [Ninja](https://ninja-build.org/) build systems to use a version of the Dart
SDK specified in the `DEPS` file to create a `sky_engine` Dart package. Then it
compiles and runs each `_test.dart` file under `testing/dart`.

To debug the test, open `src/out/ios_debug_sim_unopt/ios_scenario_app/Scenarios.xcodeproj` in
Xcode and hit CMD+U.

Dart unit tests are executed during pre-submit on our CI system when submitting
PRs to the `flutter/engine` repository.

_See also: [Flutter Test Fonts](https://github.com/flutter/flutter/blob/main/docs/contributing/testing/Flutter-Test-Fonts.md)_

### Framework tests

Dart tests in the `flutter/flutter` framework repo are also executed on top of
the `dart:ui` package and underlying engine.

These tests are executed during pre-submit on our CI system when
submitting PRs to the `flutter/engine` repository.

Assuming your `flutter` and `engine` working directories are siblings, you can run the framework tests locally using the following command from the root of your `flutter` repository:

```bash
(cd packages/flutter; ../../bin/flutter test --local-engine=host_debug_unopt --local-engine-host=host_debug_unopt)
```

## Web engine

Web tests are run via the `felt` command. More details can be found in [lib/web_ui/README.md](../../lib/web_ui/README.md).