This extracts a Dart test utilities library, containing
`expectAssertion` and `expectArgumentError` functions that simplify
running tests that test assertions across debug, profile, and release
configurations.
This change also restricts Dart unit tests to testing files whose
filename matches `*_test.dart` under `flutter/testing/dart`; previously
any file in that directory was run, but all files matched the above
pattern.
The JSON codec is awkward to use in the wrapper (since the client has to build and link one of the JSON libraries to do so). Since it would be very cumbersome to wrap in a C API, and there's essentially no reason to use it instead of the standard codec, this removes it from the wrapper entirely.
Since some system channels (internal to the engine) still use it, it's moved into common/cpp instead of being eliminated entirely. Internally we always use RapidJSON though, so the jsoncpp implementation is removed. Also adds some unit test coverage, since there wasn't any.
Fixes#30669
Currently every Linux runner has this code to allow relative resource paths; this moves it into the framework so that any embedder can get this behavior without that code needing to be in the template.
Rolls buildroot to pick up std::filesystem support in our libc++
* The outputs of all commands are not printed and not just commands that fail.
* The stdout and stderr are now printed in order.
* Clear dividers mark logs from specific subprocesses or errors.
* The test whose run failed should now be exactly on top of the error message and code.
`//flutter/testing` now contains a lot of utilities used by other test targets.
This includes stuff like working with render targets that use either OpenGL or
Metal, fixtures for interacting with the Dart VM, test assertion predicates,
etc.. However, these utilities themselves are not tested as part of a standalone
test suite. Instead, only the test targets that include it exercise these
utilities. Since these are no longer trivial, a new test target has been added
that tests the testing utilities directly.
Make flutter_tester support multithreaded testing, and run all Dart tests in both single and multithreaded configurations
This also modifies Shell::GetUIIsolateLastError() and Shell::EngineHasLivePorts() so that they must be called from the UI task runner.
The tests we write must be resilient to the order in which they are run in the
harness. That is, they must not rely on global state set by other tests that
have already run in the process. Also, these tests must themselves be
repeatable. That is, they must correctly clean up after themselves and be able
to run successfully again in the same process.
This patch adds some safeguards against (but does NOT guarantee) the addition of
tests that violate the dictum.
Additionally, test failures must be easily reproducible for folks investigating
the test failure. Also, tests that assert correctness of unrelated code must not
stop progress on the authors patch.
This changes does not hinder reproducibility of test failures because the random
seed is printed in the logs before running each test. Developers attempting to
reproduce the failure locally can do the same via the following invocation
`--gtest_shuffle --gtest_repeat=<the count> --gtest_random_seed=<seed from failing run>`.
This change does introduce potential burden on patch authors that may see
failures in unrelated code as a newly failing shuffle seed is used on their
runs. To ameliorate this, we will formulate guidance for them to aggressively
mark such tests as disabled and file bugs to enable the same.
The test seed is intentionally kept low because it’s purpose is to test that
individual tests are repeatable. It must not be used as a replacement for
fuzzing.
Without this change, a single dart test will print out 306 lines.
With this change, a single dart test will only print out 33 lines.
This helps a lot in local tests.
In order to return to the old logging, set `--verbose-dart-snapshot` flag.
Previously this script was not reporting any JUNit failures and somehow
ended up included a non-existent jar for Java tests to test against.
It looks like one of the JUnit tests is now failing. Disabling it for
now to turn on CI again as soon as possible, will fix and enable it in a
follow up.
