This PR adds a so called fileNameFactory to multiScreenGolden and screenMatchesGolden which allows further customization of the file name which is used to compare.

Use Cases

The newly added function allows one to structure the files differently. For example one can change the naming scheme and put all files to another folder (not to goldens/). In addition to that one can change the multiScreenGolden file names to include the width and height instead of the device name.

Tests

I've added tests for both multiScreenGolden and screenMatchesGolden to ensure the desired effect. The PR is fully backwards compatible so no existing test had to be changed.

Related Issues

Fixes: https://github.com/eBay/flutter_glove_box/issues/45

Instead of calling matchesGoldenFile twice to prime all images, find all Image widgets and precache them so they are available instantly afterwards.

I am not sure if this is enough because their could probably be other assets that are not images which are not loaded this way.

For a currently unknown reason, if a package does not contain any images, then the FontManifest.json is not generated, even if it depends on other packages with fonts/images

We need to investigate if this is a bug in Flutter or working as intended. If working as intended, we may need to come up with an alternative solution for loading fonts.

We have an issue in our app that requires us to add overridable fonts for the fonts to be rendered in golden tests, even though they work fine when running the app. The fonts come from one of our own packages, and the styles which we refer to from our app are also defined within the package.

Our current app setup/structure is as follows: In the app pubspec.yaml we have fonts defined as follows

flutter:
  fonts:
    - family: AvenirNext
      fonts:
        - asset: packages/package_name/fonts/AvenirNext-Regular.ttf
          weight: 400

We use the fonts within our app as follows:

Text(
   'blah blah',
   style: context.textThemeDefinedInPackage.bodyDefault,
),

Our current package setup/structure is as follows: In the package pubspec.yaml we have fonts defined as follows

flutter:
  fonts:
    - family: AvenirNext
      fonts:
        - asset: lib/fonts/AvenirNext-Regular.ttf
          weight: 400

We define styles within our package as follows:

TextStyle(fontFamily: 'AvenirNext')

When we generate golden files (when using the style define in the package as shown above), we just see the squares instead of the fonts being rendered correctly. If we explicitly create a style in our text widgets the font is displayed in the golden test. For example:

Text(
   'this works'
   style: TextStyle(fontFamily: 'AvenirNext', package: 'plentific_ui_core'),
),
Text(
   'this also works'
   style: TextStyle(fontFamily: 'packages/plentific_ui_core/AvenirNext'),
),
Text(
   'this DOES NOT work in golden tests but does in the app'
   style: TextStyle(fontFamily: 'AvenirNext'),
),

The only way I have been able to get the fonts rendering in golden tests (when using the styles defined in the package) is to add 'AvenirNext' to the list of overridableFonts in the golden toolkit package font loader file. Now I have found a more elegant way to enable this, by adding an argument, additionalOverridableFonts to the loadAppFonts() method, however I thought I would run it by you guys first and see if anyone else has experienced similar issues, or if there are any other workarounds I have not found. If not, I can create a non-breaking PR with my fix.

This is a very optimistic approach but maybe there is not much more involved.

I can't get my tests to succeed when using the package as local dependency. I have golden diffs of 0.x% related to fonts. This happens on master as well so it is not related to this change.

• [x] add tests

Problem statement:

Capturing golden files for a screen that has multiple differences across device sizes and multiple complicated ui states.

Current package says to use 'multiScreenGolden' for the same widget rendered in multiple device sizes (i.e Device.phone, Device.tabletLandscape). This however, produces a single png golden file per device. When adding multiple scenarios/states for that screen and needing still to see across multiple devices, this can grow the number of files considerably.

Proposed solution:

DeviceBuilder factory

The DeviceBuilder class allows user to create multiple scenarios across multiple devices in a single png golden file. The example added was around the default flutter counter view and capturing ui based on user taps.

  testGoldens('DeviceBuilder - multiple scenarios - with afterPump',
      (tester) async {
    final builder = DeviceBuilder()
      ..overrideDevicesForAllScenarios(devices: [
        Device.phone,
        Device.iphone11,
        Device.tabletPortrait,
        Device.tabletLandscape,
      ])
      ..addDeviceScenario(
        widget: FlutterDemoPage(),
        name: 'default page',
      )
      ..addDeviceScenario(
        widget: FlutterDemoPage(),
        name: 'tap once',
        afterPump: (scenarioWidgetKey) async {
          final finder = find.descendant(
            of: find.byKey(scenarioWidgetKey),
            matching: find.byIcon(Icons.add),
          );
          expect(finder, findsOneWidget);
          await tester.tap(finder);
        },
      )
      ..addDeviceScenario(
        widget: FlutterDemoPage(),
        name: 'tap five times',
        afterPump: (scenarioWidgetKey) async {
          final finder = find.descendant(
            of: find.byKey(scenarioWidgetKey),
            matching: find.byIcon(Icons.add),
          );
          expect(finder, findsOneWidget);

          await tester.tap(finder);
          await tester.tap(finder);
          await tester.tap(finder);
          await tester.tap(finder);
          await tester.tap(finder);
        },
      );

    await tester.pumpDeviceBuilder(builder);

    await screenMatchesGolden(tester, 'flutter_demo_page_multiple_scenarios');
  });

The DeviceBuilder exposes an 'afterPump' callback to perform any interactions required to render the scenario across all device sizes. For simpler widgets/screens where a ui state can be instantiated statically through construction, then the 'afterPump' functionality wouldn't be needed.

This produces a single golden file:

The simplest use of this doesn't require you to call overrideDevicesForAllScenarios, as by default it will use GoldenToolkit.configuration.defaultDevices. You can also omit scenario name parameter to.

final builder = DeviceBuilder()..addDeviceScenario( widget: FlutterDemoPage());

multiDeviceGolden

This is a helper method much like the existing multiScreenGolden, except that it produces one single golden file instead of multiple per device. This method can be used if you just one a single scenario and don't need to use DeviceBuilder directly.

  testGoldens('multiDeviceGolden', (tester) async {
    await multiDeviceGolden(
      tester,
      'flutter_demo_multi_device_golden',
      widget: FlutterDemoPage(),
      afterPump: (scenarioWidgetKey) async {
        final finder = find.descendant(
          of: find.byKey(scenarioWidgetKey),
          matching: find.byIcon(Icons.add),
        );
        expect(finder, findsOneWidget);
        await tester.tap(finder);
      },
    );
  });

Other fixes

The current tests were not passing on latest flutter stable 1.22.1. Had to regenerate some golden files. Also had issues with analysis options when opening root folder and looking at weather_widgets.dart. So ignored some issues in there to help analysis step pass.

I created a golden test on my local machine. But after I pushed changes to the server test checks failed. Also on some other computers test tests fail.

OS is macos.

local machine logs macos 12.6 (21G115)

✓ Authentication screen golden

ci machine logs macOS 11.7 20G817 (macos-latest in the workflow)

══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞════════════════════════════════════════════════════ The following assertion was thrown while running async test code: Golden "goldens/authentication_screen_golden.png": Pixel test failed, 0.08% diff detected. Failure feedback can be found at /Users/runner/work/metrics/metrics/test/presentation/failures

When the exception was thrown, this was the stack: #0 LocalFileComparator.compare (package:flutter_test/src/_goldens_io.dart:102:7) (elided one frame from package:stack_trace) ════════════════════════════════════════════════════════════════════════════════════════════════════

00:50 +51 -1: /Users/runner/work/metrics/metrics/test/presentation/auth_screen_test.dart: Authentication screen golden [E]
Test failed. See exception logs above. The test description was: Authentication screen golden

00:50 +51 -1: Some tests failed.
Error: Process completed with exit code 1.

flutter doctor

Doctor summary (to see all details, run flutter doctor -v): [✓] Flutter (Channel stable, 3.3.2, on macOS 12.6 21G115 darwin-arm, locale en-GB) [✓] Android toolchain - develop for Android devices (Android SDK version 32.0.0) [✓] Xcode - develop for iOS and macOS (Xcode 14.0) [✓] Chrome - develop for the web [✓] Android Studio (version 2020.3) [✓] IntelliJ IDEA Community Edition (version 2021.3) [✓] VS Code (version 1.71.2) [✓] Connected device (3 available) [✓] HTTP Host Availability

• No issues found!

The Goldens look as follows if I run them with respectively.

..overrideDevicesForAllScenarios(devices: [
        Device.iphone11,
        // Device.phone,
      ])
      
      
..overrideDevicesForAllScenarios(devices: [
        Device.phone,
      ])

Running them with the following setup messes up both tests.

..overrideDevicesForAllScenarios(devices: [
        Device.iphone11,
        Device.phone,
      ])

So the tests seem to influence each other :S

Trying to use golden_toolkit with a null safe library will generate the following warning:

The library 'package:golden_toolkit/golden_toolkit.dart' is legacy, and should not be imported into a null safe library. Try migrating the imported library.

This is because there is no -nullsafety version for golden_toolkit yet.

Quoting from the email that has been sent to applicable package authors:

You can find information on the migration at dart.dev/null-safety, including links to a migration guide and frequently asked questions.

When the main method in flutter_test_config.dart is named as "main", an exception is thrown looking for testExecutable. Renaming my main method to testExecutable runs tests fine.

I tried the golden toolkit and it works great on my machine. I'm working on a MacBook with macOS 10.15.6.

Our CI is running on a unix machine and the golden tests always fail. On the first look the screens look the same. If you look carefully, you can see little differences around texts. Obviously the anti aliasing works diffently.

If my colleague creates the goldens on his unix machine, they match on the CI.

I already tried to turn off the font smoothing in the macOS system settings. Then the rendered text is looking diffently than with smoothing. But also still differently from unix.

Is there anything we can do about? Or can you fix that in the toolkit?

We prefer the golden_toolkit methodology of setting up golden tests above other packages currently available. However, probably the number one issue at this point is consistent UI tests across platforms. Even with the Ahem font, these squares do not render consistently accross platforms, due to issues with font smoothing, text scaling differences, etc. Other packages (such as alchemist) have workarounds for this, like replacing all rendered paragraphs with rendered rectangles instead.

To allow us the best of both worlds, I exposed a new function in the configuration for golden_toolkit which allows providing a method which will run right before the golden tests are generated. This allows developers to (for example) hook-in the rectangle rendering for fonts of the alchemist package (https://github.com/Betterment/alchemist/blob/main/lib/src/blocked_text_image.dart) which will generate consistent golden tests across all platforms, or do more advanced stuff.

When SemanticsDebugger is used with a Scrollable, the semantics boxes are partially displayed :

The golden should be like this :

The fix only add a few pumps on screenMatchesGolden when it detect a SemanticsDebugger (5 pumps max if frame are scheduled).

The autoHeightExtraPumps allows to add other Finder with their number of extra pumps necessary.

Extra pump will be executed if one of the finder is valid and keep the max number of the map.

By default google_font is stored in project root dir google_fonts but the GoldenToolkit is always loading the font from assets/fonts. When using the google_fonts it is not required to enable fonts in pubspec.yaml file , we just have to add the google_fonts folder to assets.

solution : create assets/fonts and add your google_font

Description

Introduce support for overriding TargetPlatform using the debugDefaultTargetPlatformOverride.

Proposal

Introduce a non-required platform(TargetPlatform) property in Device.