Build: Generated Dart code is not visible to other builders if library is resolved twice

[simolus3]

Problem

We have two builders (one custom defined below, one from built_value) generating different Dart files that should be part of a user defined library:

// this is the input, called example.dart
import 'package:built_value/built_value.dart';

part 'example.sample.dart'; // generated by a custom builder, source below
part 'example.g.dart'; // generated by built_value with source_gen

abstract class Example implements Built<Example, ExampleBuilder> {
  SomeClass get entry; // defined in "example.sample.dart"

  Example._();
  factory Example([void Function(ExampleBuilder) updates]) = _$Example;
}

The custom builder looks like this, it basically writes SomeClass into a part file for the example.dart file above:

import 'dart:async';
import 'package:build/build.dart';
Builder createSomeBuilder([BuilderOptions options]) => SimpleBuilder();

class SimpleBuilder extends Builder {
  @override
  FutureOr<void> build(BuildStep buildStep) async {
    final input = buildStep.inputId;
    log.fine('Running build-use-generated-dart on ${input.path}');
    if (input.path.endsWith('example.dart')) {
      final output = input.changeExtension('.sample.dart');

      // the issue only occurs when the next line is present!!
      await buildStep.inputLibrary;

      await buildStep.writeAsString(output, r'''
part of 'example.dart';
class SomeClass {
  final String field;
  const SomeClass(this.field);
}''');
    }
  }
  @override
  final Map<String, List<String>> buildExtensions = const {
    'dart': ['sample.dart']
  };
}

When the await buildStep.inputLibrary is commented out, the build runs without a problem and the generated code from built_value contains the SomeClass field.
However, when await buildStep.inputLibrary is uncommented, it looks like built_value is unable to resolve the generated class:

[SEVERE] built_value_generator:built_value on lib/example.dart:
Error in BuiltValueGenerator for abstract class Example implements Built<Example, dynamic>.
Please make the following changes to use BuiltValue:

1. Make field entry have non-dynamic type. If you are already specifying a type, please make sure the type is correctly imported.
package:built_value_generator/src/value_source_class.dart 575:28        ValueSourceClass.generateCode
package:built_value_generator/built_value_generator.dart 53:52          BuiltValueGenerator.generate
package:source_gen/src/builder.dart 280:35                              _generate
package:source_gen/src/builder.dart 73:15                               _Builder._generateForLibrary
package:source_gen/src/builder.dart 67:11                               _Builder.build
package:build                                                           runBuilder

We have a build.yaml configured so that the built_value_generator will always run second.

I've uploaded the full example here: https://drive.google.com/file/d/1KQ-XA0I-ouHKzxGChHIDXZg24pto37Lu/view?usp=sharing

Potential cause

I don't know really how the Resolver works, but this seems like it's caching the analyzer result for inputLibrary. When the analyzer is invoked at a point in which example.sample.dart has already been generated, there is no problem.
However, if the first builder already uses await buildStep.inputLibrary, the analysis result at that point (which obviously doesn't include SomeClass) will also be used on built_value, causing the problem.

If there is a cache, maybe a file should be invalidated whenever a part or import changes?

Versions

My Dart SDK version is Dart VM version: 2.4.0 (Wed Jun 19 11:53:45 2019 +0200) on "linux_x64". I use the following version of packages which could be relevant here:

• analyzer: 0.36.4
• build: 1.1.5
• build_resolvers: 1.0.6
• build_runner: 1.6.5
• build_runner_core: 3.0.7
• built_value_generator: 6.7.0

[jakemac53]

Thanks for the detailed bug report!

I will take a look at this and see if I can track it down.

[jakemac53]

I can repro this consistently with a very minimal project, so that is a good first step. Still tracking down the root cause though.

[jakemac53]

I still haven't quite been able to track this down, and I also haven't been able to create a repro using analysis_driver directly following the same steps we do. Continuing to dig though.

[natebosch]

@jakemac53 - do I remember correctly that we did repro this if we omitted the driver.changeFile call for newly visible files, but that even when we started including that call for new files in build_resolvers we still had an issue?

[jakemac53]

Yes - in my isolated repro I was able to reproduce this if I omitted that call for new files, but adding that call in for build_resolvers didn't similarly solve the problem.

[thosakwe]

I think this is related, at least in part, to https://github.com/dart-lang/build/issues/2146. I can send in a PR, but I'm not 100% sure where to start (though resolver.libraryFor(assetId) was mentioned in the other issue).

[srikrsna]

I'm also stuck because of this. Is there anything I can do to help?

[simolus3]

I tried to dig around some more. I couldn't fix the problem, but I found some interesting things that might be helpful:

1. We don't call AnalysisDriverScheduler.waitForIdle() after calling AnalysisDriver.notifyChanged. I'm not sure if we should do that, it doesn't fix the problem.
2. The result of _parseDirectives in build_asset_uri_resolver.dart won't contain part of-directives because they don't inherit from UriBasedDirective. That doesn't fix the problem either.
3. When the second builder calls buildStep.inputLibrary, the library has a different AnalysisSession than the one from the first build step, so the driver definitely gets notified. The generated part file is __even listed in LibraryElement.parts__! However, library.parts.single.source == null, which I think is the root cause here. I couldn't figure out how the compilation unit can exist without a source, but in my understanding that's not supposed to happen.

[elbeshenrique]

Any uptades on this matter?

[jakemac53]

No updates, we haven't been able to diagnose the root cause yet.

Note that you can now within a single builder resolve libraries that you have already written, so that may be a workaround for some use cases (if you control all the builders and you can group them into one builder where they run sequentially).

[rrousselGit]

Is there anything we can do to help or any workaround when we don't control all builders?

[jakemac53]

Unfortunately we don't have a workaround at this time.

If you use a LibraryBuilder instead of a PartBuilder is the issue still present? It might be something specifically related to parts.

[rrousselGit]

From what I've tried, yes the problem is still there with LibraryBuilder

I have a very limited understanding of that part though, so it is not impossible that I'm testing it incorrectly.

[simolus3]

Some background information, which might be helpful to other build authors: This use case not working with PartBuilder is expected due to the nature of how it operates, which essentially boils down to:

1. builder a generates input.builder_a.part
2. builder b generates input.builder_b.part
3. source_gen concatenates the two part files into input.g.dart

Of course, the input only has a part 'input.g.dart' directive, so no builder can rely on any generated types here, the part file is generated after they finish. This is expected when using PartBuilder.

With LibraryBuilder it should be a different thing, since each builder generates the part file directly and the input is supposed to reference it. Here, what's happening is this:

1. builder a generates input.builder_a.dart, which is referenced as part directive in input.dart.
2. builder b generates input.builder_b.dart, which is also referenced in the input

With an appropriate build config that runs the builder in a serialized order, we'd expect that builder b has access to types generated by builder a. However, this only appears to be the case if builder a doesn't use the analyzer. That's what this issue is about, but it won't fix anything for PartBuilders at the moment (which is what most source_gen users use).

---

@jakemac53 I tried debugging some more, and I found something that seems fishy. I don't know if that's the root problem, maybe you know more. By looking at the resolved LibraryElement received by the second builder, the part file has a null-Source, which means the analyzer can't read it.

I've added some debug print statements to resolveAbsolute to print all calls and report whether a lookup did work:

diff --git a/build_resolvers/lib/src/build_asset_uri_resolver.dart b/build_resolvers/lib/src/build_asset_uri_resolver.dart
index c30128c1..961c0a1b 100644
--- a/build_resolvers/lib/src/build_asset_uri_resolver.dart
+++ b/build_resolvers/lib/src/build_asset_uri_resolver.dart
@@ -130,11 +130,17 @@ class BuildAssetUriResolver extends UriResolver {

   @override
   Source resolveAbsolute(Uri uri, [Uri actualUri]) {
+    print('resolveAbsolute($uri, $actualUri)');
     final cachedId = lookupAsset(uri);
-    if (cachedId == null) return null;
-    return resourceProvider
+    if (cachedId == null) {
+      print('=> cachedId == null');
+      return null;
+    }
+    final source = resourceProvider
         .getFile(assetPath(cachedId))
         .createSource(cachedId.uri);
+    print('=> source == $source');
+    return source;
   }

   @override

The first builder prints:

[WARNING] build_use_generated_dart:first on lib/example.dart:
resolveAbsolute(package:build_use_generated_dart/example.dart, package:build_use_generated_dart/example.dart)
=> source == /build_use_generated_dart/lib/example.dart
[WARNING] build_use_generated_dart:first on lib/example.dart:
resolveAbsolute(package:build_use_generated_dart/example.sample.dart, package:build_use_generated_dart/example.sample.dart)
=> lookupAsset(uri) == null

This is expected, since package:build_use_generated_dart/example.sample.dart doesn't exist at the time where the first builder resolves its input. The first builder outputs the part file in question. _However_, there are no calls to resolveAbsolute while the second builder runs! This means that the analyzer can't know that example.sample.dart now has a source, meaning that it's not picked up. I don't know if the analyzer keeps an internal cache of uri -> source or why it's not getting called a second time, but I think this is closely related to the cause here. That would also explain why this works fine if the first builder doesn't use the resolver.

[jakemac53]

@scheglov would you mind taking a look based on the recent things that @simolus3 discovered?

I seem to remember that returning null here might be invalid? Do we need to return some sort of fake source instead?

[rrousselGit]

This is not directly related but:

If this issue is caused because the first builder is reading the analyzer, which cache it for the later builders, then what was the purpose of run_before until now?

[jakemac53]

If I recall the runs_before feature was originally added for builders that weren't outputting dart files.

[scheglov]

When UriResolver.resolveAbsolute() returns null, this means that this UriResolver does not know how to handle this URI. This is not an error per se, because it is completely fine for the UriResolver that handles dart:xyz URI(s) to return null for a package: URI. There is probably another UriResovler that understands it. OTOH, if there is no UriResolver for a URI, then this URI is unresolved, and reported as an error, and the corresponding ImportDirective or PartDirective.uriSource will be null.

But if the UriResolver is supposed to handle this URI, because the URI is valid, it has to return a Source for the file that corresponds to the URI. Even if the file does not exist. UriResolver job is not reading file, but converting URIs to file paths.

[jakemac53]

See the linked PR which adds a (skipped) test that can easily be ran to debug this issue. That should help some.

[rrousselGit]

I can confirm that it fixed the problem. Thanks a lot, both you and @simolus3 😄