Trying experimental performance work

[davidmorgan]

Forked from #3800, "Discussions" seems a better place for people to add feedback, so let's try it here :)

@gmpassos it will be annoying to try without a flag; a flag will also be useful for me, so I'll look at adding one #3831.

In terms of feedback, performance measurements are actually not particularly useful yet as I still have a lot to do there :) but it would be helpful for people to start checking to see if I actually broke something, i.e. if the new version is incorrect somehow.

Here are dependency overrides that use my PR adding the flag, so if you put this in your pubspec.yaml and pass --use-experimental-resolver when you run build_runner build or build_runner watch then that's the new codepath :)

dependency_overrides:
  build:
    git:
      url: https://github.com/dart-lang/build
      path: build
      ref: 6e752db2bb54e2ff043364888e1489883d54eb99
  build_config:
    git:
      url: https://github.com/dart-lang/build
      path: build_config
      ref: 6e752db2bb54e2ff043364888e1489883d54eb99
  build_modules:
    git:
      url: https://github.com/dart-lang/build
      path: build_modules
      ref: 6e752db2bb54e2ff043364888e1489883d54eb99
  build_resolvers:
    git:
      url: https://github.com/dart-lang/build
      path: build_resolvers
      ref: 6e752db2bb54e2ff043364888e1489883d54eb99
  build_runner:
    git:
      url: https://github.com/dart-lang/build
      path: build_runner
      ref: 6e752db2bb54e2ff043364888e1489883d54eb99
  build_runner_core:
    git:
      url: https://github.com/dart-lang/build
      path: build_runner_core
      ref: 6e752db2bb54e2ff043364888e1489883d54eb99
  build_test:
    git:
      url: https://github.com/dart-lang/build
      path: build_test
      ref: 6e752db2bb54e2ff043364888e1489883d54eb99
  build_web_compilers:
    git:
      url: https://github.com/dart-lang/build
      path: build_web_compilers
      ref: 6e752db2bb54e2ff043364888e1489883d54eb99

[davidmorgan]

@gmpassos wrote:

I’ve added your dependency_overrides to our backend project, which requires reflection files to be generated.

NOTE: This is a preliminary benchmark and should not be considered final until all optimizations are fully implemented.

To force a full build, I’m running the following commands on macOS:

• Standard mode:

  dart run build_runner clean && dart run build_runner build --delete-conflicting-outputs

  Executed 5 times:

  [INFO] Running build completed, took 32.5s  
  [INFO] Running build completed, took 32.2s  
  [INFO] Running build completed, took 33.7s  
  [INFO] Running build completed, took 32.6s  
  [INFO] Running build completed, took 32.6s  
  

• Experimental mode:

  dart run build_runner clean && dart run build_runner build --delete-conflicting-outputs --use-experimental-resolver

  Executed 5 times:

  [INFO] Running build completed, took 37.2s  
  [INFO] Running build completed, took 37.2s  
  [INFO] Running build completed, took 37.0s  
  [INFO] Running build completed, took 37.2s  
  [INFO] Running build completed, took 37.6s  
  

[davidmorgan]

Thanks @gmpassos :) ... I do expect to improve initial build time, but for what it's worth, I'm more interested in incremental build time.

Actually the benchmark measures three things: clean build, no changes rebuild, rebuild after changes. And "rebuild after changes" is the most interesting one since that's what people spend most of their time waiting for.

To measure "rebuild after changes" you need to modify a file that's an input of the generator(s), for example put // CACHEBUSTER in lib/app.dart then a script:

#!/bin/bash --

random="$RANDOM$RANDOM$RANDOM"
sed -i -e "s#CACHEBUSTER.*#CACHEBUSTER$random#" lib/app.dart

The incremental build is also more interesting for correctness--it's where bugs are most likely to creep in in the new implementation :)

[gmpassos]

@davidmorgan

For our project, which is a standard backend with routes and a database, the initial build time is almost the same as the incremental build time because the dependency graph for each BuildStep is fully determined by the import tree. Since it's common to modify files shared across many backend modules, a change in a single file typically affects almost all modules.

[gmpassos]

@davidmorgan FYI:

I was investigating where the build process takes the most time.
The first call to BuildStep.inputLibrary takes 18 seconds, while subsequent calls (for each build step) are much faster since they resolve nearly the same code tree.

[davidmorgan]

Thanks--there's no investigation needed at this point though, I have several things to fix that will be big improvements. Then we can check again :)

[amrgetment]

it is failed

[SEVERE] freezed on lib/core/data/models/operation_result.dart:

NoSuchMethodError: Class '_Resource' has no instance getter 'modificationStamp'.
Receiver: Instance of '_Resource'
Tried calling: modificationStamp
dart:core                                                           Object.noSuchMethod
package:build_resolvers/src/analysis_driver_filesystem.dart 185:56  _Resource.noSuchMethod
package:build_resolvers/src/analysis_driver_filesystem.dart 152:7   _Resource.modificationStamp
package:analyzer/source/file_source.dart 66:12                      FileSource.contentsFromFile
package:analyzer/source/file_source.dart 53:12                      FileSource.contents
package:freezed/src/templates/concrete_template.dart 567:25         parseTypeSource
package:freezed/src/templates/properties.dart 79:13                 Property.fromFormalParameter
package:freezed/src/freezed_generator.dart 407:30                   FreezedGenerator._parseConstructorsNeedsGeneration
package:freezed/src/freezed_generator.dart 98:47                    FreezedGenerator.parseDeclaration
package:freezed/src/parse_generator.dart 56:28                      ParserGenerator.generate
package:source_gen/src/builder.dart 342:23                          _generate
dart:async                                                          Stream.toList.<fn>
package:source_gen/src/builder.dart 107:9                           _Builder._generateForLibrary
package:source_gen/src/builder.dart 99:5                            _Builder.build

[davidmorgan]

Thanks @amrgetment :) ... yes, I noticed that one, fixed here: #3835 ... so update your dep overrides to f3d7620091c3c59ff511475274b50ab92c342e8d to pull in that one.

[amrgetment]

I am your beta tester, mention me anytime you want to test

[davidmorgan]

New version build_runner 2.4.15 now published.

No particular difference expected yet, hopefully I didn't break anything :) ... the --use-experimental-resolver flag is there now.

[amrgetment]

@davidmorgan It seems like 2x now
[INFO] Succeeded after 3m 41s with 997 outputs (13915 actions)
that's impressive ❤️

[amrgetment]

with experimental-resolver pr without it same result
dart run build_runner build --delete-conflicting-outputs --use-experimental-resolver
[INFO] Succeeded after 3m 29s with 5161 outputs (12625 actions)

[josh-audio]

Hey @davidmorgan, thanks for your work here. I'm sure I speak for a lot of folks in saying that it's exciting to see work going towards build performance.

Just wanted to add results from my project here, in case they might be helpful. These results come from https://github.com/anthem-audio/anthem/. Some notes about my usage of build:

• I have two purpose-built generators which are both reading all files in lib (see build.yaml) which is around 180 files as of writing, and I also pull in mobx_codegen
• I have around 30 - 40 Dart files that actually trigger output from my generators
• I am outputting just over 30 Dart files, and the same number of C++ files
• One of my generators uses a part builder from source_gen, the other does not

If you want to reproduce these tests, you can clone the repo above, and do the following:

1. Update the package version for build_runner in pubspec.yaml and codegen/pubspec.yaml to the latest
2. Run dart pub get
3. Run dart run build_runner build, or dart run build_runner build --use-experimental-resolver

I am running these tests on Windows 11 / x64.

---

Current resolver

Below are timing results from a few runs, with the current behavior.

Here is the command I'm running:

dart run build_runner build --delete-conflicting-outputs 

I am making trivial edits to lib/model/project.dart to hopefully simulate real-world usage.

And the results:

[INFO] Succeeded after 27.3s with 1158 outputs (2379 actions)
[INFO] Succeeded after 23.3s with 601 outputs (1265 actions)
[INFO] Succeeded after 21.2s with 340 outputs (743 actions)
[INFO] Succeeded after 21.4s with 278 outputs (619 actions)
[INFO] Succeeded after 21.2s with 265 outputs (593 actions)

I find the decreasing actions here interesting, though I haven't paid much attention to this before and it may be normal. I ran a build_runner clean and initial build_runner build before running this test, which may be relevant.

---

Experimental resolver

Here's the command I'm using for the experimental resolver:

dart run build_runner build --use-experimental-resolver --delete-conflicting-outputs

And the results:

[INFO] Succeeded after 22.2s with 260 outputs (583 actions)
[INFO] Succeeded after 21.2s with 260 outputs (583 actions)
[INFO] Succeeded after 21.8s with 260 outputs (583 actions)
[INFO] Succeeded after 22.0s with 260 outputs (583 actions)
[INFO] Succeeded after 21.8s with 260 outputs (583 actions)

[davidmorgan]

Thanks Josh! Good to have another benchmark+check :)

[davidmorgan]

I'd say I'm about 70% of the way through the refactoring needed before I can make real performance improvements :) ... you can see from the stream of PRs what I meant when I said it's complicated ;)

Right now it should actually be a bit slower than before. But still I'd appreciate some test runs please, to check whether I broke anything :) ... the refactoring affects both running with and without --use-experimental-resolver, but there should be no change in functionality and not much change in performance. (Except probably slightly slower).

The latest hash for your pubspec.yaml if you'd like to try it is: 9c3d5f4

Thanks :)

[josh-audio]

Hey @davidmorgan, just validated against your latest merged commit, which is d71fe7d as of writing, using the dependency_overrides as suggested above. All looks good so far 🎉

[davidmorgan]

Great, thanks :)