A Dart build script that downloads the Protobuf compiler and Dart plugin to streamline .proto to .dart compilation.

I checked out your repo and tried the example and it doesn't seem to work with the latest Dart?

Overall, love the idea behind this package. :)

PS C:\Users\micro\Documents\GitHub\dart-protoc-builder\example> dart run build_runner build
[INFO] Generating build script completed, took 532ms
[INFO] Precompiling build script... completed, took 8.2s
[INFO] Building new asset graph completed, took 1.2s
[INFO] Checking for unexpected pre-existing outputs. completed, took 3ms[INFO] Running build completed, took 8ms
[INFO] Caching finalized dependency graph completed, took 44ms
[INFO] Succeeded after 72ms with 0 outputs (0 actions)
PS C:\Users\micro\Documents\GitHub\dart-protoc-builder\example> dart --version
Dart SDK version: 2.17.3 (stable) (Wed Jun 1 11:06:41 2022 +0200) on "windows_x64"
PS C:\Users\micro\Documents\GitHub\dart-protoc-builder\example> flutter 
--version
Flutter 3.0.2 • channel stable • https://github.com/flutter/flutter.git
Framework • revision cd41fdd495 (11 days ago) • 2022-06-08 09:52:13     
-0700
Engine • revision f15f824b57
Tools • Dart 2.17.3 • DevTools 2.12.2

# Generated by pub
# See https://dart.dev/tools/pub/glossary#lockfile
packages:
  _fe_analyzer_shared:
    dependency: transitive
    description:
      name: _fe_analyzer_shared
      url: "https://pub.dartlang.org"
    source: hosted
    version: "40.0.0"
  analyzer:
    dependency: transitive
    description:
      name: analyzer
      url: "https://pub.dartlang.org"
    source: hosted
    version: "4.1.0"
  archive:
    dependency: transitive
    description:
      name: archive
      url: "https://pub.dartlang.org"
    source: hosted
    version: "3.3.0"
  args:
    dependency: transitive
    description:
      name: args
      url: "https://pub.dartlang.org"
    source: hosted
    version: "2.3.1"
  async:
    dependency: transitive
    description:
      name: async
      url: "https://pub.dartlang.org"
    source: hosted
    version: "2.9.0"
  build:
    dependency: transitive
    description:
      name: build
      url: "https://pub.dartlang.org"
    source: hosted
    version: "2.3.0"
  build_config:
    dependency: transitive
    description:
      name: build_config
      url: "https://pub.dartlang.org"
    source: hosted
    version: "1.0.0"
  build_daemon:
    dependency: transitive
    description:
      name: build_daemon
      url: "https://pub.dartlang.org"
    source: hosted
    version: "3.1.0"
  build_resolvers:
    dependency: transitive
    description:
      name: build_resolvers
      url: "https://pub.dartlang.org"
    source: hosted
    version: "2.0.9"
  build_runner:
    dependency: "direct dev"
    description:
      name: build_runner
      url: "https://pub.dartlang.org"
    source: hosted
    version: "2.1.11"
  build_runner_core:
    dependency: transitive
    description:
      name: build_runner_core
      url: "https://pub.dartlang.org"
    source: hosted
    version: "7.2.3"
  built_collection:
    dependency: transitive
    description:
      name: built_collection
      url: "https://pub.dartlang.org"
    source: hosted
    version: "5.1.1"
  built_value:
    dependency: transitive
    description:
      name: built_value
      url: "https://pub.dartlang.org"
    source: hosted
    version: "8.3.3"
  checked_yaml:
    dependency: transitive
    description:
      name: checked_yaml
      url: "https://pub.dartlang.org"
    source: hosted
    version: "2.0.1"
  code_builder:
    dependency: transitive
    description:
      name: code_builder
      url: "https://pub.dartlang.org"
    source: hosted
    version: "4.1.0"
  collection:
    dependency: transitive
    description:
      name: collection
      url: "https://pub.dartlang.org"
    source: hosted
    version: "1.16.0"
  convert:
    dependency: transitive
    description:
      name: convert
      url: "https://pub.dartlang.org"
    source: hosted
    version: "3.0.2"
  crypto:
    dependency: transitive
    description:
      name: crypto
      url: "https://pub.dartlang.org"
    source: hosted
    version: "3.0.2"
  dart_style:
    dependency: transitive
    description:
      name: dart_style
      url: "https://pub.dartlang.org"
    source: hosted
    version: "2.2.3"
  file:
    dependency: transitive
    description:
      name: file
      url: "https://pub.dartlang.org"
    source: hosted
    version: "6.1.2"
  fixnum:
    dependency: transitive
    description:
      name: fixnum
      url: "https://pub.dartlang.org"
    source: hosted
    version: "1.0.1"
  frontend_server_client:
    dependency: transitive
    description:
      name: frontend_server_client
      url: "https://pub.dartlang.org"
    source: hosted
    version: "2.1.3"
  glob:
    dependency: transitive
    description:
      name: glob
      url: "https://pub.dartlang.org"
    source: hosted
    version: "2.1.0"
  googleapis_auth:
    dependency: transitive
    description:
      name: googleapis_auth
      url: "https://pub.dartlang.org"
    source: hosted
    version: "1.3.1"
  graphs:
    dependency: transitive
    description:
      name: graphs
      url: "https://pub.dartlang.org"
    source: hosted
    version: "2.1.0"
  grpc:
    dependency: "direct main"
    description:
      name: grpc
      url: "https://pub.dartlang.org"
    source: hosted
    version: "3.0.2"
  http:
    dependency: transitive
    description:
      name: http
      url: "https://pub.dartlang.org"
    source: hosted
    version: "0.13.4"
  http2:
    dependency: transitive
    description:
      name: http2
      url: "https://pub.dartlang.org"
    source: hosted
    version: "2.0.0"
  http_multi_server:
    dependency: transitive
    description:
      name: http_multi_server
      url: "https://pub.dartlang.org"
    source: hosted
    version: "3.2.1"
  http_parser:
    dependency: transitive
    description:
      name: http_parser
      url: "https://pub.dartlang.org"
    source: hosted
    version: "4.0.1"
  io:
    dependency: transitive
    description:
      name: io
      url: "https://pub.dartlang.org"
    source: hosted
    version: "1.0.3"
  js:
    dependency: transitive
    description:
      name: js
      url: "https://pub.dartlang.org"
    source: hosted
    version: "0.6.4"
  json_annotation:
    dependency: transitive
    description:
      name: json_annotation
      url: "https://pub.dartlang.org"
    source: hosted
    version: "4.5.0"
  lints:
    dependency: "direct dev"
    description:
      name: lints
      url: "https://pub.dartlang.org"
    source: hosted
    version: "1.0.1"
  logging:
    dependency: transitive
    description:
      name: logging
      url: "https://pub.dartlang.org"
    source: hosted
    version: "1.0.2"
  matcher:
    dependency: transitive
    description:
      name: matcher
      url: "https://pub.dartlang.org"
    source: hosted
    version: "0.12.12"
  meta:
    dependency: transitive
    description:
      name: meta
      url: "https://pub.dartlang.org"
    source: hosted
    version: "1.8.0"
  mime:
    dependency: transitive
    description:
      name: mime
      url: "https://pub.dartlang.org"
    source: hosted
    version: "1.0.2"
  package_config:
    dependency: transitive
    description:
      name: package_config
      url: "https://pub.dartlang.org"
    source: hosted
    version: "2.1.0"
  path:
    dependency: transitive
    description:
      name: path
      url: "https://pub.dartlang.org"
    source: hosted
    version: "1.8.2"
  pool:
    dependency: transitive
    description:
      name: pool
      url: "https://pub.dartlang.org"
    source: hosted
    version: "1.5.1"
  protobuf:
    dependency: "direct main"
    description:
      name: protobuf
      url: "https://pub.dartlang.org"
    source: hosted
    version: "2.1.0"
  protoc_builder:
    dependency: "direct dev"
    description:
      path: ".."
      relative: true
    source: path
    version: "0.2.1+1"
  pub_semver:
    dependency: transitive
    description:
      name: pub_semver
      url: "https://pub.dartlang.org"
    source: hosted
    version: "2.1.1"
  pubspec_parse:
    dependency: transitive
    description:
      name: pubspec_parse
      url: "https://pub.dartlang.org"
    source: hosted
    version: "1.2.0"
  shelf:
    dependency: transitive
    description:
      name: shelf
      url: "https://pub.dartlang.org"
    source: hosted
    version: "1.3.1"
  shelf_web_socket:
    dependency: transitive
    description:
      name: shelf_web_socket
      url: "https://pub.dartlang.org"
    source: hosted
    version: "1.0.2"
  source_span:
    dependency: transitive
    description:
      name: source_span
      url: "https://pub.dartlang.org"
    source: hosted
    version: "1.9.0"
  stack_trace:
    dependency: transitive
    description:
      name: stack_trace
      url: "https://pub.dartlang.org"
    source: hosted
    version: "1.10.0"
  stream_channel:
    dependency: transitive
    description:
      name: stream_channel
      url: "https://pub.dartlang.org"
    source: hosted
    version: "2.1.0"
  stream_transform:
    dependency: transitive
    description:
      name: stream_transform
      url: "https://pub.dartlang.org"
    source: hosted
    version: "2.0.0"
  string_scanner:
    dependency: transitive
    description:
      name: string_scanner
      url: "https://pub.dartlang.org"
    source: hosted
    version: "1.1.1"
  term_glyph:
    dependency: transitive
    description:
      name: term_glyph
      url: "https://pub.dartlang.org"
    source: hosted
    version: "1.2.1"
  timing:
    dependency: transitive
    description:
      name: timing
      url: "https://pub.dartlang.org"
    source: hosted
    version: "1.0.0"
  typed_data:
    dependency: transitive
    description:
      name: typed_data
      url: "https://pub.dartlang.org"
    source: hosted
    version: "1.3.1"
  watcher:
    dependency: transitive
    description:
      name: watcher
      url: "https://pub.dartlang.org"
    source: hosted
    version: "1.0.1"
  web_socket_channel:
    dependency: transitive
    description:
      name: web_socket_channel
      url: "https://pub.dartlang.org"
    source: hosted
    version: "2.2.0"
  yaml:
    dependency: transitive
    description:
      name: yaml
      url: "https://pub.dartlang.org"
    source: hosted
    version: "3.1.1"
sdks:
  dart: ">=2.16.0 <3.0.0"

Without this, the build system won't notice that a .proto file has been changed and will not invoke the builder again.

Tested this in the example/ directory with dart run build_runner watch and making changes to the proto/my_proto.proto file and watching the resulting .dart files be regenerated.

Fixes #1

Believe it or not, being safe and using path.join to account for Window's backslash instead of a forward slash was actually breaking Windows builds. The reason is that these aren't actual paths processed by the system, they're more like templates passed to package:build, which expects forward slashes and didn't recognize the back-slash.

This PR replaces the use of path.join with a hard-coded / instead.

Fixes #5

Summary

Bumped the default protoc_plugin version to the latest v20.0.1 (which fixes mono_repo configuration after a breaking change)

Notes

protoc_plugin v20.0.1 adds a local relative dependency override to the protobuf package so I needed to also unpack and pub get it

The latest dart protoc plugin version uses relative paths in the protoc-plugin folder (../protobuf) - this causes the dart pub get to fail as the filter removes any other folders.

Suggested: update the test to include protobuf when decompressing.

Without this, the build system won't notice that a .proto file has been changed and will not invoke the builder again.

I updated this with another commit that ensures that output files will be recreated if they are deleted. However, due to a minor bug, it seems that it only watches the first output file in the list of buildExtensions, so if one removes the '{{}}.pb.dart' file, it will trigger a regenerate, but removing the other files won't.

Tested this in the example/ directory with dart run build_runner watch and making changes to the proto/my_proto.proto file and watching the resulting .dart files be regenerated.

Fixes #1

Add support of the base google's types:

• google/protobuf/any.proto
• google/protobuf/api.proto
• google/protobuf/descriptor.proto
• google/protobuf/duration.proto
• google/protobuf/empty.proto
• google/protobuf/field_mask.proto
• google/protobuf/source_context.proto
• google/protobuf/struct.proto
• google/protobuf/timestamp.proto
• google/protobuf/type.proto
• google/protobuf/wrappers.proto

Currently, a change to the underlying .proto file doesn't invalidate the build asset cache for some reason, and will not call ProtocBuilder. It seems even that one can delete the generated files, and further calls to build_runner build will not rebuild the output.

I run the builder, it generates succesfully however I get the following error:

[SEVERE] protoc_builder on lib/proto/service.proto:

FileSystemException: Cannot open file, path = 'lib/lib/config/protos/service.pb.dart' (OS Error: The system cannot find the path specified.
, errno = 3)
[INFO] 28.5s elapsed, 301/317 actions completed.
[INFO] Running build completed, took 29.1s

looking at the hiearchy I can see the generated files exist.

This is my build.yaml:

targets:
  $default:
    builders:
      drift_dev:
        options:
          generate_connect_constructor: false
# generate_values_in_copy_with (defaults to true): Generates a Value<T?> instead of T?
# for nullable columns in copyWith. This allows to set columns back to null (by using Value(null)).
# Passing null was ignored before, making it impossible to set columns to null.
          generate_values_in_copy_with : false
      protoc_builder:
        options:
          # The version of the Protobuf compiler to use.
          protobuf_version: "3.19.1" # Make sure to use quotation marks.
          # The version of the Dart protoc_plugin package to use.
          protoc_plugin_version: "20.0.0" # Make sure to use quotation marks.
          # Include paths given to the Protobuf compiler during compilation.
          proto_paths:
            - "lib/proto/"
          # The root directory for generated Dart output files.
          out_dir: "lib/config/protos"

If I change my build.yaml where out_dir: "./config/protos" then no files get generated.

reference: https://github.com/pikaju/dart-protoc-builder/blob/90966ef0f1fd38ec56dcec9b42710d11d3943c43/lib/src/protoc_plugin_download.dart#L49-L51

The plugin's "executable" is just running dart (the vm) - loading and parsing each and every execution. I've noticed this to be a significant slowdown.

    // Compile the program for faster startup.
    await Process.run('dart', ['compile', 'exe', 'bin/protoc_plugin.dart'],
        workingDirectory: protocPluginPackageDirectory.path);

    // Move the program so no one can tell any better....
    // dart compile always writes ".exe" file on every system.
    final protocBin = '${protocPluginPackageDirectory.path/bin/protoc_plugin.exe';
    await File(protocBin).rename(protocPlugin .path);