Support Bzlmod and add rules_scala to bazel-central-registry #1482

sfc-gh-pbennes · 2023-03-16T23:26:47Z

Greetings!

It looks like rules_scala isn't part of the Bzlmod effort or added to bazel-central-registry yet. I've opened an issue both here and in https://github.com/bazelbuild/bazel-central-registry to request it: bazelbuild/bazel-central-registry#522

chrislovecnm · 2023-07-11T15:30:04Z

Just a note to anyone that starts the work native.register_toolchains is not supported when using bzlmod.

sluongng · 2023-07-11T15:35:37Z

@chrislovecnm it's still available here no? https://bazel.build/rules/lib/globals/module#register_toolchains

chrislovecnm · 2023-07-11T15:42:51Z

@sluongng When I last tried it, it did not work when using it inside an extension. In rules_python we are skipping the native call:

https://github.com/bazelbuild/rules_python/blob/36082079b514529d4009a2b104475fde1cdd5b30/python/repositories.bzl#L569

And build the toolchains here:

https://github.com/bazelbuild/rules_python/blob/36082079b514529d4009a2b104475fde1cdd5b30/python/extensions/private/pythons_hub.bzl#L56

We build the toolchains and use the following:

https://github.com/bazelbuild/rules_python/blob/36082079b514529d4009a2b104475fde1cdd5b30/MODULE.bazel#L49

sluongng · 2023-07-11T15:47:13Z

Yeah you would only want to prepare the toolchain (as a new repository) inside the extension I think. Registering it must be called from the MODULE.bazel file.

So native.register_toolchains call from the extension's starlark does not work, but having the extension create @pythons_hub//:all and then MODULE.bazel calling register_toolchains("@pythons_hub//:all") worked. 👍

mateuszkuta256 · 2024-01-24T08:56:24Z

I prepared a minimal bzlmod support here and gonna split it into a few PRs
TODOs after merging it:

write documentation on how to use rules_scala with bzlmod (some properties like SCALA_VERSION will probably be configurable via repo-env)
prepare release notes
contribute repo to the Bazel Central Registry
add tests that the project builds using both WORKSPACE and MODULE.bazel
migrate test/external workspaces to blzmod too

pcj · 2024-06-30T17:14:28Z

@mateuszkuta256 thanks for getting this started! Any progress updates or blockers?

mateuszkuta256 · 2024-07-01T08:54:25Z

@mateuszkuta256 thanks for getting this started! Any progress updates or blockers?

hi, unfortunately I'm working on other things now and won't continue with this PR in the foreseeable future
I remember this migration was on hold because interfered with 'cross compilation support'
It looks like great progress was done it this area, so maybe someone can already take over the PR

mbland · 2024-10-02T17:41:34Z

I'd like to take on the task of Bzlmodifying this repo through a series of pull requests.

I've already created a (mostly) working branch in my own fork. Though I saw a couple draft pull requests here, I ended up taking a different approach and got it mostly working. In fact, I did exactly what @sluongng suggested in #1482 (comment). (I didn't notice this comment before just now—I might've read it, but not understood it at the time—but I did study rules_python and rules_go, and ended up doing exactly that.)

There are still issues I need help to address (recorded in some of the commit messages), I didn't strictly maintain WORKSPACE compatibility (fixable), and I only tested against Scala 2.13. But I can start teasing chunks out of it in a methodical fashion, to ensure that I maintain WORKSPACE and cross-version compatibility.

For an example of what it looks like from a client perspective, here's what the MODULE.bazel stanza from my local EngFlow/example repo looks like (as opposed to the current stanza, explanatory comment notwithstanding):

bazel_dep(name = "rules_scala", repo_name = "io_bazel_rules_scala")
local_path_override(
    module_name = "rules_scala",
    path = "../../bazelbuild/rules_scala"
)   

scala_dev_deps = use_extension(
    "@io_bazel_rules_scala//scala/extensions:deps.bzl",
    "scala_deps",
    dev_dependency = True,
)   
scala_dev_deps.toolchains(
    scalatest = True,
)

So if folks are game for me to do this, I'll start carving off pieces as separate pull requests, and we can resolve any outstanding problems in the process.

cc: @BillyAutrey @jayconrod @benjaminp @TheGrizzlyDev

This begins the Bzlmod compatibility migration by updating Bazel to version 7.3.2 and adding initial `MODULE.bazel` and `WORKSPACE.bzlmod` files. Part of: bazelbuild#1482 Though Bzlmod remains disabled, updating to Bazel 7.3.2 requred updating or adding the following packages to maintain `WORKSPACE` compatibility. In `rules_scala_setup()`: - bazel_skylib: 1.4.1 => 1.7.1 - rules_cc: 0.0.6 => 0.0.10 - rules_java: 5.4.1 => 7.9.0 - rules_proto: 5.3.0-21.7 => 6.0.2 Dev dependencies in `WORKSPACE`: - com_google_protobuf: 28.2 - rules_pkg: 1.0.1 - rules_jvm_external: 6.4 - com_google_absl: abseil-cpp-20240722.0 - zlib: 1.3.1 Of all of the new, explicit dev dependencies, only `com_google_protobuf` will be necessary to include in `MODULE.bazel`. The Bzlmod mechanism will discover these other transitive dev dependencies automatically. Also removed the `rules_java_extra` repo from `WORKSPACE`, which appeared unused. --- Though the current `rules_java` version is 7.12.1, and largely works with this repo, it requires a few temporary workarounds. Rather than commit the workarounds, upgrading only to 7.9.0 now seems less crufty. What follows is a very detailed explanation of what happens with 7.12.1 with Bazel 7.3.2, just to have it on the record. --- The workaround is to change a few toolchain and macro file targets from `@bazel_tools//tools/jdk:` to `@rules_java//toolchains:`. This isn't a terribly bad or invasive workaround, but `@bazel_tools//tools/jdk:` is clearly the canonical path. Best to keep it that way, lest we build up technical debt. Without the workaround, these targets would fail: - //test/src/main/resources/java_sources:CompiledWithJava11 - //test/src/main/resources/java_sources:CompiledWithJava8 - //test/toolchains:java21_toolchain - //test:JunitRuntimePlatform - //test:JunitRuntimePlatform_test_runner - //test:scala_binary_jdk_11 with this error: ```txt ERROR: .../external/rules_java_builtin/toolchains/BUILD:254:14: While resolving toolchains for target @@rules_java_builtin//toolchains:platformclasspath (096dcc8): No matching toolchains found for types @@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type. ``` This appears to be a consequence of both upgrading the Bazel version from 6.3.0 to 7.3.2 and updating `rules_java` to 7.12.1. The `rules_java_builtin` repo is part of the `WORKSPACE` prefix that adds implicit dependencies: - https://bazel.build/external/migration#builtin-default-deps This repo was added to 7.0.0-pre.20231011.2 in the following change, mapped to `@rules_java` within the scope of the `@bazel_tools` repo: - bazelbuild/bazel: Add rules_java_builtin to the users of Java modules bazelbuild/bazel@ff1abb2 This change tried to ensure `rules_java` remained compatible with earlier Bazel versions. However, it changed all instances of `@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type` to `//toolchains:bootstrap_runtime_toolchain_type`: - bazelbuild/rules_java: Make rules_java backwards compatible with Bazel 6.3.0 bazelbuild/rules_java@30ecf3f Bazel has bumped `rules_java` in its `workspace_deps.bzl` from 7.9.0 to 7.11.0, but it's only available as of 8.0.0-pre.20240911.1. - bazelbuild/bazel: Update rules_java 7.11.1 / java_tools 13.8 bazelbuild/bazel#23571 bazelbuild/bazel@f92124a --- What I believe is happening is, under Bazel 7.3.2 and `rules_java` 7.12.1: - Bazel creates `rules_java` 7.9.0 as `@rules_java_builtin` in the `WORKSPACE` prefix. - `@bazel_tools` has `@rules_java` mapped to `@rules_java_builtin` when initialized during the `WORKSPACE` prefix, during which `@bazel_tools//tools/jdk` registers `alias()` targets to `@rules_java` toolchain targets. These aliased toolchains specify `@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type` in their `toolchains` attribute. - `WORKSPACE` loads `@rules_java` 7.12.1 and registers all its toolchains with type `@rules_java//toolchains:bootstrap_runtime_toolchain_type`. - Some `@rules_java` rules explicitly specifying toolchains from `@bazel_tools//tools/jdk` can't find them, because the `@bazel_tools//tools/jdk` toolchain aliases expect toolchains of type `@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type`. This has broken other projects in the same way: - bazelbuild/bazel: [Bazel CI] Downstream project broken by rules_java upgrade #23619 bazelbuild/bazel#23619 These problems don't appear under Bzlmod, and `@rules_java_builtin` was never required. This is because `WORKSPACE` executes its statements sequentially, while Bzlmod builds the module dependency graph _before_ instantiating repositories (within module extensions). It seems a fix is on the way that removes `@rules_java_builtin` from the `WORKSPACE` prefix, and adds `@rules_java` to the suffix. At this moment, though, it's not even in a prerelase: - bazelbuild/bazel: Remove rules_java_builtin in WORKSPACE prefix bazelbuild/bazel@7506690 --- Note that the error message matches that from the following resolved issue, but that issue was for non-Bzlmod child repos when `WORKSPACE` was disabled. - bazelbuild/bazel: Undefined @@rules_java_builtin repository with --noenable_workspace option bazelbuild/bazel#22754

Related to bazelbuild#1482, bazelbuild#1618, and bazelbuild#1619. Results from the investigation documented at: - bazelbuild#1619 (comment) Updates `_import_paths()` in `scala_proto_aspect.bzl` to handle differences `ProtoInfo.proto_source_root` and `ProtoInfo.direct_sources` values between Bazel 6 and Bazel 7. Without this change, `_import_paths()` emits incorrect values under Bazel 7, causing targets containing generated `.proto` inputs to fail, e.g. `//test/proto3:test_generated_proto`. See also: - Fix paths for sibling repository setup and generated .proto files bazelbuild/bazel@6c6c196 - The docstring for `ProtoInfo.proto_source_root` in the Bazel sources: https://github.com/bazelbuild/bazel/blob/7.3.2/src/main/starlark/builtins_bzl/common/proto/proto_info.bzl#L155-L172 - Remove incompatible_generated_protos_in_virtual_imports bazelbuild/bazel@3efaa32 - Comment from: Generated Protos are no longer considered as virtual_imports in Bazel 7 bazelbuild/bazel#21075 (comment) --- I cherrypicked this commit into bazelbuild#1618. While it fixed the `//test/proto3` build failure, it does _not_ fix the hanging scalapb_workers from the ProtoScalaPBRule aspect. I'll have to investiate further whether than hang is related to Bazel, rules_proto, com_google_protobuf, or some mixture thereof. Still, progress!

Part of bazelbuild#1482. Splits the last component off of canonical repo names to produce the expected repo name. Without Bzlmod, it returns the original name. With Bzlmod enabled, it avoids generating output like: scala_import( name = "_main~scala_deps~io_bazel_rules_scala_scala_compiler", jars = ["scala-compiler-2.12.18.jar"], ) resulting in errors like: ``` ERROR: .../_main~_repo_rules~io_bazel_rules_scala/scala/BUILD: no such target '@@_main~scala_deps~io_bazel_rules_scala_scala_compiler//:io_bazel_rules_scala_scala_compiler': target 'io_bazel_rules_scala_scala_compiler' not declared in package '' defined by .../_main~scala_deps~io_bazel_rules_scala_scala_compiler/BUILD and referenced by '@@_main~_repo_rules~io_bazel_rules_scala//scala:default_toolchain_scala_compile_classpath_provider' ``` Also fixes the following error when attaching resources from custom repos to targets under Bzlmod: ```txt $ bazel test //test/src/main/scala/scalarules/test/resources:all 1) Scala library depending on resources from external resource-only jar::allow to load resources(scalarules.test.resources.ScalaLibResourcesFromExternalDepTest) java.lang.NullPointerException at scalarules.test.resources.ScalaLibResourcesFromExternalDepTest.get(ScalaLibResourcesFromExternalDepTest.scala:17) at scalarules.test.resources.ScalaLibResourcesFromExternalDepTest.$anonfun$new$3(ScalaLibResourcesFromExternalDepTest.scala:11) at scalarules.test.resources.ScalaLibResourcesFromExternalDepTest.$anonfun$new$2(ScalaLibResourcesFromExternalDepTest.scala:11) ``` Can be replaced with a future bazel-skylib implementation, if accepted into that repo. --- We can't rely on the specific canonical repository name format: > Repos generated by extensions have canonical names in the form of > `module_repo_canonical_name+extension_name+repo_name`. For extensions > hosted in the root module, the `module_repo_canonical_name` part is > replaced with the string `_main`. Note that the canonical name format is > not an API you should depend on — it's subject to change at any time. > > - https://bazel.build/external/extension#repository_names_and_visibility The change to no longer encode module versions in canonical repo names in Bazel 7.1.0 is a recent example of Bazel maintainers altering the format: - bazelbuild/bazel#21316 And the maintainers recently replaced the `~` delimiter with `+` in the upcoming Bazel 8 release due to build performance issues on Windows: - bazelbuild/bazel#22865 This function assumes the only valid `repo_name` characters are letters, numbers, '_', '-', and '.'. It finds the last character not in this set, and returns the contents of `name` following this character. This is valid so long as this condition holds: - https://github.com/bazelbuild/bazel/blob/7.3.2/src/main/java/com/google/devtools/build/lib/cmdline/RepositoryName.java#L159-L162

Related to #1482, #1618, and #1619. Results from the investigation documented at: - #1619 (comment) Updates `_import_paths()` in `scala_proto_aspect.bzl` to handle differences `ProtoInfo.proto_source_root` and `ProtoInfo.direct_sources` values between Bazel 6 and Bazel 7. Without this change, `_import_paths()` emits incorrect values under Bazel 7, causing targets containing generated `.proto` inputs to fail, e.g. `//test/proto3:test_generated_proto`. See also: - Fix paths for sibling repository setup and generated .proto files bazelbuild/bazel@6c6c196 - The docstring for `ProtoInfo.proto_source_root` in the Bazel sources: https://github.com/bazelbuild/bazel/blob/7.3.2/src/main/starlark/builtins_bzl/common/proto/proto_info.bzl#L155-L172 - Remove incompatible_generated_protos_in_virtual_imports bazelbuild/bazel@3efaa32 - Comment from: Generated Protos are no longer considered as virtual_imports in Bazel 7 bazelbuild/bazel#21075 (comment) --- I cherrypicked this commit into #1618. While it fixed the `//test/proto3` build failure, it does _not_ fix the hanging scalapb_workers from the ProtoScalaPBRule aspect. I'll have to investiate further whether than hang is related to Bazel, rules_proto, com_google_protobuf, or some mixture thereof. Still, progress!

Part of bazelbuild#1482. Splits the last component off of canonical repo names to produce the expected repo name. Without Bzlmod, it returns the original name. With Bzlmod enabled, it avoids generating output like: scala_import( name = "_main~scala_deps~io_bazel_rules_scala_scala_compiler", jars = ["scala-compiler-2.12.18.jar"], ) resulting in errors like: ``` ERROR: .../_main~_repo_rules~io_bazel_rules_scala/scala/BUILD: no such target '@@_main~scala_deps~io_bazel_rules_scala_scala_compiler//:io_bazel_rules_scala_scala_compiler': target 'io_bazel_rules_scala_scala_compiler' not declared in package '' defined by .../_main~scala_deps~io_bazel_rules_scala_scala_compiler/BUILD and referenced by '@@_main~_repo_rules~io_bazel_rules_scala//scala:default_toolchain_scala_compile_classpath_provider' ``` Also fixes the following error when attaching resources from custom repos to targets under Bzlmod: ```txt $ bazel test //test/src/main/scala/scalarules/test/resources:all 1) Scala library depending on resources from external resource-only jar::allow to load resources(scalarules.test.resources.ScalaLibResourcesFromExternalDepTest) java.lang.NullPointerException at scalarules.test.resources.ScalaLibResourcesFromExternalDepTest.get(ScalaLibResourcesFromExternalDepTest.scala:17) at scalarules.test.resources.ScalaLibResourcesFromExternalDepTest.$anonfun$new$3(ScalaLibResourcesFromExternalDepTest.scala:11) at scalarules.test.resources.ScalaLibResourcesFromExternalDepTest.$anonfun$new$2(ScalaLibResourcesFromExternalDepTest.scala:11) ``` Can be replaced with a future bazel-skylib implementation, if accepted into that repo. --- We can't rely on the specific canonical repository name format: > Repos generated by extensions have canonical names in the form of > `module_repo_canonical_name+extension_name+repo_name`. For extensions > hosted in the root module, the `module_repo_canonical_name` part is > replaced with the string `_main`. Note that the canonical name format is > not an API you should depend on — it's subject to change at any time. > > - https://bazel.build/external/extension#repository_names_and_visibility The change to no longer encode module versions in canonical repo names in Bazel 7.1.0 is a recent example of Bazel maintainers altering the format: - bazelbuild/bazel#21316 And the maintainers recently replaced the `~` delimiter with `+` in the upcoming Bazel 8 release due to build performance issues on Windows: - bazelbuild/bazel#22865 This function assumes the only valid `repo_name` characters are letters, numbers, '_', '-', and '.'. It finds the last character not in this set, and returns the contents of `name` following this character. This is valid so long as this condition holds: - https://github.com/bazelbuild/bazel/blob/7.3.2/src/main/java/com/google/devtools/build/lib/cmdline/RepositoryName.java#L159-L162

mbland · 2024-10-11T16:49:17Z

A quick update for visibility: I'm very close to having the Bzlmodified rules_scala passing 100% of the tests, down to the last couple of failures:

I need to replace the bind() calls from twitter_scrooge with alias() or some such to get test_version.sh to pass.
I need to fix the Scala 3.1.2 and Scala 3.3.3 failures for the Scalafmt targets from test/shell/test_cross_build.sh (corresponding to the test3 and library3 test cases in test_scalafmt()).

I also need to get test_lint.sh working again, and update the dt_patches repos for Bzlmod. But the following all pass:

test_rules_scala.sh
test_reproducibility.sh
test_examples.sh
test_coverage.sh

Hopefully I can get these fixed up today, and I'll start peeling off the next pull request or two. And thanks to @simuons and @liucijus for reviewing and merging #1619 and #1620 already.

* Toolchainize //scala/scalafmt:scalafmt_toolchain This is a pretty straightforward and easy update on top of the previous `setup_toolchains` and `scala_toolchains_repo` changes. Part of #1482. * Make test_reproducibility.sh build test/coverage_* Replaces the nonexistent `//test/coverage/...` target with packages that actually exist, rendering the test more meaningful. A spurious failure caused me to run the following to inspect what was happening: ```txt RULES_SCALA_TEST_ONLY=test_build_is_identical ./test_reproducibility.sh ``` This caused me to see that the following command was failing because `//test/coverage/...` didn't exist: ```txt bazel build --collect_code_coverage -- //test/coverage/... ``` * Export `SCALAFMT_TOOLCHAIN_TYPE` constant Reduces string duplication in `//scala/scalafmt/toolchain/*.bzl` files. * Remove scalafmt_toolchain dep_providers default It's likely that no one will ever rely on the default when defining their own `scalafmt_toolchain`. While investigating #1675, I realized the `dep_providers` default was set to a nonexistent target. This didn't break our tests because our Scalafmt toolchains are created by `setup_scala_toolchain`, which sets `dep_providers` explicitly. I thought about aliasing the provider generated for the toolchain for `SCALA_VERSION` in `@io_bazel_rules_scala_toolchains//scalafmt`, or generating a new one. I ultimately decided to remove the default, because the `deps_providers` is literally the only attribute. Anyone defining their own `scalafmt_toolchain` will certainly define their own `deps_providers` target(s). * Update tests for Bazel 6 + Bzlmod Copied the regular expression used to optionally match canonical repo name prefixes in `buildozer` commands from #1622. These never failed when building with Bzlmod and Bazel 7.4.1, but did under Bazel 6.5.0. * Fix `//third_party` tests with `$(rootpath)` Fixes `{strict_deps,unused_dependency_checker}_test` from `//third_party/dependency_analyzer/src/test` under Bazel 8 by updating `$(location)` to `$(rootpath)`. Part of #1652. `//third_party/dependency_analyzer/src/test:strict_deps_test` and `//third_party/dependency_analyzer/src/test:unused_dependency_checker_test` both failed with errors of the form: ```txt StrictDepsTest: - error on indirect dependency target *** FAILED *** (379 milliseconds) nice errors on sequence of strings.this.infos.exists(nice errors on sequence of strings.this.checkErrorContainsMessage(target)) was false expected an error on //commons:Target to appear in errors (with buildozer command)! Errors: List(object apache is not a member of package org) (StrictDepsTest.scala:85) ``` This happened both under `WORKSPACE` and Bzlmod. Copying the test script with the following (with `$ID` being one of `7`, `8`, `7-updated`, or `8-updated`) helped reveal the differences: ```txt cp bazel-bin/third_party/dependency_analyzer/src/test/strict_deps_test \ strict_deps_test-$ID.sh ``` Under Bazel 7, we find the following lines whether using `$(location)` or `$(rootpath)` on the `org.apache.commons` artifact (the other artifacts already use `$(rootpath)`: ```txt -Dscala.library.location=external/io_bazel_rules_scala_scala_library_2_12_20/scala-library-2.12.20.jar -Dscala.reflect.location=external/io_bazel_rules_scala_scala_reflect_2_12_20/scala-reflect-2.12.20.jar -Dguava.jar.location=external/com_google_guava_guava_21_0_with_file/guava-21.0.jar -Dapache.commons.jar.location=external/org_apache_commons_commons_lang_3_5_without_file/commons-lang3-3.5.jar ``` Under Bazel 8, notice that the `external/` prefix has become `../` for the other paths already using `$(rootpath)`, but remains `external/` for the `$(location)` artifact. This breaks the Bazel 8 build: ```txt -Dscala.library.location=../io_bazel_rules_scala_scala_library_2_12_20/scala-library-2.12.20.jar -Dscala.reflect.location=../io_bazel_rules_scala_scala_reflect_2_12_20/scala-reflect-2.12.20.jar -Dguava.jar.location=../com_google_guava_guava_21_0_with_file/guava-21.0.jar -Dapache.commons.jar.location=external/org_apache_commons_commons_lang_3_5_without_file/commons-lang3-3.5.jar ``` Under Bazel 8, using `$(rootpath)` for the `org.apache.commons` artifact converts its `external/`, fixing the Bazel 8 build: ```txt -Dscala.library.location=../io_bazel_rules_scala_scala_library_2_12_20/scala-library-2.12.20.jar -Dscala.reflect.location=../io_bazel_rules_scala_scala_reflect_2_12_20/scala-reflect-2.12.20.jar -Dguava.jar.location=../com_google_guava_guava_21_0_with_file/guava-21.0.jar -Dapache.commons.jar.location=../org_apache_commons_commons_lang_3_5_without_file/commons-lang3-3.5.jar ```

Adds scala_proto toolchains to `scala_toolchains()`. Part of bazelbuild#1482. The most significant part of the change is moving all the toolchain rules from `scala_proto/BUILD` to `setup_scala_toolchains()` in `scala_proto/toolchains.bzl`. Adds the `scala_proto_deps_providers()` macro to replace `//scala_proto:scalapb_{compile,grpc,worker}_deps_provider` targets in the `dep_providers` parameter of `scala_proto_deps_toolchain()`. Examples of this are in `test/proto/custom_generator/BUILD`. Excludes `@scala_proto_rules_scalapb_protoc_gen` from `DEFAULT_SCALAPB_WORKER_DEPS` in `scala_proto/default/default_deps.bzl` for Scala 2.11. For other Scala versions, this repo name will have the Scala version appended. This is to avoid build failures under Bzlmod, since: - This repo is required by ScalaPB 0.11.17, but Scala 2.11 is capped at ScalaPB 0.9.8. - Importing the nonexistent `scala_proto_rules_scalapb_protoc_gen` under Scala 2.11 results in an error under Bzlmod, as does importing it multiple times when configuring multiple Scala versions. - `MODULE.bazel` can iterate over a list of Scala versions, filtering out Scala 2.11, and call `use_repo()` on each version specific repo. A lot of the other changes are more opportunistic removals of `@io_bazel_rules_scala` label prefixes and application of `Label()` where appropriate. Doing this will allow Bzlmod users to use `rules_scala` without setting `repo_name = "@io_bazel_rules_scala"` in `bazel_dep()`.

Adds the jmh toolchain to `scala_toolchains()` and moves `jmh_repositories()` to `jmh/toolchain/toolchain.bzl` for `rules_java` 8 compatibility. Part of bazelbuild#1482 and bazelbuild#1652.

This is the last of the toolchain to receive the "toolchainization" treatment prior to Bzlmodification, and moves `twitter_scrooge()` to `twitter_scrooge/toolchain/toolchain.bzl` for `rules_java` 8 compatibility. Part of bazelbuild#1482 and bazelbuild#1652.

mbland · 2025-01-28T21:52:54Z

Weekly update!

Overview

With the merge of #1678, we're down to three toolchainization pull requests remaining, including the newly opened #1687.

I've broken my one-open-pull-request-at-a-time plan. The new plan is to have open simultaneously at most two orthogonal pull requests:

One toolchainization/Bzlmodification/Bazel 8-ification pull request at a time. This is pretty much a necessity.
One pull request at a time to bump dependency versions that don't affect any core logic from the other PRs.

The reason being, I'm trying to make sure all the dependency versions are as current as they can be by the next major release(s). With the recent changes to scala/deps.bzl, the WORKSPACE files, and scripts/create_repository.py, and the upcoming MODULE.bazel, doing so has become much easier.

As always, I've continued updating my bzlmod and bzlmod-bazel-8 branches with updates from master and various dependencies (rules_java, protobuf, et. al.). I've just updated both branches after testing them with the following Bazel versions:

	`bzlmod`	`bzlmod-bazel-8`
`WORKSPACE`	6.5.0, 7.4.1	6.5.0 (with `--cxxopt=-std=c++17`), 7.4.1, 8.0.1
Bzlmod	6.5.0, 7.4.1	7.4.1, 8.0.1, `rolling`, `last_green`

Merged

Toolchainize //scala/scalafmt:scalafmt_toolchain #1678

Open

Bzlmod pending changes

The bzlmod branch contains all the changes from the following branches, and adds MODULE.bazel files and module extensions.

bzlmod-toolchain-jmh: based on Toolchainize //scala_proto:{,deps_}toolchain_type #1687
bzlmod-toolchain-twitter-scrooge: based on bzlmod-toolchain-jmh, and the last of the toolchainization changes
Bump several Maven artifacts, Go: I'll keep updating this with all the latest dependencies that are compatible with Bazel 6.5.0
Bump Scalafmt from 3.8.3 to 3.8.6: a more significant update that changes scala/scalafmt/scalafmt_repositories.bzl and .scalafmt.conf files, and removes obsolete artifacts from third_party/repositories/scala_*.bzl.
Remove @io_bazel_rules_scala or replace with Label: prevents forcing users to specify repo_name = "io_bazel_rules_scala" in bazel_dep()

Bazel 8 pending changes (#1652)

bzlmod-bazel-8 contains merges from the bzlmod branch and the branch below.

bazel-8-compatibility-split-files: version bumps, WORKSPACE updates resulting from said bumps, updates to third_party/repositories/scala_*.bzl generated by scripts/create_repository.py, and changes to drop Scala 2.11 runs from test_thirdparty_version.sh and test_version.sh

Adds scala_proto toolchains to `scala_toolchains()`. Part of bazelbuild#1482. The most significant part of the change is moving all the toolchain rules from `scala_proto/BUILD` to `setup_scala_toolchains()` in `scala_proto/toolchains.bzl`. Adds the `scala_proto_deps_providers()` macro to replace `//scala_proto:scalapb_{compile,grpc,worker}_deps_provider` targets in the `dep_providers` parameter of `scala_proto_deps_toolchain()`. Examples of this are in `test/proto/custom_generator/BUILD`. Excludes `@scala_proto_rules_scalapb_protoc_gen` from `DEFAULT_SCALAPB_WORKER_DEPS` in `scala_proto/default/default_deps.bzl` for Scala 2.11. For other Scala versions, this repo name will have the Scala version appended. This is to avoid build failures under Bzlmod, since: - This repo is required by ScalaPB 0.11.17, but Scala 2.11 is capped at ScalaPB 0.9.8. - Importing the nonexistent `scala_proto_rules_scalapb_protoc_gen` under Scala 2.11 results in an error under Bzlmod, as does importing it multiple times when configuring multiple Scala versions. - `MODULE.bazel` can iterate over a list of Scala versions, filtering out Scala 2.11, and call `use_repo()` on each version specific repo. A lot of the other changes are more opportunistic removals of `@io_bazel_rules_scala` label prefixes and application of `Label()` where appropriate. Doing this will allow Bzlmod users to use `rules_scala` without setting `repo_name = "@io_bazel_rules_scala"` in `bazel_dep()`.

Adds the jmh toolchain to `scala_toolchains()` and moves `jmh_repositories()` to `jmh/toolchain/toolchain.bzl` for `rules_java` 8 compatibility. Part of bazelbuild#1482 and bazelbuild#1652.

This is the last of the toolchain to receive the "toolchainization" treatment prior to Bzlmodification, and moves `twitter_scrooge()` to `twitter_scrooge/toolchain/toolchain.bzl` for `rules_java` 8 compatibility. Part of bazelbuild#1482 and bazelbuild#1652.

Bumps the following Maven artifact versions, as well as the Go language version. Part of bazelbuild#1482. - `jline`: 3.27.1 => 3.29.0 - `jline-terminal-jni`: 3.27.1 => 3.29.0 - `protobuf-java`: 4.29.0 => 4.29.3 - `sbt-compiler-interface`: 1.10.4 => 1.10.7 - `sbt-util-interface`: 1.10.5 => 1.10.7 - `grpc`: 1.68.1 => 1.70.0 - `google-common-protos`: 2.48.0 => 2.51.0 - `guava`: 33.3.1-jre => 33.4.0-jre - Go: 1.23.4 => 1.23.5 The `protobuf-java` bump ensures compatibility with `protobuf` versions up to v29.3 once we incorporate Bazel 8 compatibility. The other bumps are out of convenience.

Adds scala_proto toolchains to `scala_toolchains()`. Part of #1482. The most significant part of the change is moving all the toolchain rules from `scala_proto/BUILD` to `setup_scala_toolchains()` in `scala_proto/toolchains.bzl`. Adds the `scala_proto_deps_providers()` macro to replace `//scala_proto:scalapb_{compile,grpc,worker}_deps_provider` targets in the `dep_providers` parameter of `scala_proto_deps_toolchain()`. Examples of this are in `test/proto/custom_generator/BUILD`. Excludes `@scala_proto_rules_scalapb_protoc_gen` from `DEFAULT_SCALAPB_WORKER_DEPS` in `scala_proto/default/default_deps.bzl` for Scala 2.11. For other Scala versions, this repo name will have the Scala version appended. This is to avoid build failures under Bzlmod, since: - This repo is required by ScalaPB 0.11.17, but Scala 2.11 is capped at ScalaPB 0.9.8. - Importing the nonexistent `scala_proto_rules_scalapb_protoc_gen` under Scala 2.11 results in an error under Bzlmod, as does importing it multiple times when configuring multiple Scala versions. - `MODULE.bazel` can iterate over a list of Scala versions, filtering out Scala 2.11, and call `use_repo()` on each version specific repo. A lot of the other changes are more opportunistic removals of `@io_bazel_rules_scala` label prefixes and application of `Label()` where appropriate. Doing this will allow Bzlmod users to use `rules_scala` without setting `repo_name = "@io_bazel_rules_scala"` in `bazel_dep()`.

Bumps the following Maven artifact versions, as well as the Go language version. Part of #1482. - `jline`: 3.27.1 => 3.29.0 - `jline-terminal-jni`: 3.27.1 => 3.29.0 - `protobuf-java`: 4.29.0 => 4.29.3 - `sbt-compiler-interface`: 1.10.4 => 1.10.7 - `sbt-util-interface`: 1.10.5 => 1.10.7 - `grpc`: 1.68.1 => 1.70.0 - `google-common-protos`: 2.48.0 => 2.51.0 - `guava`: 33.3.1-jre => 33.4.0-jre - Go: 1.23.4 => 1.23.5 The `protobuf-java` bump ensures compatibility with `protobuf` versions up to v29.3 once we incorporate Bazel 8 compatibility. The other bumps are out of convenience.

Adds the jmh toolchain to `scala_toolchains()` and moves `jmh_repositories()` to `jmh/toolchain/toolchain.bzl` for `rules_java` 8 compatibility. Part of bazelbuild#1482 and bazelbuild#1652.

This is the last of the toolchain to receive the "toolchainization" treatment prior to Bzlmodification, and moves `twitter_scrooge()` to `twitter_scrooge/toolchain/toolchain.bzl` for `rules_java` 8 compatibility. Part of bazelbuild#1482 and bazelbuild#1652.

mbland · 2025-02-05T19:13:40Z

Weekly update!

Overview

We're down to two toolchainization pull requests remaining! I've also been landing orthogonal updates to Maven dependencies to keep up with the latest releases. Thanks to @simuons and @liucijus as always!

I updated my bzlmod and bzlmod-bazel-8 branches yesterday, but am in the process of updating them again to pick up all the latest merges. I've just updated, tested, and pushed bzlmod again, and will do the same for bzlmod-bazel-8 later today.

I haven't yet produced documentation changes or added Publish to BCR files, but I'm likely to prepare both changes soon.

Merged

Open

Bzlmod pending changes

The bzlmod branch contains all the changes from the following branches, and adds MODULE.bazel files and module extensions.

bzlmod-toolchain-twitter-scrooge: based on Toolchainize //jmh/toolchain:jmh_toolchain_type #1691, and the last of the toolchainization changes
Remove @io_bazel_rules_scala or replace with Label: prevents forcing users to specify repo_name = "io_bazel_rules_scala" in bazel_dep()

Bazel 8 pending changes (#1652)

bzlmod-bazel-8 contains merges from the bzlmod branch and the branch below.

bazel-8-compatibility-split-files: version bumps, WORKSPACE updates resulting from said bumps, updates to third_party/repositories/scala_*.bzl generated by scripts/create_repository.py, and changes to drop Scala 2.11 runs from test_thirdparty_version.sh and test_version.sh

Known uses of `bzlmod` and `bzlmod-bazel-8` branches

I'm aware of the following repos using git_override with commits from my bzlmod or bzlmod-bazel-8 branches. Once the official rules_scala releases are ready, I can notify these users to switch over (and maybe even send them pull requests to do so).

bazelbuild/rules_webtesting

bazelbuild/intellij

Finding and reporting Bazel issues

A nice side effect of this work has been discovering and reporting issues in new Bazel releases and pre-release builds.

bazelbuild/bazel#25192: New `repository_ctx.original_name` produces "canonical reproducible" warnings

I hit this yesterday while running dt_patches/dt_patch_test.sh against the last_green Bazel build.

A fix is imminent: Don't suggest updates to private repo rule attributes bazel#25194 (comment)

Note that the new repository_ctx.original_name feature request comes from bazelbuild/bazel#24667, which was inspired by the following issues, which all stemmed from this issue:

bazelbuild/bazel#25198: Bazel 8 `$(location)` expands to `$(execpath)` under bazel build

I discovered this issue when testing with Bazel 8.0.0, fixed it as part of #1678, and mentioned it in an earlier weekly update. However, I didn't dig in until receiving inspiration from @shs96c and @fmeum in a #general thread in the Bazel Slack workspace on 2025-02-03.

Repro repo: mbland/bazel-8-location

Adds the jmh toolchain to `scala_toolchains()` and moves `jmh_repositories()` to `jmh/toolchain/toolchain.bzl` for `rules_java` 8 compatibility. Part of #1482 and #1652.

This is the last of the toolchain to receive the "toolchainization" treatment prior to Bzlmodification, and moves `twitter_scrooge()` to `twitter_scrooge/toolchain/toolchain.bzl` for `rules_java` 8 compatibility. Part of bazelbuild#1482 and bazelbuild#1652.

Prepares for `repository_ctx.original_name` support in Bazel 8.1.0 per bazelbuild/bazel#24467 and bumps Go to 1.23.6. Part of bazelbuild#1482 and bazelbuild#1652. The underlying repo rules will now use `repository_ctx.original_name` if it's available. I've confirmed that these changes use `repository_ctx.original_name` under Bazel 8.1.0rc2 after commenting out the wrappers and restoring the original repo rule symbols. --- This change ensures compatibility with future Bazel releases. New comments indicate where to simplify code after dropping support for Bazel versions that don't have `repository_ctx.original_name`. The Go 1.23.6 bump is a convenience update that isn't essential to the rest of the change. Under Bzlmod, `repository_ctx.name` contains the canonical repository name. This broke the `_alias_repository` and `jvm_import_external` repository rules, which used `repository_ctx.name` to generate default top level target names for their repos. bazelbuild#1650 added wrapper macros passing the original name as an extra attribute to the underlying repo rules as a portable workaround. The Bazel maintainers eventually recognized this as a common use case and added `repository_ctx.original_name` to eliminate the need for such wrappers (eventually). - https://bazel.build/rules/lib/builtins/repository_ctx#name - https://bazel.build/rules/lib/builtins/repository_ctx#original_name

Fulfills most of bazelbuild#1482. Compatible with Bazel 6.5.0 and 7.5.0, but not Bazel 8. Documentation and "Publish to BCR" changes will come in future commits. After that, we can close bazelbuild#1482, or we can wait to add Bazel 8 compatibility and close bazelbuild#1652 at the same time.

Fulfills most of bazelbuild#1482. Compatible with Bazel 6.5.0 and 7.5.0, but not Bazel 8. Documentation and "Publish to BCR" changes will come in future commits. After that, we can publish a new major version and then close bazelbuild#1482. Per bazelbuild#1647, `com_google_protobuf` remains at v21.7, and versions up to v25.5 can work after bumping `com_google_absl` to 20240722.0 and setting C++17 compiler flags in `.bazelrc`. This change enables Bazel 6 and 7 users to migrate their `rules_scala` dependency from `WORKSPACE` to `MODULE.bazel` whenever they're ready. After the next major version release includes this change, we can update `com_google_protobuf`, `rules_java`, and related dependencies as part of enabling Bazel 8 compatibility (bazelbuild#1652). The `rules_jvm_external` update in that change will effectively end Bzlmod support for Bazel 6.5.0, effectively requiring another major version release after that.

sfc-gh-pbennes mentioned this issue Mar 16, 2023

wanted: bazelbuild/rules_scala bazelbuild/bazel-central-registry#522

Open

1 task

liucijus added the help wanted label Mar 17, 2023

mateuszkuta256 mentioned this issue Jan 24, 2024

Draft: Add bzlmod minimal support #1541

Closed

mbland mentioned this issue Apr 15, 2024

Convert WORKSPACE to MODULE.bazel, update README EngFlow/example#312

Merged

agluszak mentioned this issue Apr 22, 2024

Migrate from sbt to bazel build-server-protocol/build-server-protocol#615

Merged

agluszak mentioned this issue Aug 7, 2024

Draft: Add bzlmod minimal support (continuation) #1595

Closed

mbland mentioned this issue Oct 2, 2024

Transfer rules_scala to bazel-contrib #1616

Open

4 tasks

This was referenced Oct 3, 2024

Bump Bazel and dependencies to prepare for Bzlmod #1618

Closed

Bazel 7 compatibility updates #1619

Merged

mbland mentioned this issue Oct 5, 2024

Fix Bazel 7 related protobuf build failures #1620

Merged

liucijus closed this as completed in #1619 Oct 8, 2024

simuons reopened this Oct 8, 2024

This was referenced Jan 27, 2025

Toolchainize //scala_proto:{,deps_}toolchain_type #1687

Merged

Bump protoc-bridge to 0.9.8 for Scala >= 2.12 #1688

Merged

Wyverald mentioned this issue Jan 27, 2025

[Bazel CI] Missing rules_java dependency for aswb package bazelbuild/intellij#7232

Open

mbland mentioned this issue Feb 3, 2025

Bump several Maven artifacts, Go #1690

Merged

mbland mentioned this issue Feb 4, 2025

New repository_ctx.original_name produces "canonical reproducible" warnings bazelbuild/bazel#25192

Closed

This was referenced Feb 5, 2025

Toolchainize //jmh/toolchain:jmh_toolchain_type #1691

Merged

Bump Scalafmt from 3.8.3 to 3.8.6 #1692

Open

mbland mentioned this issue Feb 6, 2025

Toolchainize twitter_scrooge #1693

Open

mbland mentioned this issue Feb 6, 2025

Use rctx.original_name if available, Go 1.23.6 #1694

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support Bzlmod and add rules_scala to bazel-central-registry #1482

Support Bzlmod and add rules_scala to bazel-central-registry #1482

sfc-gh-pbennes commented Mar 16, 2023 •

edited

Loading

chrislovecnm commented Jul 11, 2023

sluongng commented Jul 11, 2023

chrislovecnm commented Jul 11, 2023 •

edited

Loading

sluongng commented Jul 11, 2023

mateuszkuta256 commented Jan 24, 2024

pcj commented Jun 30, 2024 •

edited

Loading

mateuszkuta256 commented Jul 1, 2024

mbland commented Oct 2, 2024

mbland commented Oct 11, 2024

mbland commented Jan 28, 2025

mbland commented Feb 5, 2025

Support Bzlmod and add rules_scala to bazel-central-registry #1482

Support Bzlmod and add rules_scala to bazel-central-registry #1482

Comments

sfc-gh-pbennes commented Mar 16, 2023 • edited Loading

chrislovecnm commented Jul 11, 2023

sluongng commented Jul 11, 2023

chrislovecnm commented Jul 11, 2023 • edited Loading

sluongng commented Jul 11, 2023

mateuszkuta256 commented Jan 24, 2024

pcj commented Jun 30, 2024 • edited Loading

mateuszkuta256 commented Jul 1, 2024

mbland commented Oct 2, 2024

mbland commented Oct 11, 2024

mbland commented Jan 28, 2025

Overview

Merged

Open

Bzlmod pending changes

Bazel 8 pending changes (#1652)

mbland commented Feb 5, 2025

Overview

Merged

Open

Bzlmod pending changes

Bazel 8 pending changes (#1652)

Known uses of bzlmod and bzlmod-bazel-8 branches

bazelbuild/rules_webtesting

bazelbuild/intellij

Finding and reporting Bazel issues

bazelbuild/bazel#25192: New repository_ctx.original_name produces "canonical reproducible" warnings

bazelbuild/bazel#25198: Bazel 8 $(location) expands to $(execpath) under bazel build

sfc-gh-pbennes commented Mar 16, 2023 •

edited

Loading

chrislovecnm commented Jul 11, 2023 •

edited

Loading

pcj commented Jun 30, 2024 •

edited

Loading

Known uses of `bzlmod` and `bzlmod-bazel-8` branches

bazelbuild/bazel#25192: New `repository_ctx.original_name` produces "canonical reproducible" warnings

bazelbuild/bazel#25198: Bazel 8 `$(location)` expands to `$(execpath)` under bazel build