Skip to content

Commit

Permalink
Use distinct avro-compiler and avro-test configuration
Browse files Browse the repository at this point in the history
  • Loading branch information
@RustedBones
RustedBones committed Jan 7, 2025
1 parent ebc6f82 commit df0e4a0
Show file tree
Hide file tree
Showing 10 changed files with 62 additions and 96 deletions.
65 changes: 32 additions & 33 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -36,31 +36,31 @@ to generate the avro classes.

### Project settings

| Name | Default | Description |
|:-------------------------------|:-------------------------------------------------------------------------|:----------------------------------------------------------------------------------------|
| `avroAdditionalDependencies` | `avro-compiler % avroVersion % "avro"`, `avro % avroVersion % "compile"` | Additional dependencies to be added to library dependencies. |
| `avroCompiler` | `com.github.sbt.avro.AvroCompilerBridge` | Sbt avro compiler class. |
| `avroCreateSetters` | `true` | Generate setters. |
| `avroEnableDecimalLogicalType` | `true` | Use `java.math.BigDecimal` instead of `java.nio.ByteBuffer` for logical type `decimal`. |
| `avroFieldVisibility` | `public` | Field visibility for the properties. Possible values: `private`, `public`. |
| `avroOptionalGetters` | `false` (requires avro `1.10+`) | Generate getters that return `Optional` for nullable fields. |
| `avroStringType` | `CharSequence` | Type for representing strings. Possible values: `CharSequence`, `String`, `Utf8`. |
| `avroVersion` | `1.12.0` | Avro version to use in the project. |
| Name | Default | Description |
|:-------------------------------|:------------------------------------------------------------------------------------|:----------------------------------------------------------------------------------------|
| `avroAdditionalDependencies` | `avro-compiler % avroVersion % "avro-compiler"`<br>`avro % avroVersion % "compile"` | Additional dependencies to be added to library dependencies. |
| `avroCompiler` | `com.github.sbt.avro.AvroCompilerBridge` | Sbt avro compiler class. |
| `avroCreateSetters` | `true` | Generate setters. |
| `avroEnableDecimalLogicalType` | `true` | Use `java.math.BigDecimal` instead of `java.nio.ByteBuffer` for logical type `decimal`. |
| `avroFieldVisibility` | `public` | Field visibility for the properties. Possible values: `private`, `public`. |
| `avroOptionalGetters` | `false` (requires avro `1.10+`) | Generate getters that return `Optional` for nullable fields. |
| `avroStringType` | `CharSequence` | Type for representing strings. Possible values: `CharSequence`, `String`, `Utf8`. |
| `avroVersion` | `1.12.0` | Avro version to use in the project. |

### Scoped settings (Compile/Test)

| Name | Default | Description |
|:-------------------------------------------|:-----------------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------------------------|
| `avroGenerate` / `target` | `sourceManaged` / `compiled_avro` / `$config` | Source directory for generated `.java` files. |
| `avroSource` | `sourceDirectory` / `$config` / `avro` | Default Avro source directory for `*.avsc`, `*.avdl` and `*.avpr` files. |
| `avroSpecificRecords` | `Seq.empty` | List of fully qualified Avro record class names to recompile with current avro version and settings. |
| `avroDependencyIncludeFilter` | `Compile`: `avro` classifier artifacts in `Avro` config<br>`Test`: nothing | Filter for including modules containing avro dependencies. |
| `avroUmanagedSourceDirectories` | `Seq(avroSource)` | Unmanaged Avro source directories, which contain manually created sources. |
| `avroUnpackDependencies` / `excludeFilter` | `HiddenFileFilter` | Filter for excluding avro specification files from unpacking. |
| `avroUnpackDependencies` / `includeFilter` | `AllPassFilter` | Filter for including avro specification files to unpack. |
| `avroUnpackDependencies` / `target` | `sourceManaged` / `avro` / `$config` | Target directory for schemas packaged in the dependencies |
| `packageAvro` / `artifactClassifier` | `Some("avro")` | Classifier for avro artifact |
| `packageAvro` / `publishArtifact` | `false` | Enable / Disable avro artifact publishing |
| Name | Default | Description |
|:-------------------------------------------|:----------------------------------------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------------------------|
| `avroGenerate` / `target` | `sourceManaged` / `compiled_avro` / `$config` | Source directory for generated `.java` files. |
| `avroSource` | `sourceDirectory` / `$config` / `avro` | Default Avro source directory for `*.avsc`, `*.avdl` and `*.avpr` files. |
| `avroSpecificRecords` | `Seq.empty` | List of fully qualified Avro record class names to recompile with current avro version and settings. |
| `avroDependencyIncludeFilter` | `Compile`: avro classifier artifacts in `Avro` config<br>`Test`: avro classifier artifacts in `AvroTest` config | Filter for including modules containing avro dependencies. |
| `avroUmanagedSourceDirectories` | `Seq(avroSource)` | Unmanaged Avro source directories, which contain manually created sources. |
| `avroUnpackDependencies` / `excludeFilter` | `HiddenFileFilter` | Filter for excluding avro specification files from unpacking. |
| `avroUnpackDependencies` / `includeFilter` | `AllPassFilter` | Filter for including avro specification files to unpack. |
| `avroUnpackDependencies` / `target` | `sourceManaged` / `avro` / `$config` | Target directory for schemas packaged in the dependencies |
| `packageAvro` / `artifactClassifier` | `Some("avro")` | Classifier for avro artifact |
| `packageAvro` / `publishArtifact` | `false` | Enable / Disable avro artifact publishing |


## Scoped Tasks (Compile/Test)
Expand All @@ -83,13 +83,14 @@ If you depend on an artifact with previously generated avro java classes with st
you can recompile them with `String` by also adding the following

```sbt
Compile / avroSpecificRecords += "com.example.MyAvroRecord" // lib must be added in the avro scope
libraryDependencies += "org" % "name" % "rev" % "avro-compiler"
Compile / avroSpecificRecords += "com.example.MyAvroRecord"
```

## Packaging Avro files

Avro sources (`*.avsc`, `*.avdl` and `*.avpr` files) can be packaged in a separate jar with the `source` type and
`avro` classifier by running `packageAvro`.
Avro sources (`*.avsc`, `*.avdl` and `*.avpr` files) can be packaged in a separate jar with `avro` classifier
by running `packageAvro`.

By default, `sbt-avro` does not publish this. You can enable it with

Expand All @@ -102,23 +103,21 @@ Compile / packageAvro / publishArtifact := true
You can specify a dependency on an avro source artifact that contains the schemas like so:

```sbt
libraryDependencies += "org" % "name" % "rev" % "avro" classifier "avro"
libraryDependencies += "org" % "name" % "rev" % "avro" classifier "avro" intransitive()
```

If some avro schemas are not packaged in a `source/avro` artifact, you can update the `avroDependencyIncludeFilter`
If some avro schemas are not packaged in a `avro` artifact, you can update the `avroDependencyIncludeFilter`
setting to instruct the plugin to look for schemas in the desired dependency:

```sbt
libraryDependencies += "org" % "name" % "rev" % "avro" // module containing avro schemas
libraryDependencies += "org" % "name" % "rev" % "avro" intransitive() // module containing avro schemas
Compile / avroDependencyIncludeFilter := configurationFilter("avro") && moduleFilter(organization = "org", name = "name")
```

If some artifact is meant to be used in the test scope only, you can do the following

```sbt
libraryDependencies += "org" % "name" % "rev" % "avro" classifier "avro"
Compile / avroDependencyIncludeFilter ~= { old => old -- moduleFilter(organization = "org", name = "name") }
Test / avroDependencyIncludeFilter := configurationFilter("avro") && moduleFilter(organization = "org", name = "name")
libraryDependencies += "org" % "name" % "rev" % "avro-test" classifier "avro" intransitive()
```

# License
Expand All @@ -127,8 +126,8 @@ This program is distributed under the BSD license. See the file `LICENSE` for mo

# Credits

`sbt-avro` is maintained by the [sbt Community](http://www.scala-sbt.org/release/docs/Community-Plugins.html). The
initial code was based on a similar plugin: [`sbt-protobuf`](https://github.com/gseitz/sbt-protobuf). Feel free to file
`sbt-avro` is maintained by the [sbt Community](http://www.scala-sbt.org/release/docs/Community-Plugins.html).
The code was based on a similar plugin: [`sbt-protoc`](https://github.com/thesamet/sbt-protoc). Feel free to file
issues or pull requests.

# Contributors
Expand Down
35 changes: 19 additions & 16 deletions plugin/src/main/scala/com/github/sbt/avro/SbtAvro.scala
View file
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,10 @@ object SbtAvro extends AutoPlugin {
// Force Log4J to not use JMX to avoid duplicate mbeans registration due to multiple classloader
sys.props("log4j2.disableJmx") = "true"

val AvroCompiler: Configuration = config("avro-compiler")
val Avro: Configuration = config("avro")
val AvroTest: Configuration = config("avro-test")

val AvroClassifier = "avro"

private[avro] val AvroAvrpFilter: NameFilter = "*.avpr"
Expand Down Expand Up @@ -67,11 +70,11 @@ object SbtAvro extends AutoPlugin {
avroUnpackDependencies / target := sourceManaged.value / "avro",
avroGenerate / target := sourceManaged.value / "compiled_avro",
// setup avro configuration. Use library management to fetch the compiler and schema sources
ivyConfigurations ++= Seq(Avro),
ivyConfigurations ++= Seq(AvroCompiler, Avro, AvroTest),
avroVersion := "1.12.0",
avroAdditionalDependencies := Seq(
"com.github.sbt" % "sbt-avro-compiler-bridge" % BuildInfo.version % Avro,
"org.apache.avro" % "avro-compiler" % avroVersion.value % Avro,
"com.github.sbt" % "sbt-avro-compiler-bridge" % BuildInfo.version % AvroCompiler,
"org.apache.avro" % "avro-compiler" % avroVersion.value % AvroCompiler,
"org.apache.avro" % "avro" % avroVersion.value
),
libraryDependencies ++= avroAdditionalDependencies.value
Expand All @@ -84,13 +87,10 @@ object SbtAvro extends AutoPlugin {
avroSpecificRecords := Seq.empty,
// dependencies
avroDependencyIncludeFilter := (configuration.value match {
case Compile =>
// avro classifier artifact in Avro config are considered for compile scope
configurationFilter(Avro.name) && artifactFilter(classifier = AvroClassifier)
case _ =>
// ignore all dependencies for scopes other than compile
configurationFilter(NothingFilter)
}),
case Compile => configurationFilter(Avro.name)
case Test => configurationFilter(AvroTest.name)
case _ => configurationFilter(NothingFilter)
}) && artifactFilter(classifier = AvroClassifier),
avroUnpackDependencies / includeFilter := AllPassFilter,
avroUnpackDependencies / excludeFilter := HiddenFileFilter,
avroUnpackDependencies / target := configSrcSub(avroUnpackDependencies / target).value,
Expand Down Expand Up @@ -123,16 +123,16 @@ object SbtAvro extends AutoPlugin {

override def requires: Plugins = sbt.plugins.JvmPlugin

override def projectConfigurations: Seq[Configuration] = Seq(Avro)
override def projectConfigurations: Seq[Configuration] = Seq(AvroCompiler, Avro, AvroTest)

override lazy val projectSettings: Seq[Setting[?]] = defaultSettings ++
inConfig(Avro)(Defaults.configSettings) ++
Seq(AvroCompiler, Avro, AvroTest).flatMap(c => inConfig(c)(Defaults.configSettings)) ++
Seq(Compile, Test).flatMap(c => inConfig(c)(configScopedSettings))

// This filter is meant evaluate for all dependant submodules
// eg. source files / unpack dependencies
private val filterDependsOn = ScopeFilter(
inDependencies(ThisProject, transitive = false),
inDependencies(ThisProject),
inConfigurations(Compile)
)

Expand All @@ -150,14 +150,15 @@ object SbtAvro extends AutoPlugin {
inStyle = FilesInfo.lastModified,
outStyle = FilesInfo.exists
) { deps =>
val filter = includeFilter -- excludeFilter

// dedicated directory per artifact to avoid name conflicts
val depTarget = extractTarget / jar.base
IO.createDirectory(depTarget)
deps.flatMap { dep =>
val filter = includeFilter -- excludeFilter
val (avroSpecs, filtered) = IO
.unzip(dep, depTarget, AvroFilter)
.partition(filter.accept)
.partition(_.relativeTo(depTarget).forall(filter.accept))
IO.delete(filtered)
if (avroSpecs.nonEmpty) {
streams.log.info("Extracted from " + dep + avroSpecs.mkString(":\n * ", "\n * ", ""))
Expand All @@ -183,10 +184,12 @@ object SbtAvro extends AutoPlugin {
crossPaths.value
)

// Classpaths.managedJars does not cross-build
val avroArtifacts = update.value
.filter(avroDependencyIncludeFilter.value)
.toSeq
.map { case (_, _, _, f) => f }
.distinct

val unpacked = unpack(
cacheBaseDirectory = cacheBaseDirectory,
Expand Down Expand Up @@ -243,7 +246,7 @@ object SbtAvro extends AutoPlugin {

// TODO Cache class loader
val avroClassLoader = new URLClassLoader(
(Avro / dependencyClasspath).value
(AvroCompiler / dependencyClasspath).value
.map(toNioPath)
.map(_.toUri.toURL)
.toArray,
Expand Down
6 changes: 2 additions & 4 deletions plugin/src/sbt-test/sbt-avro/avscparser/build.sbt
View file
Original file line number Diff line number Diff line change
@@ -1,5 +1,3 @@


lazy val parser = project
.in(file("parser"))
.settings(
Expand All @@ -14,7 +12,7 @@ lazy val parser = project
lazy val root = project
.in(file("."))
.enablePlugins(SbtAvro)
.dependsOn(parser % "avro")
.dependsOn(parser % "avro-compiler")
.settings(
avroCompiler := "com.github.sbt.avro.CustomAvroCompiler"
)
)
14 changes: 3 additions & 11 deletions plugin/src/sbt-test/sbt-avro/local-dependency/build.sbt
View file
Original file line number Diff line number Diff line change
Expand Up @@ -39,29 +39,21 @@ lazy val `transitive`: Project = project
name := "transitive",
version := "0.0.1-SNAPSHOT",
libraryDependencies ++= Seq(
// when using avro scope, it won't be part of the pom dependencies -> intransitive
// to declare transitive dependency use the compile scope
"com.github.sbt" % "external" % "0.0.1-SNAPSHOT" classifier "avro"
),
Compile / avroDependencyIncludeFilter := artifactFilter(classifier = "avro"),
// create a test jar with a schema as resource
Test / packageBin / publishArtifact := true,
("com.github.sbt" % "external" % "0.0.1-SNAPSHOT" % "avro").classifier("avro").intransitive()
)
)

lazy val root: Project = project
.in(file("."))
.enablePlugins(SbtAvro)
.dependsOn(`transitive` % "avro->avro")
.dependsOn(`transitive` % "avro")
.settings(commonSettings)
.settings(
name := "local-dependency",
crossScalaVersions := Seq("2.13.15", "2.12.20"),
libraryDependencies ++= Seq(
"org.specs2" %% "specs2-core" % "4.20.9" % Test
),
// add additional avro source test jar
Test / avroDependencyIncludeFilter := artifactFilter(name = "transitive", classifier = "tests"),

Compile / checkUnpacked := {
exists((`transitive` / crossTarget).value / "src_managed" / "avro" / "main" / "external-avro" / "avdl.avdl")
exists((`transitive` / crossTarget).value / "src_managed" / "avro" / "main" / "external-avro" / "avpr.avpr")
Expand Down
10 changes: 0 additions & 10 deletions ...sbt-test/sbt-avro/local-dependency/src/test/scala/com/github/sbt/avro/test/AvroTest.scala
View file

This file was deleted.

0 ...github/sbt/avro/test/transitive/avsc.avsc → ...ndency/transitive/src/main/avro/avsc.avsc
View file
File renamed without changes.
Loading

0 comments on commit df0e4a0

Please sign in to comment.