diff --git a/README.md b/README.md index b3d8ff7..8cd86b2 100644 --- a/README.md +++ b/README.md @@ -36,31 +36,31 @@ to generate the avro classes. ### Project settings -| Name | Default | Description | -|:-------------------------------|:-------------------------------------------------------------------------|:----------------------------------------------------------------------------------------| -| `avroAdditionalDependencies` | `avro-compiler % avroVersion % "avro"`, `avro % avroVersion % "compile"` | Additional dependencies to be added to library dependencies. | -| `avroCompiler` | `com.github.sbt.avro.AvroCompilerBridge` | Sbt avro compiler class. | -| `avroCreateSetters` | `true` | Generate setters. | -| `avroEnableDecimalLogicalType` | `true` | Use `java.math.BigDecimal` instead of `java.nio.ByteBuffer` for logical type `decimal`. | -| `avroFieldVisibility` | `public` | Field visibility for the properties. Possible values: `private`, `public`. | -| `avroOptionalGetters` | `false` (requires avro `1.10+`) | Generate getters that return `Optional` for nullable fields. | -| `avroStringType` | `CharSequence` | Type for representing strings. Possible values: `CharSequence`, `String`, `Utf8`. | -| `avroVersion` | `1.12.0` | Avro version to use in the project. | +| Name | Default | Description | +|:-------------------------------|:------------------------------------------------------------------------------------|:----------------------------------------------------------------------------------------| +| `avroAdditionalDependencies` | `avro-compiler % avroVersion % "avro-compiler"`
`avro % avroVersion % "compile"` | Additional dependencies to be added to library dependencies. | +| `avroCompiler` | `com.github.sbt.avro.AvroCompilerBridge` | Sbt avro compiler class. | +| `avroCreateSetters` | `true` | Generate setters. | +| `avroEnableDecimalLogicalType` | `true` | Use `java.math.BigDecimal` instead of `java.nio.ByteBuffer` for logical type `decimal`. | +| `avroFieldVisibility` | `public` | Field visibility for the properties. Possible values: `private`, `public`. | +| `avroOptionalGetters` | `false` (requires avro `1.10+`) | Generate getters that return `Optional` for nullable fields. | +| `avroStringType` | `CharSequence` | Type for representing strings. Possible values: `CharSequence`, `String`, `Utf8`. | +| `avroVersion` | `1.12.0` | Avro version to use in the project. | ### Scoped settings (Compile/Test) -| Name | Default | Description | -|:-------------------------------------------|:-----------------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------------------------| -| `avroGenerate` / `target` | `sourceManaged` / `compiled_avro` / `$config` | Source directory for generated `.java` files. | -| `avroSource` | `sourceDirectory` / `$config` / `avro` | Default Avro source directory for `*.avsc`, `*.avdl` and `*.avpr` files. | -| `avroSpecificRecords` | `Seq.empty` | List of fully qualified Avro record class names to recompile with current avro version and settings. | -| `avroDependencyIncludeFilter` | `Compile`: `avro` classifier artifacts in `Avro` config
`Test`: nothing | Filter for including modules containing avro dependencies. | -| `avroUmanagedSourceDirectories` | `Seq(avroSource)` | Unmanaged Avro source directories, which contain manually created sources. | -| `avroUnpackDependencies` / `excludeFilter` | `HiddenFileFilter` | Filter for excluding avro specification files from unpacking. | -| `avroUnpackDependencies` / `includeFilter` | `AllPassFilter` | Filter for including avro specification files to unpack. | -| `avroUnpackDependencies` / `target` | `sourceManaged` / `avro` / `$config` | Target directory for schemas packaged in the dependencies | -| `packageAvro` / `artifactClassifier` | `Some("avro")` | Classifier for avro artifact | -| `packageAvro` / `publishArtifact` | `false` | Enable / Disable avro artifact publishing | +| Name | Default | Description | +|:-------------------------------------------|:----------------------------------------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------------------------| +| `avroGenerate` / `target` | `sourceManaged` / `compiled_avro` / `$config` | Source directory for generated `.java` files. | +| `avroSource` | `sourceDirectory` / `$config` / `avro` | Default Avro source directory for `*.avsc`, `*.avdl` and `*.avpr` files. | +| `avroSpecificRecords` | `Seq.empty` | List of fully qualified Avro record class names to recompile with current avro version and settings. | +| `avroDependencyIncludeFilter` | `Compile`: avro classifier artifacts in `Avro` config
`Test`: avro classifier artifacts in `AvroTest` config | Filter for including modules containing avro dependencies. | +| `avroUmanagedSourceDirectories` | `Seq(avroSource)` | Unmanaged Avro source directories, which contain manually created sources. | +| `avroUnpackDependencies` / `excludeFilter` | `HiddenFileFilter` | Filter for excluding avro specification files from unpacking. | +| `avroUnpackDependencies` / `includeFilter` | `AllPassFilter` | Filter for including avro specification files to unpack. | +| `avroUnpackDependencies` / `target` | `sourceManaged` / `avro` / `$config` | Target directory for schemas packaged in the dependencies | +| `packageAvro` / `artifactClassifier` | `Some("avro")` | Classifier for avro artifact | +| `packageAvro` / `publishArtifact` | `false` | Enable / Disable avro artifact publishing | ## Scoped Tasks (Compile/Test) @@ -83,13 +83,14 @@ If you depend on an artifact with previously generated avro java classes with st you can recompile them with `String` by also adding the following ```sbt -Compile / avroSpecificRecords += "com.example.MyAvroRecord" // lib must be added in the avro scope +libraryDependencies += "org" % "name" % "rev" % "avro-compiler" +Compile / avroSpecificRecords += "com.example.MyAvroRecord" ``` ## Packaging Avro files -Avro sources (`*.avsc`, `*.avdl` and `*.avpr` files) can be packaged in a separate jar with the `source` type and -`avro` classifier by running `packageAvro`. +Avro sources (`*.avsc`, `*.avdl` and `*.avpr` files) can be packaged in a separate jar with `avro` classifier +by running `packageAvro`. By default, `sbt-avro` does not publish this. You can enable it with @@ -102,23 +103,21 @@ Compile / packageAvro / publishArtifact := true You can specify a dependency on an avro source artifact that contains the schemas like so: ```sbt -libraryDependencies += "org" % "name" % "rev" % "avro" classifier "avro" +libraryDependencies += "org" % "name" % "rev" % "avro" classifier "avro" intransitive() ``` -If some avro schemas are not packaged in a `source/avro` artifact, you can update the `avroDependencyIncludeFilter` +If some avro schemas are not packaged in a `avro` artifact, you can update the `avroDependencyIncludeFilter` setting to instruct the plugin to look for schemas in the desired dependency: ```sbt -libraryDependencies += "org" % "name" % "rev" % "avro" // module containing avro schemas +libraryDependencies += "org" % "name" % "rev" % "avro" intransitive() // module containing avro schemas Compile / avroDependencyIncludeFilter := configurationFilter("avro") && moduleFilter(organization = "org", name = "name") ``` If some artifact is meant to be used in the test scope only, you can do the following ```sbt -libraryDependencies += "org" % "name" % "rev" % "avro" classifier "avro" -Compile / avroDependencyIncludeFilter ~= { old => old -- moduleFilter(organization = "org", name = "name") } -Test / avroDependencyIncludeFilter := configurationFilter("avro") && moduleFilter(organization = "org", name = "name") +libraryDependencies += "org" % "name" % "rev" % "avro-test" classifier "avro" intransitive() ``` # License @@ -127,8 +126,8 @@ This program is distributed under the BSD license. See the file `LICENSE` for mo # Credits -`sbt-avro` is maintained by the [sbt Community](http://www.scala-sbt.org/release/docs/Community-Plugins.html). The -initial code was based on a similar plugin: [`sbt-protobuf`](https://github.com/gseitz/sbt-protobuf). Feel free to file +`sbt-avro` is maintained by the [sbt Community](http://www.scala-sbt.org/release/docs/Community-Plugins.html). +The code was based on a similar plugin: [`sbt-protoc`](https://github.com/thesamet/sbt-protoc). Feel free to file issues or pull requests. # Contributors diff --git a/plugin/src/main/scala/com/github/sbt/avro/SbtAvro.scala b/plugin/src/main/scala/com/github/sbt/avro/SbtAvro.scala index 6c0bc85..489d082 100644 --- a/plugin/src/main/scala/com/github/sbt/avro/SbtAvro.scala +++ b/plugin/src/main/scala/com/github/sbt/avro/SbtAvro.scala @@ -15,7 +15,10 @@ object SbtAvro extends AutoPlugin { // Force Log4J to not use JMX to avoid duplicate mbeans registration due to multiple classloader sys.props("log4j2.disableJmx") = "true" + val AvroCompiler: Configuration = config("avro-compiler") val Avro: Configuration = config("avro") + val AvroTest: Configuration = config("avro-test") + val AvroClassifier = "avro" private[avro] val AvroAvrpFilter: NameFilter = "*.avpr" @@ -67,11 +70,11 @@ object SbtAvro extends AutoPlugin { avroUnpackDependencies / target := sourceManaged.value / "avro", avroGenerate / target := sourceManaged.value / "compiled_avro", // setup avro configuration. Use library management to fetch the compiler and schema sources - ivyConfigurations ++= Seq(Avro), + ivyConfigurations ++= Seq(AvroCompiler, Avro, AvroTest), avroVersion := "1.12.0", avroAdditionalDependencies := Seq( - "com.github.sbt" % "sbt-avro-compiler-bridge" % BuildInfo.version % Avro, - "org.apache.avro" % "avro-compiler" % avroVersion.value % Avro, + "com.github.sbt" % "sbt-avro-compiler-bridge" % BuildInfo.version % AvroCompiler, + "org.apache.avro" % "avro-compiler" % avroVersion.value % AvroCompiler, "org.apache.avro" % "avro" % avroVersion.value ), libraryDependencies ++= avroAdditionalDependencies.value @@ -84,13 +87,10 @@ object SbtAvro extends AutoPlugin { avroSpecificRecords := Seq.empty, // dependencies avroDependencyIncludeFilter := (configuration.value match { - case Compile => - // avro classifier artifact in Avro config are considered for compile scope - configurationFilter(Avro.name) && artifactFilter(classifier = AvroClassifier) - case _ => - // ignore all dependencies for scopes other than compile - configurationFilter(NothingFilter) - }), + case Compile => configurationFilter(Avro.name) + case Test => configurationFilter(AvroTest.name) + case _ => configurationFilter(NothingFilter) + }) && artifactFilter(classifier = AvroClassifier), avroUnpackDependencies / includeFilter := AllPassFilter, avroUnpackDependencies / excludeFilter := HiddenFileFilter, avroUnpackDependencies / target := configSrcSub(avroUnpackDependencies / target).value, @@ -123,16 +123,16 @@ object SbtAvro extends AutoPlugin { override def requires: Plugins = sbt.plugins.JvmPlugin - override def projectConfigurations: Seq[Configuration] = Seq(Avro) + override def projectConfigurations: Seq[Configuration] = Seq(AvroCompiler, Avro, AvroTest) override lazy val projectSettings: Seq[Setting[?]] = defaultSettings ++ - inConfig(Avro)(Defaults.configSettings) ++ + Seq(AvroCompiler, Avro, AvroTest).flatMap(c => inConfig(c)(Defaults.configSettings)) ++ Seq(Compile, Test).flatMap(c => inConfig(c)(configScopedSettings)) // This filter is meant evaluate for all dependant submodules // eg. source files / unpack dependencies private val filterDependsOn = ScopeFilter( - inDependencies(ThisProject, transitive = false), + inDependencies(ThisProject), inConfigurations(Compile) ) @@ -150,14 +150,15 @@ object SbtAvro extends AutoPlugin { inStyle = FilesInfo.lastModified, outStyle = FilesInfo.exists ) { deps => + val filter = includeFilter -- excludeFilter + // dedicated directory per artifact to avoid name conflicts val depTarget = extractTarget / jar.base IO.createDirectory(depTarget) deps.flatMap { dep => - val filter = includeFilter -- excludeFilter val (avroSpecs, filtered) = IO .unzip(dep, depTarget, AvroFilter) - .partition(filter.accept) + .partition(_.relativeTo(depTarget).forall(filter.accept)) IO.delete(filtered) if (avroSpecs.nonEmpty) { streams.log.info("Extracted from " + dep + avroSpecs.mkString(":\n * ", "\n * ", "")) @@ -183,10 +184,12 @@ object SbtAvro extends AutoPlugin { crossPaths.value ) + // Classpaths.managedJars does not cross-build val avroArtifacts = update.value .filter(avroDependencyIncludeFilter.value) .toSeq .map { case (_, _, _, f) => f } + .distinct val unpacked = unpack( cacheBaseDirectory = cacheBaseDirectory, @@ -243,7 +246,7 @@ object SbtAvro extends AutoPlugin { // TODO Cache class loader val avroClassLoader = new URLClassLoader( - (Avro / dependencyClasspath).value + (AvroCompiler / dependencyClasspath).value .map(toNioPath) .map(_.toUri.toURL) .toArray, diff --git a/plugin/src/sbt-test/sbt-avro/avscparser/build.sbt b/plugin/src/sbt-test/sbt-avro/avscparser/build.sbt index c9129df..bb43bf9 100644 --- a/plugin/src/sbt-test/sbt-avro/avscparser/build.sbt +++ b/plugin/src/sbt-test/sbt-avro/avscparser/build.sbt @@ -1,5 +1,3 @@ - - lazy val parser = project .in(file("parser")) .settings( @@ -14,7 +12,7 @@ lazy val parser = project lazy val root = project .in(file(".")) .enablePlugins(SbtAvro) - .dependsOn(parser % "avro") + .dependsOn(parser % "avro-compiler") .settings( avroCompiler := "com.github.sbt.avro.CustomAvroCompiler" - ) \ No newline at end of file + ) diff --git a/plugin/src/sbt-test/sbt-avro/local-dependency/build.sbt b/plugin/src/sbt-test/sbt-avro/local-dependency/build.sbt index d05e3c5..75b5a7d 100644 --- a/plugin/src/sbt-test/sbt-avro/local-dependency/build.sbt +++ b/plugin/src/sbt-test/sbt-avro/local-dependency/build.sbt @@ -39,19 +39,14 @@ lazy val `transitive`: Project = project name := "transitive", version := "0.0.1-SNAPSHOT", libraryDependencies ++= Seq( - // when using avro scope, it won't be part of the pom dependencies -> intransitive - // to declare transitive dependency use the compile scope - "com.github.sbt" % "external" % "0.0.1-SNAPSHOT" classifier "avro" - ), - Compile / avroDependencyIncludeFilter := artifactFilter(classifier = "avro"), - // create a test jar with a schema as resource - Test / packageBin / publishArtifact := true, + ("com.github.sbt" % "external" % "0.0.1-SNAPSHOT" % "avro").classifier("avro").intransitive() + ) ) lazy val root: Project = project .in(file(".")) .enablePlugins(SbtAvro) - .dependsOn(`transitive` % "avro->avro") + .dependsOn(`transitive` % "avro") .settings(commonSettings) .settings( name := "local-dependency", @@ -59,9 +54,6 @@ lazy val root: Project = project libraryDependencies ++= Seq( "org.specs2" %% "specs2-core" % "4.20.9" % Test ), - // add additional avro source test jar - Test / avroDependencyIncludeFilter := artifactFilter(name = "transitive", classifier = "tests"), - Compile / checkUnpacked := { exists((`transitive` / crossTarget).value / "src_managed" / "avro" / "main" / "external-avro" / "avdl.avdl") exists((`transitive` / crossTarget).value / "src_managed" / "avro" / "main" / "external-avro" / "avpr.avpr") diff --git a/plugin/src/sbt-test/sbt-avro/local-dependency/src/test/scala/com/github/sbt/avro/test/AvroTest.scala b/plugin/src/sbt-test/sbt-avro/local-dependency/src/test/scala/com/github/sbt/avro/test/AvroTest.scala deleted file mode 100644 index c22c138..0000000 --- a/plugin/src/sbt-test/sbt-avro/local-dependency/src/test/scala/com/github/sbt/avro/test/AvroTest.scala +++ /dev/null @@ -1,10 +0,0 @@ -package com.github.sbt.avro.test - -import com.github.sbt.avro.test.transitive.Test - -object AvroTest extends App { - - Test.newBuilder().setStringField("external").build() - - println("success") -} \ No newline at end of file diff --git a/plugin/src/sbt-test/sbt-avro/local-dependency/transitive/src/main/avro/com/github/sbt/avro/test/transitive/avsc.avsc b/plugin/src/sbt-test/sbt-avro/local-dependency/transitive/src/main/avro/avsc.avsc similarity index 100% rename from plugin/src/sbt-test/sbt-avro/local-dependency/transitive/src/main/avro/com/github/sbt/avro/test/transitive/avsc.avsc rename to plugin/src/sbt-test/sbt-avro/local-dependency/transitive/src/main/avro/avsc.avsc diff --git a/plugin/src/sbt-test/sbt-avro/local-dependency/transitive/src/test/resources/test.avsc b/plugin/src/sbt-test/sbt-avro/local-dependency/transitive/src/test/resources/test.avsc deleted file mode 100644 index 1a87638..0000000 --- a/plugin/src/sbt-test/sbt-avro/local-dependency/transitive/src/test/resources/test.avsc +++ /dev/null @@ -1,15 +0,0 @@ -{ - "name": "Test", - "namespace": "com.github.sbt.avro.test.transitive", - "type": "record", - "fields": [ - { - "name": "stringField", - "type": "string" - }, - { - "name": "referencedTypeField", - "type": "com.github.sbt.avro.test.external.Avsc" - } - ] -} diff --git a/plugin/src/sbt-test/sbt-avro/publishing/build.sbt b/plugin/src/sbt-test/sbt-avro/publishing/build.sbt index f5a788f..33dd75f 100644 --- a/plugin/src/sbt-test/sbt-avro/publishing/build.sbt +++ b/plugin/src/sbt-test/sbt-avro/publishing/build.sbt @@ -56,15 +56,14 @@ lazy val root: Project = project name := "publishing-test", crossScalaVersions := Seq("2.13.15", "2.12.20"), libraryDependencies ++= Seq( - "com.github.sbt" % "transitive" % "0.0.1-SNAPSHOT" % "avro" classifier "avro", // external as transitive - "com.github.sbt" % "transitive" % "0.0.1-SNAPSHOT" % "test" classifier "tests", + ("com.github.sbt" % "transitive" % "0.0.1-SNAPSHOT" % "avro").classifier("avro"), // external as transitive + ("com.github.sbt" % "transitive" % "0.0.1-SNAPSHOT" % "avro-test").classifier("tests").intransitive(), "org.specs2" %% "specs2-core" % "4.20.9" % Test ), - // add additional avro source test jar + // add additional avro source test jar whithout avro classifier Test / avroDependencyIncludeFilter := artifactFilter(name = "transitive", classifier = "tests"), // exclude specific avsc file - Compile / avroUnpackDependencies / excludeFilter := (Compile / avroUnpackDependencies / excludeFilter).value || "exclude.avsc", - + Compile / avroUnpackDependencies / excludeFilter ~= { filter => filter || "exclude.avsc" }, Compile / checkUnpacked := { exists(crossTarget.value / "src_managed" / "avro" / "main" / "external-avro" / "avdl.avdl") exists(crossTarget.value / "src_managed" / "avro" / "main" / "external-avro" / "avpr.avpr") diff --git a/plugin/src/sbt-test/sbt-avro/recompile/build.sbt b/plugin/src/sbt-test/sbt-avro/recompile/build.sbt index b65d4a5..18f2afc 100644 --- a/plugin/src/sbt-test/sbt-avro/recompile/build.sbt +++ b/plugin/src/sbt-test/sbt-avro/recompile/build.sbt @@ -4,7 +4,7 @@ name := "recompile-test" scalaVersion := "2.13.11" libraryDependencies ++= Seq( // depend on test jar to get some generated records in the build - "org.apache.avro" % "avro" % avroVersion.value % "avro" classifier "tests", + ("org.apache.avro" % "avro" % avroVersion.value % "avro-compiler").classifier("tests"), "org.specs2" %% "specs2-core" % "4.20.9" % Test ) diff --git a/plugin/src/sbt-test/sbt-avro/settings/build.sbt b/plugin/src/sbt-test/sbt-avro/settings/build.sbt index bf23d25..8b77ab8 100644 --- a/plugin/src/sbt-test/sbt-avro/settings/build.sbt +++ b/plugin/src/sbt-test/sbt-avro/settings/build.sbt @@ -4,7 +4,7 @@ name := "settings-test" scalaVersion := "2.13.11" libraryDependencies ++= Seq( // depend on test jar to get some generated records in the build - "org.apache.avro" % "avro" % avroVersion.value % "avro" classifier "tests", + ("org.apache.avro" % "avro" % avroVersion.value % "avro-compiler").classifier("tests"), "org.specs2" %% "specs2-core" % "4.20.9" % Test )