Scala 2.13 Support

[tculshaw]

Hi,

Apache 3.4.0 onwards can also compile against Scala 2.13.

Is there any plan to provide a cross compiled version of spline for this?

[wajda]

Related to #705

[wajda]

The agent-core is compiled against Spark 2.4 that does not support Scala 2.13.
To add Scala 2.13 cross-compilation option we need to decouple the agent core from a particular Spark version and make the dependency dynamic. It requires more work, so I'm postponing the ticket for now.

[wajda]

There are more issues with 2.13. A lot of 3rd dependencies and 3rd party connectors some bundled plugins are using only support 2.13 in their very recent versions. At the same time support for 2.12 and 2.11 are dropped leaving the dependency matrix for Spline in a hell complex state.
I believe, to reliably proceed with Scala 2.13 we need to abandon support for 2.11 and 2.12 which also means to abandon support for Spark 3.1 or less.
In my opinion, considering natural enterprise inertia it's too early for that kind of decision. Spark 2.4 is still widely used among Spline customers. And also considering the fact that Spark still supports 2.12 in all its recent versions I would wait for some more time before upgrading to 2.13.