Skip to content

Commit

Permalink
Documenting assigned cluster SQL registration working on DBR 13.3.
Browse files Browse the repository at this point in the history
  • Loading branch information
mjohns-databricks committed Jan 25, 2024
1 parent 7950146 commit fa65964
Show file tree
Hide file tree
Showing 5 changed files with 28 additions and 18 deletions.
6 changes: 3 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,9 +57,9 @@ __Language Bindings__

As of Mosaic 0.4.0 (subject to change in follow-on releases)...

* _Mosaic SQL expressions cannot yet be registered with [Unity Catalog](https://www.databricks.com/product/unity-catalog) due to API changes affecting DBRs >= 13._
* [Assigned Clusters](https://docs.databricks.com/en/compute/configure.html#access-modes): Mosaic Python, R, and Scala APIs.
* [Assigned Clusters](https://docs.databricks.com/en/compute/configure.html#access-modes): Mosaic Python, SQL, R, and Scala APIs.
* [Shared Access Clusters](https://docs.databricks.com/en/compute/configure.html#access-modes): Mosaic Scala API (JVM) with Admin [allowlisting](https://docs.databricks.com/en/data-governance/unity-catalog/manage-privileges/allowlist.html); _Python bindings to Mosaic Scala APIs are blocked by Py4J Security on Shared Access Clusters._
* _Mosaic SQL expressions cannot yet be registered with [Unity Catalog](https://www.databricks.com/product/unity-catalog) due to API changes affecting DBRs >= 13, more [here](https://docs.databricks.com/en/udf/index.html)._

__Additional Notes:__

Expand Down Expand Up @@ -141,7 +141,7 @@ import com.databricks.labs.mosaic.JTS
val mosaicContext = MosaicContext.build(H3, JTS)
mosaicContext.register(spark)
```
__Note: Mosaic 0.4.x SQL bindings for DBR 13 not yet available in Unity Catalog due to API changes.__
__Note: Mosaic 0.4.x SQL bindings for DBR 13 can register with Assigned clusters, but not Shared Access due to API changes, more [here](https://docs.databricks.com/en/udf/index.html).__

## Examples

Expand Down
7 changes: 5 additions & 2 deletions docs/source/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@ We currently recommend using Databricks Runtime with Photon enabled;
this will leverage the Databricks H3 expressions when using H3 grid system.

Mosaic provides:

* easy conversion between common spatial data encodings (WKT, WKB and GeoJSON);
* constructors to easily generate new geometries from Spark native data types;
* many of the OGC SQL standard :code:`ST_` functions implemented as Spark Expressions for transforming, aggregating and joining spatial datasets;
Expand All @@ -71,9 +72,11 @@ Mosaic 0.4.x series issues the following ERROR on a standard, non-Photon cluster
**DEPRECATION ERROR: Please use a Databricks Photon-enabled Runtime for performance benefits or Runtime ML for spatial AI benefits; Mosaic 0.4.x series restricts executing this cluster.**

As of Mosaic 0.4.0 (subject to change in follow-on releases)
* No Mosaic SQL expressions cannot yet be registered with `Unity Catalog <https://www.databricks.com/product/unity-catalog>`_ due to API changes affecting DBRs >= 13.
* `Assigned Clusters <https://docs.databricks.com/en/compute/configure.html#access-modes>`_ : Mosaic Python, R, and Scala APIs.

* `Assigned Clusters <https://docs.databricks.com/en/compute/configure.html#access-modes>`_ : Mosaic Python, SQL, R, and Scala APIs.
* `Shared Access Clusters <https://docs.databricks.com/en/compute/configure.html#access-modes>`_ : Mosaic Scala API (JVM) with Admin `allowlisting <https://docs.databricks.com/en/data-governance/unity-catalog/manage-privileges/allowlist.html>`_ ; Python bindings to Mosaic Scala APIs are blocked by Py4J Security on Shared Access Clusters.
- Mosaic SQL expressions cannot yet be registered with `Unity Catalog <https://www.databricks.com/product/unity-catalog>`_
due to API changes affecting DBRs >= 13, more `here <https://docs.databricks.com/en/udf/index.html>`_.

.. note::
As of Mosaic 0.4.0 (subject to change in follow-on releases)
Expand Down
10 changes: 7 additions & 3 deletions docs/source/usage/automatic-sql-registration.rst
Original file line number Diff line number Diff line change
Expand Up @@ -19,13 +19,13 @@ Pre-requisites
In order to use Mosaic, you must have access to a Databricks cluster running
Databricks Runtime 13. If you have cluster creation permissions in your Databricks
workspace, you can create a cluster using the instructions
`here <https://docs.databricks.com/clusters/create.html#use-the-cluster-ui>`__.
`here <https://docs.databricks.com/clusters/create.html#use-the-cluster-ui>`_.

You will also need "Can Manage" permissions on this cluster in order to attach init script
to your cluster. A workspace administrator will be able to grant
these permissions and more information about cluster permissions can be found
in our documentation
`here <https://docs.databricks.com/security/access-control/cluster-acl.html#cluster-level-permissions>`__.
`here <https://docs.databricks.com/security/access-control/cluster-acl.html#cluster-level-permissions>`_.

Installation
************
Expand Down Expand Up @@ -82,7 +82,11 @@ To test the installation, create a new Python notebook and run the following com
You should see all the supported functions registered by Mosaic appear in the output.
.. note::
You may see some :code:`ST_` functions from other libraries, so pay close attention to the provider.
You may see some :code:`ST_` functions from other libraries, so pay close attention to the provider.
.. warning::
Mosaic 0.4.x SQL bindings for DBR 13 can register with Assigned clusters, but not Shared Access due to API changes,
more `here <https://docs.databricks.com/en/udf/index.html>`_.
.. warning::
Issue 317: https://github.com/databrickslabs/mosaic/issues/317
Expand Down
14 changes: 7 additions & 7 deletions docs/source/usage/install-gdal.rst
Original file line number Diff line number Diff line change
Expand Up @@ -8,17 +8,17 @@ In order to use Mosaic 0.4 series, you must have access to a Databricks cluster
Databricks Runtime 13.3 LTS.
If you have cluster creation permissions in your Databricks
workspace, you can create a cluster using the instructions
`here <https://docs.databricks.com/clusters/create.html#use-the-cluster-ui>`__.
`here <https://docs.databricks.com/clusters/create.html#use-the-cluster-ui>`_.

You will also need "Can Manage" permissions on this cluster in order to attach the
Mosaic library to your cluster. A workspace administrator will be able to grant
these permissions and more information about cluster permissions can be found
in our documentation
`here <https://docs.databricks.com/security/access-control/cluster-acl.html#cluster-level-permissions>`__.
`here <https://docs.databricks.com/security/access-control/cluster-acl.html#cluster-level-permissions>`_.

.. warning::
These instructions assume an Assigned cluster is being used (vs a Shared Access cluster),
more on access modes `here <https://docs.databricks.com/en/compute/configure.html#access-modes>`__.
more on access modes `here <https://docs.databricks.com/en/compute/configure.html#access-modes>`_.

GDAL Installation
####################
Expand All @@ -30,10 +30,10 @@ the :code:`setup_gdal` function.

.. note::
- This is close in behavior to Mosaic < 0.4 series (prior to DBR 13), with new options
to pip install Mosaic for either ubuntugis gdal (3.4.3) or jammy default (3.4.1).
to pip install Mosaic for either ubuntugis gdal (3.4.3) or jammy default (3.4.1).
- Param "to_fuse_dir" can be one of "/Volumes/..", "/Workspace/..", "/dbfs/..";
however, you should consider :code:`setup_fuse_install()` for Volume based installs as that
exposes more options, to include copying JAR and JNI Shared Objects.
however, you should consider :code:`setup_fuse_install()` for Volume based installs as that
exposes more options, to include copying JAR and JNI Shared Objects.

.. function:: setup_gdal()

Expand Down Expand Up @@ -107,5 +107,5 @@ code at the top of the notebook:
GDAL 3.4.1, released 2021/12/27
.. note::
You can configure init script from default ubuntu GDAL (3.4.1) to `ubuntugis ppa <https://launchpad.net/~ubuntugis/+archive/ubuntu/ppa>`__ (3.4.3)
You can configure init script from default ubuntu GDAL (3.4.1) to `ubuntugis ppa <https://launchpad.net/~ubuntugis/+archive/ubuntu/ppa>`_ (3.4.3)
with :code:`setup_gdal(with_ubuntugis=True)`
9 changes: 6 additions & 3 deletions docs/source/usage/installation.rst
Original file line number Diff line number Diff line change
Expand Up @@ -17,9 +17,11 @@ Mosaic 0.4.x series issues the following ERROR on a standard, non-Photon cluster
**DEPRECATION ERROR: Please use a Databricks Photon-enabled Runtime for performance benefits or Runtime ML for spatial AI benefits; Mosaic 0.4.x series restricts executing this cluster.**

As of Mosaic 0.4.0 (subject to change in follow-on releases)
* No Mosaic SQL expressions cannot yet be registered with `Unity Catalog <https://www.databricks.com/product/unity-catalog>`_ due to API changes affecting DBRs >= 13.
* `Assigned Clusters <https://docs.databricks.com/en/compute/configure.html#access-modes>`_ : Mosaic Python, R, and Scala APIs.

* `Assigned Clusters <https://docs.databricks.com/en/compute/configure.html#access-modes>`_ : Mosaic Python, SQL, R, and Scala APIs.
* `Shared Access Clusters <https://docs.databricks.com/en/compute/configure.html#access-modes>`_ : Mosaic Scala API (JVM) with Admin `allowlisting <https://docs.databricks.com/en/data-governance/unity-catalog/manage-privileges/allowlist.html>`_ ; Python bindings to Mosaic Scala APIs are blocked by Py4J Security on Shared Access Clusters.
- Mosaic SQL expressions cannot yet be registered with `Unity Catalog <https://www.databricks.com/product/unity-catalog>`_
due to API changes affecting DBRs >= 13, more `here <https://docs.databricks.com/en/udf/index.html>`_.

.. note::
As of Mosaic 0.4.0 (subject to change in follow-on releases)
Expand Down Expand Up @@ -119,4 +121,5 @@ register the Mosaic SQL functions in your SparkSession from a Scala notebook cel
mosaicContext.register(spark)
.. warning::
Mosaic 0.4.x SQL bindings for DBR 13 not yet available in Unity Catalog due to API changes.
Mosaic 0.4.x SQL bindings for DBR 13 can register with Assigned clusters, but not Shared Access due to API changes,
more `here <https://docs.databricks.com/en/udf/index.html>`_.

0 comments on commit fa65964

Please sign in to comment.