Databricks Connector Fails #6197

P2W2 · 2023-02-23T15:52:10Z

Describe the bug
Hello,
I am in the process of trying out cube.js. I wanted to use the Databricks JDBC connector with Docker.
I tried to follow this Documentation.
The Container Starts up but if I try to open the schema (http://localhost:4000/#/schema) I got following failure:

Error: ENOENT: no such file or directory, open 'package.json'
    at DatabricksDriver.testConnection (/cube/node_modules/@cubejs-backend/jdbc-driver/src/JDBCDriver.ts:188:13)
    at CubejsServerCore.getDriver (/cube/node_modules/@cubejs-backend/server-core/src/core/server.ts:782:7)
    at /cube/node_modules/@cubejs-backend/server-core/src/core/DevServer.ts:108:22
    at /cube/node_modules/@cubejs-backend/server-core/src/core/DevServer.ts:78:9

To Reproduce

Follow the Documentation to build the Docker Image
docker run -p 4000:4000 -p 3000:3000 --env-file=.env -v ${PWD}:/cube/conf -e CUBEJS_DEV_MODE=true cubejs-jdk:latest

Warning. There is no cube.js file. Continue with environment variables
🔥 Cube Store (0.31.64) is assigned to 3030 port.
Warning. Option apiSecret is required in dev mode. Cube.js has generated it as 5bc38414df47015c0432991a0b8f2cbe
🔓 Authentication checks are disabled in developer mode. Please use NODE_ENV=production to enable it.
🦅 Dev environment available at http://localhost:4000
🚀 Cube API server (0.31.64) is listening on 4000
2023-02-23 11:26:41,803 INFO  [cubestored] <pid:29> Cube Store version 0.31.64
2023-02-23 11:26:41,890 INFO  [cubestore::http::status] <pid:29> Serving status probes at 0.0.0.0:3031
2023-02-23 11:26:41,902 INFO  [cubestore::metastore::rocks_fs] <pid:29> Using existing metastore in /cube/conf/.cubestore/data/metastore
2023-02-23 11:26:42,315 INFO  [cubestore::mysql] <pid:29> MySQL port open on 0.0.0.0:13306
2023-02-23 11:26:42,321 INFO  [cubestore::http] <pid:29> Http Server is listening on 0.0.0.0:3030
You accepted Terms & Conditions for JDBC driver from DataBricks by CUBEJS_DB_DATABRICKS_ACCEPT_POLICY
Downloading DatabricksJDBC42-2.6.29.1051
Release notes: https://databricks-bi-artifacts.s3.us-east-2.amazonaws.com/simbaspark-drivers/jdbc/2.6.29/docs/release-notes.txt

Open the Playground schema page on localhost:4000

.env file

CUBEJS_DB_TYPE=databricks-jdbc
# You can find this inside the cluster's configuration
CUBEJS_DB_DATABRICKS_URL=jdbc:spark://***.databricks.com:443/default;transportMode=http;ssl=1;httpPath=sql/protocolv1/o/*******/****-******-******;AuthMech=3;UID=token
# You can specify the personal access token separately from `CUBEJS_DB_DATABRICKS_URL` by doing this:
CUBEJS_DB_DATABRICKS_TOKEN=************************
# This accepts the Databricks usage policy and must be set to `true` to use the Databricks JDBC driver
CUBEJS_DB_DATABRICKS_ACCEPT_POLICY=true

Additional context
I use the dev mode otherwise I get following error message:

Error: apiSecret is required option(s)
    at OptsHandler.initializeCoreOptions (/cube/node_modules/@cubejs-backend/server-core/src/core/OptsHandler.ts:533:15)
    at new OptsHandler (/cube/node_modules/@cubejs-backend/server-core/src/core/OptsHandler.ts:49:36)
    at new CubejsServerCore (/cube/node_modules/@cubejs-backend/server-core/src/core/server.ts:170:24)
    at CubejsServer.createCoreInstance (/cube/node_modules/@cubejs-backend/server/src/server.ts:84:12)
    at new CubejsServer (/cube/node_modules/@cubejs-backend/server/src/server.ts:79:22)
    at ServerContainer.createServer (/cube/node_modules/@cubejs-backend/server/src/server/container.ts:254:12)
    at ServerContainer.runServerInstance (/cube/node_modules/@cubejs-backend/server/src/server/container.ts:235:25)
    at makeInstance (/cube/node_modules/@cubejs-backend/server/src/server/container.ts:347:33)
    at ServerContainer.start (/cube/node_modules/@cubejs-backend/server/src/server/container.ts:360:20)
    at Server.run (/cube/node_modules/@cubejs-backend/server/src/command/server.ts:25:5)
    at Command.<anonymous> (/cube/node_modules/cubejs-cli/src/command/proxy-command.ts:51:13)

Need some help? -------------------------------------

Thanks for the help.

The text was updated successfully, but these errors were encountered:

jelstongreen · 2023-03-21T14:17:01Z

+1

caiobelfort · 2023-04-18T22:10:58Z

Same problem here!

carlagouveia · 2023-05-01T19:39:08Z

Having the same issue. Could this be a problem with the base image?

cube_example  | You accepted Terms & Conditions for JDBC driver from DataBricks by CUBEJS_DB_DATABRICKS_ACCEPT_POLICY
cube_example  | Downloading DatabricksJDBC42-2.6.29.1051
cube_example  | Release notes: https://databricks-bi-artifacts.s3.us-east-2.amazonaws.com/simbaspark-drivers/jdbc/2.6.29/docs/release-notes.txt
cube_example  | Error: ENOENT: no such file or directory, open 'package.json'
cube_example  |     at DatabricksDriver.testConnection (/cube/node_modules/@cubejs-backend/jdbc-driver/src/JDBCDriver.ts:205:13)
cube_example  |     at CubejsServerCore.getDriver (/cube/node_modules/@cubejs-backend/server-core/src/core/server.ts:792:7)
cube_example  |     at /cube/node_modules/@cubejs-backend/server-core/src/core/DevServer.ts:108:22
cube_example  |     at /cube/node_modules/@cubejs-backend/server-core/src/core/DevServer.ts:78:9

@paveltiunov any other place where we should file this as a regression?

carlagouveia · 2023-05-01T20:11:21Z

The issue seems to be related to this line: https://github.com/cube-js/cube/blob/40195c97d5b0d2caf89b04ed4f7767a1ff767f01/packages/cubejs-databricks-jdbc-driver/src/DatabricksDriver.ts#LL30C14-L30C14

Which is used here:

cube/packages/cubejs-databricks-jdbc-driver/src/DatabricksDriver.ts

Line 213 in 40195c9

UserAgentEntry: `CubeDev+Cube/${version} (Databricks)`,

I made a PR to fix this issue. In my case there was an exception while connecting to Databricks, but it was hidden with the error related to not finding package.json. This is my actual error, after my fix is applied:

cube_example  | Error: Error running static method
cube_example  | java.sql.SQLException: [Databricks][JDSI](20280) Required setting UID is not present in connection settings.
cube_example  |  at com.databricks.client.dsi.core.impl.DSIConnection.getRequiredSetting(Unknown Source)
cube_example  |  at com.databricks.client.hivecommon.core.HiveJDBCCommonConnection.establishConnection(Unknown Source)
cube_example  |  at com.databricks.client.spark.core.SparkJDBCConnection.establishConnection(Unknown Source)
cube_example  |  at com.databricks.client.jdbc.core.LoginTimeoutConnection.connect(Unknown Source)
cube_example  |  at com.databricks.client.jdbc.common.BaseConnectionFactory.doConnect(Unknown Source)
cube_example  |  at com.databricks.client.jdbc.common.AbstractDriver.connect(Unknown Source)
cube_example  |  at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:677)
cube_example  |  at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:189)
cube_example  |  at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
cube_example  |  at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
cube_example  |  at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
cube_example  |  at java.base/java.lang.reflect.Method.invoke(Method.java:566)
cube_example  | Caused by: com.databricks.client.dsi.exceptions.BadAuthException: [Databricks][JDSI](20280) Required setting UID is not present in connection settings.

PR: #6521

jjunior84 · 2023-05-05T17:15:37Z

I had the same problem, the tutorial says to copy the JDBC URL from you Databricks SQL Warehouse to use as connection, but the connection string is missing ;UID=token.

When I added it in the end of the connection, it worked fine!

I also was tested with Unity Catalog and I had to specify the catalog name from my database to work too.
CUBEJS_DB_DATABRICKS_CATALOG=<catalog_name>

paveltiunov · 2023-05-16T22:43:52Z

Fixed by #6521

asanchezdelc · 2023-05-24T17:23:20Z

I'm still getting the same error after pulling the latest image.

docker run -p 4000:4000 -p 3000:3000 --env-file=.env -v ${PWD}:/cube/conf -e CUBEJS_DEV_MODE=true cubejs-jdk:latest
Warning. There is no cube.js file. Continue with environment variables
🔥 Cube Store (0.33.12) is assigned to 3030 port.
Warning. Option apiSecret is required in dev mode. Cube has generated it as 93c98045d89c258a6d77d5210824904e
🔓 Authentication checks are disabled in developer mode. Please use NODE_ENV=production to enable it.
🦅 Dev environment available at http://localhost:4000
🔗 Cube SQL (pg) is listening on 0.0.0.0:15432
🚀 Cube API server (0.33.12) is listening on 4000
2023-05-24 17:09:04,326 INFO  [cubestored] <pid:19> Cube Store version 0.33.11
2023-05-24 17:09:04,330 INFO  [cubestore::http::status] <pid:19> Serving status probes at 0.0.0.0:3031
2023-05-24 17:09:04,332 INFO  [cubestore::metastore::rocks_fs] <pid:19> Using existing metastore in /cube/conf/.cubestore/data/metastore
2023-05-24 17:09:04,428 INFO  [cubestore::mysql] <pid:19> MySQL port open on 0.0.0.0:13306
2023-05-24 17:09:04,428 INFO  [cubestore::http] <pid:19> Http Server is listening on 0.0.0.0:3030
You accepted Terms & Conditions for JDBC driver from DataBricks by CUBEJS_DB_DATABRICKS_ACCEPT_POLICY
Downloading DatabricksJDBC42-2.6.29.1051
Release notes: https://databricks-bi-artifacts.s3.us-east-2.amazonaws.com/simbaspark-drivers/jdbc/2.6.29/docs/release-notes.txt
Error: ENOENT: no such file or directory, open 'package.json'
    at DatabricksDriver.testConnection (/cube/node_modules/@cubejs-backend/jdbc-driver/src/JDBCDriver.ts:205:13)
    at CubejsServerCore.getDriver (/cube/node_modules/@cubejs-backend/server-core/src/core/server.ts:792:7)
    at /cube/node_modules/@cubejs-backend/server-core/src/core/DevServer.ts:108:22
    at /cube/node_modules/@cubejs-backend/server-core/src/core/DevServer.ts:78:9

carlagouveia · 2023-05-24T18:45:45Z

@asanchezdelc Are you using the image cubejs/cube:jdk?

caiobelfort · 2023-05-24T19:08:44Z

Hi, tried again and still getting the same error. Using last image of tag jdk uploaded 2 days ago. Em qua., 24 de mai. de 2023 15:45, Carla Gouveia ***@***.***> escreveu:

…

@asanchezdelc <https://github.com/asanchezdelc> Are you using the image cubejs/cube:jdk? — Reply to this email directly, view it on GitHub <#6197 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABHSS7QZDRCB3LYGPUJPD73XHZJOHANCNFSM6AAAAAAVFZ33F4> . You are receiving this because you commented.Message ID: ***@***.***>

asanchezdelc · 2023-05-24T19:34:18Z

@asanchezdelc Are you using the image cubejs/cube:jdk?

Yea, I'm using the latest tag. I also checked this #6651 code is in the container when running it.

caiobelfort · 2023-05-24T21:06:49Z

Ok, I managed to make it work. Apparently when we create docker compose and we don't provide the 'package.json' file in the root folder it is not possible to make the connection. I suggest a revision on documentation about this process.

My package.json is based on cube cloud development environment

{
  "name": "cubejs-project-template",
  "version": "0.0.1",
  "private": true,
  "scripts": {
    "dev": "cubejs-server"
  },
  "template": "docker",
  "templateVersion":   "0.33.11"    --> cube cloud set this as 0.28.19
}

asanchezdelc · 2023-05-24T22:43:12Z

@caiobelfort I didn't have a package.json file, but adding this file worked for me as well.

jacolei · 2023-08-24T01:17:21Z

For me as well, adding ;UID=token at the end of the connection string solved the problem

P2W2 changed the title ~~Databricks Connector fails~~ Databricks Connector Fails Feb 23, 2023

carlagouveia mentioned this issue May 1, 2023

Remove package.json dependency from DatabricksDriver.ts carlagouveia/cube#1

Closed

carlagouveia mentioned this issue May 1, 2023

Remove package.json dependency from DatabricksDriver.ts #6521

Merged

paveltiunov closed this as completed May 16, 2023

MikeNitsenko mentioned this issue Sep 25, 2023

fix(databricks-jdbc-driver): ensure default UID property is set #7168

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Databricks Connector Fails #6197

Databricks Connector Fails #6197

P2W2 commented Feb 23, 2023 •

edited

Loading

jelstongreen commented Mar 21, 2023

caiobelfort commented Apr 18, 2023

carlagouveia commented May 1, 2023

carlagouveia commented May 1, 2023 •

edited

Loading

jjunior84 commented May 5, 2023

paveltiunov commented May 16, 2023

asanchezdelc commented May 24, 2023

carlagouveia commented May 24, 2023

caiobelfort commented May 24, 2023 via email

asanchezdelc commented May 24, 2023

caiobelfort commented May 24, 2023 •

edited

Loading

asanchezdelc commented May 24, 2023

jacolei commented Aug 24, 2023

Databricks Connector Fails #6197

Databricks Connector Fails #6197

Comments

P2W2 commented Feb 23, 2023 • edited Loading

jelstongreen commented Mar 21, 2023

caiobelfort commented Apr 18, 2023

carlagouveia commented May 1, 2023

carlagouveia commented May 1, 2023 • edited Loading

jjunior84 commented May 5, 2023

paveltiunov commented May 16, 2023

asanchezdelc commented May 24, 2023

carlagouveia commented May 24, 2023

caiobelfort commented May 24, 2023 via email

asanchezdelc commented May 24, 2023

caiobelfort commented May 24, 2023 • edited Loading

asanchezdelc commented May 24, 2023

jacolei commented Aug 24, 2023

P2W2 commented Feb 23, 2023 •

edited

Loading

carlagouveia commented May 1, 2023 •

edited

Loading

caiobelfort commented May 24, 2023 •

edited

Loading