Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Gdal jammy 3 #495

Merged
merged 107 commits into from
Jan 9, 2024
Merged

Gdal jammy 3 #495

merged 107 commits into from
Jan 9, 2024

Conversation

mjohns-databricks
Copy link
Contributor

[WIP] DBR 13.3

  1. Throw exception for unsupported (standard) cluster
  2. enable_mosaic
    • for assigned cluster - python (+scala) good
    • for shared access cluster - scala initializing after admin allowlisting on UC + Volume
    • for shared access cluster - python hitting a number of allowlist issues due to Py4JSecurity
  3. GDAL install (via init script)
    • for assigned cluster - good
    • shared access cluster - not yet finalized
  4. Script path will be changed after this PR accepted - currently is pointing to gdal-jammy-3 branch in fork

@milos-colic wanted to make sure you have a look at this build as we implement initial Shared Access support for DBR 13.3.

@mjohns-databricks
Adjusted various URLs now that PR databrickslabs#492 merged. Expanded…
40c2106 
… testing for the

variations supported by `setup_fuse_install(...)` function.
@mjohns-databricks
Adjusted from https://raw.githubuserconent.com to
c0a1df1 
https://github.com/../raw url pattern to avoid needing an additional
github username header token.
@mjohns-databricks
Fixed broken import for testing.
6c34ac9 
Added error vs warning for standard DBR.
@mjohns-databricks
- updated scala checks since *.clusterType no longer a tag.
309e68e 
- `*.sparkVersion` works for (1) DBR major, (2) isML and (3) isPhoton.
- verified on all flavors of DBR 13.3 and 12.2
@mjohns-databricks
Missed a change in test harness.
ce5414b
@mjohns-databricks
Adjustments to spark testing vs DBR.
5d7c2b5 
Resetting spark conf per test.
@mjohns-databricks
missed a '.'
293caba
@mjohns-databricks
Streamlining isTest check further.
c2cc8af
@mjohns-databricks
spark conf standardization for testing.
010f114
@mjohns-databricks
spark.conf.set(MOSAIC_TEST, "true") where needed.
2fdfe5e
@mjohns-databricks
added additional system property since spark
dfdfdff 
not always used in tests involving MosaicContext.
@mjohns-databricks
Simplified isTest logic in MosaicContext.
63a0aa6
@mjohns-databricks
streamline isTest logic.
e5e4176
@mjohns-databricks
cleanup scala tests from exploratory isTest logic.
304b688
@mjohns-databricks
Python setup_ test related.
b2c31bd
@mjohns-databricks
adjust do_op.
15ff5b9
@mjohns-databricks
Modify to use branch script temporarily.
ccaeda4
@mjohns-databricks
Adding sudo to cp in script.
851384e
@mjohns-databricks
Adjusting Fuse cp for so files to be no-clobber.
6ec91ab
@mjohns-databricks
- override to main for tests.
a82ec46 
- use sudo to download resources (for tests).
@mjohns-databricks
- Adjusted test to avoid repeat installs.
f1a1ab9 
- better mosaic + github version handling.
@mjohns-databricks
- reduce logging for tests.
d6e5fd8 
- wait for downloads.
@mjohns-databricks
wait for downloads for tests.
3658257
@mjohns-databricks
download back to requests.
c793c38
@mjohns-databricks
don't cleanup temp dir until finished.
805a08f
@mjohns-databricks
logging silencing.
8c99872
@mjohns-databricks
controlling tempdir for fuse install tests.
d217676
@mjohns-databricks
temp dir obj now a string.
6c9c0d4
@mjohns-databricks
removing sc conflict.
76bcf1a
@mjohns-databricks
quiet log4j2-defaults.properties
298dce5
@mjohns-databricks
parentPath defaults to empty string vs null.
0ddc056
@mjohns-databricks
adjust string literal handling.
a7ea6db
@mjohns-databricks
adjusting quotes in testing.
1d8f786
@mjohns-databricks
Stripping back tests for string literal.
8d25fd7
@mjohns-databricks
add back R tests and string literal.
383aef0
@mjohns-databricks
literal string vs null.
9f27423
@mjohns-databricks
lit(null).cast(StringType) testing.
bc97d1a
@mjohns-databricks
remove parentPath arg from rst_fromcontent.
fa369e7
@mjohns-databricks
add path null check for cleanup
98d9060
@mjohns-databricks
safer cleanup for null or SecurityException paths.
384d81c
@mjohns-databricks
SecurityException handling.
1d8ad69
@mjohns-databricks
try catch fix.
21f8ae3
@mjohns-databricks
fix Files.exists.
cf473e5
@mjohns-databricks
relaxed subdataset cleanup criteria.
3879809
@mjohns-databricks
revert cleanup, wrap in try catch.
a4c7c56
@mjohns-databricks
Added PathUtils.NO_PATH_STRING
b447246
@mjohns-databricks
numpy to 1.22.4 (avoid scipy conflict). R off.
f493a50
@mjohns-databricks
numpy to 1.22.4 (removed gdal[numpy]).
19dfdac
@mjohns-databricks
numpy rationalization.
9930f47
@mjohns-databricks
install native for python gdal.
c95f878
@mjohns-databricks
standardize install setups.
47cef69
@mjohns-databricks
streamline python deps for testing.
1f09809
@mjohns-databricks
address python build failure.
5910c82
@mjohns-databricks
Adding --no-build-isolation for gdal.
2f6d0fc
@mjohns-databricks
gdal --no-build-isolation; binary limit logic.
93276cb
@mjohns-databricks
--no-build-isolation only for python build.
125e088
@mjohns-databricks
python build requires args for gdal.
da5b5fd
@mjohns-databricks
update rst_fromcontent signature.
b579239
@mjohns-databricks
Copy link
Contributor Author

@milos-colic please give this another look. Recommend we merge soonish and then come back with doc changes.

@milos-colic milos-colic merged commit 5c245f9 into databrickslabs:main Jan 9, 2024
5 of 7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@milos-colic milos-colic milos-colic approved these changes

@sllynn sllynn Awaiting requested review from sllynn

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@mjohns-databricks @CLAassistant @milos-colic