Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[dask] generating dask/dask.sh from template #1287

Draft
wants to merge 136 commits into
base: master
Choose a base branch
from

Conversation

cjac
Copy link
Contributor

@cjac cjac commented Jan 5, 2025

generate dask/dask.sh from templates developed for #1282

@cjac cjac self-assigned this Jan 5, 2025
@cjac
Copy link
Contributor Author

cjac commented Jan 5, 2025

/gcbrun

1 similar comment
@cjac
Copy link
Contributor Author

cjac commented Jan 5, 2025

/gcbrun

@cjac cjac force-pushed the dask-template-20250104 branch from 027d1e2 to 55bed36 Compare January 5, 2025 01:50
@cjac
Copy link
Contributor Author

cjac commented Jan 5, 2025

/gcbrun

10 similar comments
@cjac
Copy link
Contributor Author

cjac commented Jan 5, 2025

/gcbrun

@cjac
Copy link
Contributor Author

cjac commented Jan 5, 2025

/gcbrun

@cjac
Copy link
Contributor Author

cjac commented Jan 5, 2025

/gcbrun

@cjac
Copy link
Contributor Author

cjac commented Jan 5, 2025

/gcbrun

@cjac
Copy link
Contributor Author

cjac commented Jan 5, 2025

/gcbrun

@cjac
Copy link
Contributor Author

cjac commented Jan 6, 2025

/gcbrun

@cjac
Copy link
Contributor Author

cjac commented Jan 6, 2025

/gcbrun

@cjac
Copy link
Contributor Author

cjac commented Jan 6, 2025

/gcbrun

@cjac
Copy link
Contributor Author

cjac commented Jan 6, 2025

/gcbrun

@cjac
Copy link
Contributor Author

cjac commented Jan 6, 2025

/gcbrun

@cjac cjac force-pushed the dask-template-20250104 branch from 2767343 to 7b767be Compare January 6, 2025 04:55
@cjac
Copy link
Contributor Author

cjac commented Jan 6, 2025

/gcbrun

@cjac cjac changed the title [template] generating dask/dask.sh from template [dask] generating dask/dask.sh from template Jan 8, 2025
@cjac
Copy link
Contributor Author

cjac commented Jan 8, 2025

/gcbrun

1 similar comment
@cjac
Copy link
Contributor Author

cjac commented Jan 8, 2025

/gcbrun

cjac added 5 commits January 8, 2025 22:09
* increased minimum memory threshold for ram disk
* moved apt_add_repo and friends to common/install_functions

templates/dask/util_functions:
* validating conda tarball before caching to gcs

templates/generate-action.pl:
* improved usage documentation a little

templates/gpu/install_functions
* using /opt/conda/miniconda3/bin/python3 instead of /usr/bin/ for
  venv pre-install
* increase wait time for scheduler to come online
* reduce noise from tar -t

templates/gpu/yarn_functions,
templates/gpu/install_functions:
* protect many functions from running without attached accelerator

templates/gpu/install_gpu_driver.sh.in
* set +e in exit handler

templates/gpu/spark_functions:
* re-factor new function into this template

templates/spark-rapids/spark-rapids.sh.in
* removed redundant call to configure_gpu_script
* set +e in exit handler
@cjac cjac force-pushed the dask-template-20250104 branch from 43a6efe to 1c8e7bd Compare January 9, 2025 19:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant