Skip to content

Commit

Permalink
Fix deployment of spark_python_operator (#1033)
Browse files Browse the repository at this point in the history
This properly packages the spark operation "kernelspec"
and adds spark_python_operator to the corresponding list
of exposed kernelspecs in the k8s deployment scripts.

The operator-related scripts/templates have been moved to
kernel-launchers/operators to more easily enable deployment 
and not clutter other Kubernetes kernelspecs with unrelated 
scripts/templates.
  • Loading branch information
kevin-bates authored Jan 27, 2022
1 parent 3ac7277 commit fcd24fd
Show file tree
Hide file tree
Showing 5 changed files with 4 additions and 2 deletions.
3 changes: 2 additions & 1 deletion etc/Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -52,9 +52,10 @@ TOREE_LAUNCHER_FILES:=$(shell find kernel-launchers/scala/toree-launcher/src -ty
@echo ../build/kernelspecs/spark_python_{conductor*,yarn*} | xargs -t -n 1 cp -r kernel-launchers/python/scripts
@echo ../build/kernelspecs/spark_R_{conductor*,yarn*} | xargs -t -n 1 cp -r kernel-launchers/R/scripts
@echo ../build/kernelspecs/spark_scala_{conductor*,yarn*} | xargs -t -n 1 cp -r kernel-launchers/scala/lib
# Container-based kernelspecs just get the container launchers
# Container-based kernelspecs (and operators) just get the container launchers
@echo ../build/kernelspecs/{python,R,scala,python_tf,python_tf_gpu}_kubernetes | xargs -t -n 1 cp -r kernel-launchers/kubernetes/*
@echo ../build/kernelspecs/{python,R,scala,python_tf,python_tf_gpu}_docker | xargs -t -n 1 cp -r kernel-launchers/docker/*
@echo ../build/kernelspecs/spark_python_operator | xargs -t -n 1 cp -r kernel-launchers/operators/*
# Populate kernel resources. Because tensorflow is also python, it should be last.
@echo ../build/kernelspecs/*R* | xargs -t -n 1 cp -r kernel-resources/ir/*
@echo ../build/kernelspecs/*scala* | xargs -t -n 1 cp -r kernel-resources/apache_toree/*
Expand Down
2 changes: 1 addition & 1 deletion etc/kubernetes/enterprise-gateway.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -184,7 +184,7 @@ spec:
value: "60"

- name: EG_KERNEL_WHITELIST
value: "'r_kubernetes','python_kubernetes','python_tf_kubernetes','python_tf_gpu_kubernetes','scala_kubernetes','spark_r_kubernetes','spark_python_kubernetes','spark_scala_kubernetes'"
value: "'r_kubernetes','python_kubernetes','python_tf_kubernetes','python_tf_gpu_kubernetes','scala_kubernetes','spark_r_kubernetes','spark_python_kubernetes','spark_scala_kubernetes','spark_python_operator'"

- name: EG_DEFAULT_KERNEL_NAME
value: "python_kubernetes"
Expand Down
1 change: 1 addition & 0 deletions etc/kubernetes/helm/enterprise-gateway/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ kernel:
- spark_r_kubernetes
- spark_python_kubernetes
- spark_scala_kubernetes
- spark_python_operator
# Default kernel name should be something from the whitelist
defaultKernelName: python_kubernetes

Expand Down

0 comments on commit fcd24fd

Please sign in to comment.