Alternate spark runtime implementation #406
Conversation
blublinsky
force-pushed
the
spark_experimental
branch
from
2dc2629 to
9413086
Compare
experimental/spark/src/data_processing_spark/runtime/spark/transform_launcher.py
Outdated
Show resolved
Hide resolved
blublinsky
force-pushed
the
spark_experimental
branch
from
3d8f301 to
81d41ac
Compare
data-processing-lib/python/src/data_processing/transform/abstract_transform.py
Outdated
Show resolved
Hide resolved
data-processing-lib/spark/src/data_processing_spark/runtime/spark/execution_configuration.py
Outdated
Show resolved
Hide resolved
data-processing-lib/spark/src/data_processing_spark/runtime/spark/transform_file_processor.py
Show resolved
Hide resolved
data-processing-lib/spark/src/data_processing_spark/runtime/spark/transform_file_processor.py
Outdated
Show resolved
Hide resolved
data-processing-lib/spark/src/data_processing_spark/runtime/spark/transform_file_processor.py
Outdated
Show resolved
Hide resolved
| super().__init__(runtime_config, data_access_factory) | ||
| self.execution_config = SparkTransformExecutionConfiguration(name=runtime_config.get_name()) | ||
|
|
||
| def __get_parameters(self) -> bool: |
There was a problem hiding this comment.
maybe its time to move these up to the super class
There was a problem hiding this comment.
Which ones? We are using different classes
data-processing-lib/spark/src/data_processing_spark/runtime/spark/transform_orchestrator.py
Show resolved
Hide resolved
| logger.debug("Building job metadata") | ||
| input_params = runtime_config.get_transform_metadata() | execution_config.get_input_params() | ||
| metadata = { | ||
| "pipeline": execution_config.pipeline_id, |
There was a problem hiding this comment.
is the "parallelization" being captured as metadata somewhere?
There was a problem hiding this comment.
Yes. input parameters
blublinsky
force-pushed
the
spark_experimental
branch
3 times, most recently
from
052e929 to
625132c
Compare
There was a problem hiding this comment.
Please reinstate the two-stage build process for the Dockerfile. I also suggest that we keep an argument whether we want to use Hadoop or not during the build process, with the value of that argument set to false by default.
I have also created a requirements page for supporting fuzzy dedup. I suggest that we provide support for fuzzy dedup in this runtime before merging it into the dev code branch.
blublinsky
force-pushed
the
spark_experimental
branch
from
75325f7 to
21105a8
Compare
|
I think its time to merge |
blublinsky
force-pushed
the
spark_experimental
branch
from
576483b to
7117079
Compare
Signed-off-by: Constantin M Adam <[email protected]>
Why are these changes needed?
Related issue number (if any).
#197