Remote execution of worklow/pipeline with remote environments values

[rdubois-mel]

Hello everyone,

I'm trying to understand how remote execution of workflows and pipelines works.

I set up a hop server that listens on specific port.
I configured remote execution from hop-gui.
If I check the box "Export linked resources to server", it works, it generates the zip for me, which it transmits to the server and executes the workflow.
But using the variables values (from environments) found on the hop-gui workstation.

I would like it to use differents values located in a file on the server.

So I unchecked the "Export linked resources to server" box, and I entered the source and destination folders so that it uses the files on the server, but it doesn't work. Linux server always looks for my linked files on C:\etc...

I specify that I would like to have a hop server which is not necessarily launched with a particular project, because I have several projects and would potentially like to reuse it for all.

Previously I used Kettle and launched my jobs via Carte, I understood that this had changed.

Is it possible to work like this or am I wrong?

Remi

[bamaer]

Correct, that has changed. Hop Server is stateless, the workflows and pipelines you want to run on the server need to be registered there. One approach you could take is to create a server environment in your hop-gui and use the "export linked resources" to get your workflows and pipelines on the server correctly.

[rdubois-mel]

OK thanks.
This is how I thought it would work.

So if I want to launch my workflows from an orchestrator automatically, I would instead have to use hop-run on a server, and store the environment configuration files on this server securely. Projects files would be synchronised between dev workstation and execution server by git. (But I have to setup all of this on one of the orchestrator nodes)
And possibly if I want to distribute the load I launch the workflows remotely on another server.

Or should I use Dockers containers in short-lived mode ?

I'll think about all that.

Thank you very much for you answer, and for this project which is great and which really gives a boost of freshness for legacy Pentaho users ;)

[hansva]

IMHO if you can... I would go for short-lived containers. The server does not really provide any advantages (except maybe for the web UI)

[bamaer]

agreed, short-lived containers will definitely make your life easier.

[thadguidry]

If you run into technical challenges in your environments/servers with Hop directly, you could always use Ansible for the automation for running/transferring things.

[rdubois-mel]

Thanks everyone, I'll look at this and do some testing.

[rdubois-mel]

Is there a way to define an additional path for the plugins used by the short-lived container ?

[bamaer]

we have a HOP_PLUGIN_BASE_FOLDERS env var, but it looks like that isn't accepted in the container.

[rdubois-mel]

Ok I'm testing something else while waiting for the possible addition of this parameter perhaps one day. Indeed some of my workflows use the AtolCD GIS plugin.

I had one last question. the HOP_CUSTOM_ENTRYPOINT_EXTENSION_SHELL_FILE_PATH parameter is described as such:
"Optional path to a custom entrypoint extension script file. You can use this for example to fetch Hop project files from S3 or gitlab."

But I can't add git to the container because of the hop user permissions. Would it be possible to include the git command by default in the container? The other solution is to prepare my files folder externally before launching the container.

[hansva]

you can extend the docker image and in your docker file, you should be able to just go back to root user install things and then switch back to the Hop user.

[bamaer]

there's no technical reason that prevents us to add anything to the default container image. However, this is intended to be a general purpose image. Adding git makes sense for you (and probably a lot of other people), but there will also be use cases where adding python, R or tons of other packages would make sense.

The easiest option is to create your own custom image and add your modifications, projects + environments files etc. You can extend the existing image or use the default Dockerfile as a starting point to build your own image from scratch.
If you're in AWS, one option could be to use AWS Copilot to build a custom image and run it on AWS ECS/Fargate.

[rdubois-mel]

Good point,
I took the opportunity to add git AND the GIS AtolCD plugin in a custom image.
I'm going to have fun now :)
Thanks

[rdubois-mel]

My docker container works very well, but I still have one last question.

Currently to determine if my workflow (or pipeline) worked, I do a little regular expression on the logs, looking for sentences like "Pipeline detected one or more transforms with errors|ERROR: Unable to run workflow". Is there a cleaner way to retrieve the execution status from the container?

(Previously with Pentaho, we used Carte webservices and queried the status.)

[hansva]

You can check the exit code of the docker run command. It should propagate the exit status of the process inside

[rdubois-mel]

Sorry for the inconvenience, it was indeed that simple :)