Multi pulsar deployment

[pauldg]

This PR loops over the galaxyproject.pulsar role with variables specific to each deployment.
The shared pulsar variables have been copied over from https://github.com/usegalaxy-eu/vgcn/blob/dev/ansible/group_vars/pulsar.yml

[bgruening]

There are so many pulsar.yml files ... can we give them more descriptinve names maybe?

[mira-miracoli]

Should this be configurable?
I could imagine the playbook iterates over a list of dicts which is provided by a terraform variable, if we want to stick to deployment using one single command; similar to this

[mtangaro]

@gm-ds is further working on this. We need the list of queues as input for endpoint configuration, with also the “short name” of the endpoint, e.g. it01..., for path configuration.
We can, probably, parse the queue to retrieve the short name. Do you think this can be fine?

[mira-miracoli]

Very cool, thank you @pauldg 🚀

[sanjaysrikakulam]

Cool, thanks, @pauldg. I assume this deployment style would mean that the admin/user deploying this should know about the different pulsars (diff Galaxy instances) that they plan to deploy in advance, right? What happens when one wants to deploy/set up multiple pulsars in a later stage (after the initial deployment with one)? Will Terraform run the playbook, or will it say there are no infrastructural changes and that it does not execute/deploy the additional pulsar conf?

[mira-miracoli]

What happens when one wants to deploy/set up multiple pulsars in a later stage (after the initial deployment with one)? Will Terraform run the playbook, or will it say there are no infrastructural changes and that it does not execute/deploy the additional pulsar conf?

terraform is not aware of software config as far as I am aware of.
I would recommend in the docs to tear down and recreate. Maybe we should make sure that galaxy recovers jobs in these cases

[sanjaysrikakulam]

What happens when one wants to deploy/set up multiple pulsars in a later stage (after the initial deployment with one)? Will Terraform run the playbook, or will it say there are no infrastructural changes and that it does not execute/deploy the additional pulsar conf?

terraform is not aware of software config as far as I am aware of. I would recommend in the docs to tear down and recreate. Maybe we should make sure that galaxy recovers jobs in these cases

Tearing down every time and recreating is not a viable or sustainable solution. I would suggest the following simple way.

1. Keep this method that this PR implements (for the scenarios where the admin/user already knows that they want to configure multiple pulsars)
2. Extract this also as a playbook (a separate one) and write documentation that would allow the admins/users to deploy multiple pulsars at their will without affecting the existing deployment(s).

[mtangaro]

2. Extract this also as a playbook (a separate one) and write documentation that would allow the admins/users to deploy multiple pulsars at their will without affecting the existing deployment(s).

Hi @sanjaysrikakulam, I've extracted the playbook and I'm configuring it. I've configured the endpoint for running with "eu" and "it". The strange thing is that if I run a job from .eu, it is using the wrong assign method, with ids, in the staging dir.

The app.yml is
`---

assign_ids: none
conda_auto_init: false
conda_auto_install: false
container_image_cache_path: /data/share/var/database/container_cache
dependency_resolvers_config_file: dependency_resolvers_conf.xml
job_metrics_config_file: /opt/pulsar/config/job_metrics_conf.xml
managers:
benchmarking:
submit_universe: vanilla
type: queued_condor
production:
submit_universe: vanilla
type: queued_condor
preprocess_action_max_retries: 10
preprocess_action_interval_start: 2
preprocess_action_interval_step: 2
preprocess_action_interval_max: 30
postprocess_action_max_retries: 10
postprocess_action_interval_start: 2
postprocess_action_interval_step: 2
postprocess_action_interval_max: 30
test:
submit_universe: vanilla
type: queued_condor
message_queue_url: pyamqp://galaxy_it03:[email protected]:5671//pulsar/galaxy_it03?ssl=1
min_polling_interval: 0.5
persistence_directory: /data/share/eu/persisted_data
staging_directory: /data/share/eu/staging
tool_dependency_dir: /data/share/tools/`

But I get:
root@it03-central-manager-usegalaxy-it:/data/share/eu/staging$ ls
344 345 361 75395955 75395967 75396384 75396386 75396528 75396539 75396568

Did you experience something like this?

ps. 344, 345, 361 I think were very old jobs that eu submitted when I've reconfigured the endpoint.

[sanjaysrikakulam]

2. Extract this also as a playbook (a separate one) and write documentation that would allow the admins/users to deploy multiple pulsars at their will without affecting the existing deployment(s).

Hi @sanjaysrikakulam, I've extracted the playbook and I'm configuring it. I've configured the endpoint for running with "eu" and "it". The strange thing is that if I run a job from .eu, it is using the wrong assign method, with ids, in the staging dir.

The app.yml is `---

assign_ids: none conda_auto_init: false conda_auto_install: false container_image_cache_path: /data/share/var/database/container_cache dependency_resolvers_config_file: dependency_resolvers_conf.xml job_metrics_config_file: /opt/pulsar/config/job_metrics_conf.xml managers: benchmarking: submit_universe: vanilla type: queued_condor production: submit_universe: vanilla type: queued_condor preprocess_action_max_retries: 10 preprocess_action_interval_start: 2 preprocess_action_interval_step: 2 preprocess_action_interval_max: 30 postprocess_action_max_retries: 10 postprocess_action_interval_start: 2 postprocess_action_interval_step: 2 postprocess_action_interval_max: 30 test: submit_universe: vanilla type: queued_condor message_queue_url: pyamqp://galaxy_it03:[email protected]:5671//pulsar/galaxy_it03?ssl=1 min_polling_interval: 0.5 persistence_directory: /data/share/eu/persisted_data staging_directory: /data/share/eu/staging tool_dependency_dir: /data/share/tools/`

But I get: root@it03-central-manager-usegalaxy-it:/data/share/eu/staging$ ls 344 345 361 75395955 75395967 75396384 75396386 75396528 75396539 75396568

Did you experience something like this?

ps. 344, 345, 361 I think were very old jobs that eu submitted when I've reconfigured the endpoint.

Hey Marco,

I don't totally understand your problem (please clarify if I have misunderstood it) because the IDs seem okay. When Pulsar is not configured with assign_ids=uuid, it will use the IDs assigned by the Galaxy instances. The old ones you have in the staging directory are probably differently configured or something in the past. Our (EU) job IDs are currently in this range 75396568.

[mtangaro]

I don't totally understand your problem (please clarify if I have misunderstood it) because the IDs seem okay. When Pulsar is not configured with assign_ids=uuid, it will use the IDs assigned by the Galaxy instances. The old ones you have in the staging directory are probably differently configured or something in the past. Our (EU) job IDs are currently in this range 75396568.

The problem is that, as usual, Pulsar is configured with "assign_ids: none" but I have IDs, so jobs are failing.

[mtangaro]

update (ty @sanjaysrikakulam).
The endpoint need proper configuration in the destinations.yml file, as shown here.
I'll update properly the documentation and upload the playbook after cleaning it a bit.
In principle we can keep EU in the usual path and national instances in sub dirs.

[mtangaro]

Multi pulsar deployment:

• PR with tf and ansible: Multiple Pulsar Services Deployment #23
• PR with documentation: New Pages for the Multiple Pulsar Services Deployment pulsar-network-docs#14