Worklflow frozen/not updating after ~100 nodes.

[khairulashraff]

I have a workflow which can spawn a lot of nodes, in the thousands. Some months ago when I did it, it gets stuck at around ~4,000 nodes. I found out I need to enable persistence for such a large workflow. Enabled it and all has been working well.

Fast forward to today, the same workflow but it gets frozen at about ~100 nodes with the same symptoms. The nodes' status does not get updated and proceed to the next steps even though the pods itself has completed and stops (I checked both log and kubectl get pods, the pods has definitely completed its run, even the nodes has started to scale down).

For the pods that get stuck, there will be quite a lot of these logs from workflow-controller (following is just an excerpt of a very long repetition of the same thing).

time="2021-07-05T08:39:06.301Z" level=info msg="node foobar-xj8wx-1212838881 phase Running -> Succeeded" namespace=default workflow=foobar-xj8wx
time="2021-07-05T08:39:06.301Z" level=info msg="node foobar-xj8wx-1212838881 finished: 2021-07-05 08:39:06.301609813 +0000 UTC" namespace=default workflow=foobar-xj8wx
time="2021-07-05T08:39:40.426Z" level=info msg="node foobar-xj8wx-1212838881 phase Running -> Succeeded" namespace=default workflow=foobar-xj8wx
time="2021-07-05T08:39:40.426Z" level=info msg="node foobar-xj8wx-1212838881 finished: 2021-07-05 08:39:40.426623122 +0000 UTC" namespace=default workflow=foobar-xj8wx
time="2021-07-05T08:40:14.557Z" level=info msg="node foobar-xj8wx-1212838881 phase Running -> Succeeded" namespace=default workflow=foobar-xj8wx
time="2021-07-05T08:40:14.557Z" level=info msg="node foobar-xj8wx-1212838881 finished: 2021-07-05 08:40:14.557380114 +0000 UTC" namespace=default workflow=foobar-xj8wx

Telling it to Stop or Terminate will not do anything either (again, same symptom as workflow being too large for etcd).

Since persistence has been enabled, and it's only ~100 nodes, where/how/what do I debug to find out why is this happening?

Specs (I tried a whole bunch of stuff, they all behave the same):

• GKE 1.19 / 1.20
• Argo 2.11 / 2.12.4 / 3.1.1
• Persistence using postgres

---

EDIT

I thought the workflow froze, but actually it didn't and manage to run all tasks successfully. It was just the UI not updating according to the actual status of the nodes. Any idea on what to check for debugging?

[tczhao]

have a look at #6256, I think they are related

[khairulashraff]

Yup, that definitely sums up my issue. I'll follow up on that issue then, thanks!