Pan-os version 11 genericpolling DT fails

[adambaumeister]

All generic polling playbooks that are looking at jobs like download/install software fail on version 11 due to duplicate job ids in the context.

[GrantGabbert]

This issue seems to've started in 10.2 NGFW code

[alperenkose]

PANOS XML API literally returns multiple records for the same job ids after 10.2, which I believe is the root cause for this. See below response I tested on 10.2.9-h1.

<response status="success">
    <result>
        <job>
            <tenq>2024/07/08 04:57:51</tenq>
            <tdeq>04:57:51</tdeq>
            <id>4</id>
            <user/>
            <type>Downld</type>
            <status>FIN</status>
            <queued>NO</queued>
            <stoppable>no</stoppable>
            <result>OK</result>
            <tfin>2024/07/08 04:58:03</tfin>
            <description/>
            <positionInQ>0</positionInQ>
            <progress>2024/07/08 04:58:03</progress>
            <details>
                <line>Successfully downloaded</line>
                <line>Transferring a copy of image to HA peer</line>
                <line>Preloading into software manager</line>
                <line>Successfully loaded into software manager</line>
            </details>
            <warnings/>
        </job>
        <job>
            <tenq>2024/07/03 01:02:03</tenq>
            <tdeq>01:02:03</tdeq>
            <id>3</id>
            <user/>
            <type>Downld</type>
            <status>FIN</status>
            <queued>NO</queued>
            <stoppable>no</stoppable>
            <result>OK</result>
            <tfin>2024/07/03 01:02:05</tfin>
            <description/>
            <positionInQ>0</positionInQ>
            <progress>2024/07/03 01:02:05</progress>
            <details>
                <line>File successfully downloaded </line>
            </details>
            <warnings/>
        </job>
        <job>
            <tenq>2024/07/01 06:39:16</tenq>
            <tdeq>06:39:29</tdeq>
            <id>2</id>
            <user/>
            <type>AutoCom</type>
            <status>FIN</status>
            <queued>NO</queued>
            <stoppable>no</stoppable>
            <result>OK</result>
            <tfin>2024/07/01 06:40:26</tfin>
            <description/>
            <positionInQ>0</positionInQ>
            <progress>100</progress>
            <details>
                <line>Configuration committed successfully</line>
                <line>Successfully committed last configuration</line>
            </details>
            <warnings/>
        </job>
        <job>
            <tenq>2024/07/08 04:57:51</tenq>
            <tdeq>04:57:51</tdeq>
            <id>4</id>
            <user/>
            <type>Downld</type>
            <status>FIN</status>
            <queued>NO</queued>
            <stoppable>no</stoppable>
            <result>OK</result>
            <tfin>2024/07/08 04:58:03</tfin>
            <description/>
            <positionInQ>0</positionInQ>
            <progress>2024/07/08 04:58:03</progress>
            <details>
                <line>Successfully downloaded</line>
                <line>Transferring a copy of image to HA peer</line>
                <line>Preloading into software manager</line>
                <line>Successfully loaded into software manager</line>
            </details>
            <warnings/>
        </job>
        <job>
            <tenq>2024/07/03 01:02:03</tenq>
            <tdeq>01:02:03</tdeq>
            <id>3</id>
            <user/>
            <type>Downld</type>
            <status>FIN</status>
            <queued>NO</queued>
            <stoppable>no</stoppable>
            <result>OK</result>
            <tfin>2024/07/03 01:02:05</tfin>
            <description/>
            <positionInQ>0</positionInQ>
            <progress>2024/07/03 01:02:05</progress>
            <details>
                <line>File successfully downloaded </line>
            </details>
            <warnings/>
        </job>
    </result>
</response>

[GrantGabbert]

I did notice that the PAN job API was returning duplicate entries in the 10.2/11 codebase, but not before. And that seemed like the root cause. But, there's also something else going on in XSOAR.

The DT for the poller is - PANOS.JobStatus(val.status!=="FIN").id. So even with duplicate job entries returned from PANOS, I would expect that DT should still give 0 results once the job is completed.

What I'm seeing in the context is more of a history, instead of just the last run.

For example, the last upgrade I did, the command result show the 2 "FIN" entries in the war room...

PAN-OS Job Status

hostid	id	positionInQ	progress	result	status	stoppable	tenq	tfin	type
021201010640	37	0	2024/06/17 16:22:25	OK	FIN	no	2024/06/17 16:16:09	2024/06/17 16:22:25	SWInstall
021201010640	37	0	2024/06/17 16:22:25	OK	FIN	no	2024/06/17 16:16:09	2024/06/17 16:22:25	SWInstall

But the context shows 4 entries with different percentages....

{
  "JobStatus": [
    {
      "description": null,
      "hostid": "021201010640",
      "id": 37,
      "positionInQ": "0",
      "progress": "0",
      "result": "PEND",
      "status": "ACT",
      "stoppable": "no",
      "tenq": "2024/06/17 16:16:09",
      "tfin": null,
      "type": "SWInstall",
      "user": null
    },
    {
      "description": null,
      "hostid": "021201010640",
      "id": 37,
      "positionInQ": "0",
      "progress": "37",
      "result": "PEND",
      "status": "ACT",
      "stoppable": "no",
      "tenq": "2024/06/17 16:16:09",
      "tfin": null,
      "type": "SWInstall",
      "user": null
    },
    {
      "description": null,
      "hostid": "021201010640",
      "id": 37,
      "positionInQ": "0",
      "progress": "30",
      "result": "PEND",
      "status": "ACT",
      "stoppable": "no",
      "tenq": "2024/06/17 16:16:09",
      "tfin": null,
      "type": "SWInstall",
      "user": null
    },
    {
      "description": null,
      "hostid": "021201010640",
      "id": 37,
      "positionInQ": "0",
      "progress": "2024/06/17 16:22:25",
      "result": "OK",
      "status": "FIN",
      "stoppable": "no",
      "tenq": "2024/06/17 16:16:09",
      "tfin": "2024/06/17 16:22:25",
      "type": "SWInstall",
      "user": null
    }
  ]
}

[alperenkose]

@GrantGabbert I did a PR on demisto/content to filter by job id using show jobs id command. It resolves this issue as it returns a single entry when you query with a job id now. But your point on DT filtering remains the same. Maybe we can close this issue and investigate DT filtering separately.

[GrantGabbert]

So the issue was for generic polling failures, and that hasn't been resolved yet. So I think it would make the most sense to keep this issue open until that is resolved (or it's decided to not fix it).

Also, with the changes to the pan-os integration, are there also changes that need to be added in this repo to utilize new show job functionality?

[alperenkose]

No changes needed on this repo to utilize the new functionality. I have tested Generic Polling for downloading software and software installation on panos 10.2, both works as expected now and tracks the specific job id.
Why DT filtering didn't work with multiple results before this change is an issue with the XSOAR itself actually.