TASK Manage NDFC Fabric VRFs fails for existing VRFs

[marehler]

Ansible Version

ansible [core 2.17.0]

Ansible Collection Versions

Collection                               Version
---------------------------------------- -------
ansible.netcommon                        5.3.0  
ansible.posix                            1.5.4  
ansible.utils                            2.11.0 
cisco.dcnm                               3.5.0  
cisco.nac_dc_vxlan                       0.3.0  
cisco.nxos                               5.3.0  
community.general                        8.5.0

Cisco NDFC Version

12.1.3b

Cisco NX-OS Version

9.3(13)

Which role is this issue related to?

cisco.nac_dc_vxlan.dtc.create

Which section of the data model is this issue related to?

vxlan.overlay_services

Expected Behavior

Existing VRF is maintained.

Actual Behavior

ansible-playbook -i inventory.yaml deploy.yaml --tags cr_manage_vrfs_networks -vvv

<...>

TASK [cisco.nac_dc_vxlan.create : Manage NDFC Fabric VRFs] ******************************************************************************************
task path: /Users/marehler/nac-vxlan/ansible-dc-vxlan-example/collections/ansible_collections/cisco/nac_dc_vxlan/roles/dtc/create/tasks/vrfs_networks.yml:35
fatal: [marehler_vnd2_1]: FAILED! => {
    "changed": false,
    "module_stderr": "invalid syntax (<unknown>, line 0)",
    "module_stdout": "",
    "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error"
}

Ansible Playbook

---
# This is the main entry point playbook for calling the various roles in this collection.
- hosts: marehler_vnd3
  any_errors_fatal: true
  gather_facts: no

  roles:
    # -----------------------
    # DataCenter Roles
    #   Role: cisco.netascode_dc_vxlan.dtc manages NDFC workflows
    #
    - role: cisco.nac_dc_vxlan.dtc.create
    - role: cisco.nac_dc_vxlan.dtc.deploy
    - role: cisco.nac_dc_vxlan.dtc.remove

Steps to Reproduce

• Manually create and attach a VRF in NDFC
• Add the VRF with the same name, VRF ID, and VLAN to the vxlan.overlay_services.vrfs configuration data.
• Run the playbook as shown above.

Comments

• The error is caused by the cisco.dcnm.dcnm_vrf module.
• The issue does not occur if you create a new VRF.
• The issue does not occur if the existing VRF is not attached.

Relevant Debug Output

No response

[marehler]

NDFC screenshots of sample VRF:

[marehler]

vrfs.yaml Configuration Data:

---
vxlan:
  overlay_services:
    vrfs:
      - name: BLUE
        vrf_id: 30001
        vlan_id: 101
        vrf_attach_group: all_leaf

    vrf_attach_groups:
      - name: all_leaf
        switches:
          - hostname: N9Kv-LEAF1
          - hostname: N9Kv-LEAF2

[marehler]

The issue can also be reproduced by isolating the task in a playbook and inserting the config data directly:

---

- name: Debug DCNM VRF
  hosts: marehler_vnd2_1
  any_errors_fatal: true
  gather_facts: no

  tasks:

  - name: Manage NDFC Fabric VRFs
    cisco.dcnm.dcnm_vrf:
      fabric: VXLAN-FABRIC-1
      state: replaced
      config:
      - vrf_name: BLUE
        vrf_id: 30001
        vlan_id: 101
        attach:
        - ip_address: 22.1.172.21
        - ip_address: 22.1.172.22

ansible-playbook -i inventory.yaml dcnm-vrf.yaml -v  
Using /Users/marehler/nac-vxlan/ansible-dc-vxlan-example/ansible.cfg as config file

PLAY [Debug DCNM VRF] *******************************************************************************************************************************

TASK [Manage NDFC Fabric VRFs] **********************************************************************************************************************
fatal: [marehler_vnd2_1]: FAILED! => {"changed": false, "module_stderr": "invalid syntax (<unknown>, line 0)", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error"}

[mikewiebe]

@marehler quick update here. I was able to recreate this issue but my error message is slightly different.

fatal: [nac-fabric1]: FAILED! => {
    "changed": false,
    "module_stderr": "'loopbackId'",
    "module_stdout": "",
    "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error"
}

My version of NDFC is more recent so that could explain the difference.

The good bit of news is that we are working on a series of fixes in the following PR and when I run your scenario using that branch I don't see the issue.

CiscoDevNet/ansible-dcnm#354

Can you retest using the dcnm-vrf-fix-issue-351 branch from the PR and let me know if you still see the issue?

[devegupt]

@marehler quick update here. I was able to recreate this issue but my error message is slightly different.

fatal: [nac-fabric1]: FAILED! => {
    "changed": false,
    "module_stderr": "'loopbackId'",
    "module_stdout": "",
    "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error"
}

My version of NDFC is more recent so that could explain the difference.

The good bit of news is that we are working on a series of fixes in the following PR and when I run your scenario using that branch I don't see the issue.

CiscoDevNet/ansible-dcnm#354

Can you retest using the dcnm-vrf-fix-issue-351 branch from the PR and let me know if you still see the issue?

@mikewiebe, I was testing VRF create in a brownfield setup yesterday for a customer POC with the project team and we faced exact same issue. Somehow with multiple testing the error got away, we were running it on the NDFC 12.2.2

TASK [cisco.nac_dc_vxlan.create : Manage NDFC Fabric VRFs] ********************************************************************************
task path: /opt/vxlanascode/nac-ndfc/collections/ansible_collections/cisco/nac_dc_vxlan/roles/dtc/create/tasks/vrfs_networks.yml:34
Thursday 19 December 2024  05:22:10 -0500 (0:00:00.195)       0:00:38.166 ***** 
Thursday 19 December 2024  05:22:10 -0500 (0:00:00.195)       0:00:38.165 ***** 
fatal: [nac-ndfc-dalian]: FAILED! => changed=false 
  module_stderr: '''loopbackId'''
  module_stdout: ''
  msg: |-
    MODULE FAILURE
    See stdout/stderr for the exact error

[marehler]

@mikewiebe I have installed and retested with branch dcnm-vrf-fix-issue-351, still using NDFC 12.1.3. I still see the issue.

[marehler]

I have now tested with NDFC 12.2.2 too. I do not see the issue with 12.2.2.

[marehler]

dcnm_vrf.py fails in 'def diff_for_attach_deploy()' in line 840:

have_inst_values = ast.literal_eval(have["instanceValues"])

ast.literal_eval('') raises a syntax error. The VRF instanceValues must not be empty.

Debug ouput:

(Epdb) p want_a["lanAttachList"]
[{'import_evpn_rt': '', 'export_evpn_rt': '', 'vrf_lite': None, 'extensionValues': '', 'fabric': 'VXLAN-FABRIC-1', 'vrfName': 'BLUE', 'vlan': 101, 'deployment': True, 'isAttached': True, 'serialNumber': '9CB27ZKT24E', 'is_deploy': True, 'freeformConfig': '', 'instanceValues': '{"loopbackId":"","loopbackIpAddress":"","loopbackIpV6Address":"","switchRouteTargetImportEvpn":"","switchRouteTargetExportEvpn":""}'}, {'import_evpn_rt': '', 'export_evpn_rt': '', 'vrf_lite': None, 'extensionValues': '', 'fabric': 'VXLAN-FABRIC-1', 'vrfName': 'BLUE', 'vlan': 101, 'deployment': True, 'isAttached': True, 'serialNumber': '9QSL59WP7O6', 'is_deploy': True, 'freeformConfig': '', 'instanceValues': '{"loopbackId":"","loopbackIpAddress":"","loopbackIpV6Address":"","switchRouteTargetImportEvpn":"","switchRouteTargetExportEvpn":""}'}]

(Epdb) p have_a["lanAttachList"]
[{'vrfName': 'BLUE', 'peerSerialNo': None, 'instanceValues': '', 'entityName': 'BLUE', 'fabric': 'VXLAN-FABRIC-1', 'vlan': 101, 'serialNumber': '9CB27ZKT24E', 'deployment': True, 'extensionValues': '', 'isAttached': True, 'is_deploy': True, 'freeformConfig': ''}, {'vrfName': 'BLUE', 'peerSerialNo': None, 'instanceValues': '', 'entityName': 'BLUE', 'fabric': 'VXLAN-FABRIC-1', 'vlan': 101, 'serialNumber': '9QSL59WP7O6', 'deployment': True, 'extensionValues': '', 'isAttached': True, 'is_deploy': True, 'freeformConfig': ''}]

(Epdb) p want_a
[{'import_evpn_rt': '', 'export_evpn_rt': '', 'vrf_lite': None, 'extensionValues': '', 'fabric': 'VXLAN-FABRIC-1', 'vrfName': 'BLUE', 'vlan': 101, 'deployment': True, 'isAttached': True, 'serialNumber': '9CB27ZKT24E', 'is_deploy': True, 'freeformConfig': '', 'instanceValues': '{"loopbackId":"","loopbackIpAddress":"","loopbackIpV6Address":"","switchRouteTargetImportEvpn":"","switchRouteTargetExportEvpn":""}'}, {'import_evpn_rt': '', 'export_evpn_rt': '', 'vrf_lite': None, 'extensionValues': '', 'fabric': 'VXLAN-FABRIC-1', 'vrfName': 'BLUE', 'vlan': 101, 'deployment': True, 'isAttached': True, 'serialNumber': '9QSL59WP7O6', 'is_deploy': True, 'freeformConfig': '', 'instanceValues': '{"loopbackId":"","loopbackIpAddress":"","loopbackIpV6Address":"","switchRouteTargetImportEvpn":"","switchRouteTargetExportEvpn":""}'}]

(Epdb) p have_a
[{'vrfName': 'BLUE', 'peerSerialNo': None, 'instanceValues': '', 'entityName': 'BLUE', 'fabric': 'VXLAN-FABRIC-1', 'vlan': 101, 'serialNumber': '9CB27ZKT24E', 'deployment': True, 'extensionValues': '', 'isAttached': True, 'is_deploy': True, 'freeformConfig': ''}, {'vrfName': 'BLUE', 'peerSerialNo': None, 'instanceValues': '', 'entityName': 'BLUE', 'fabric': 'VXLAN-FABRIC-1', 'vlan': 101, 'serialNumber': '9QSL59WP7O6', 'deployment': True, 'extensionValues': '', 'isAttached': True, 'is_deploy': True, 'freeformConfig': ''}]

(Epdb) p have
{'vrfName': 'BLUE', 'peerSerialNo': None, 'instanceValues': '', 'entityName': 'BLUE', 'fabric': 'VXLAN-FABRIC-1', 'vlan': 101, 'serialNumber': '9CB27ZKT24E', 'deployment': True, 'extensionValues': '', 'isAttached': True, 'is_deploy': True, 'freeformConfig': ''}

(Epdb) p want["instanceValues"]
'{"loopbackId":"","loopbackIpAddress":"","loopbackIpV6Address":"","switchRouteTargetImportEvpn":"","switchRouteTargetExportEvpn":""}'

(Epdb) p have["instanceValues"]
''

(Epdb) n
-> have_inst_values = ast.literal_eval(have["instanceValues"])
(Epdb) n
SyntaxError: invalid syntax

[mikewiebe]

Thanks for the additional analysis @marehler