Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

A lot of reconfigurations cause confd to crash #861

Open
mattiaswal opened this issue Dec 9, 2024 · 2 comments
Open

A lot of reconfigurations cause confd to crash #861

mattiaswal opened this issue Dec 9, 2024 · 2 comments
Assignees
@mattiaswal
Labels
bug Something isn't working
Milestone
Infix v25.02

Comments

@mattiaswal
Copy link
Contributor

mattiaswal commented Dec 9, 2024 •
edited
Loading

Current Behavior

Running a test in a loop result i confd failure, after 251 loops:

e=0; while [ $? -eq 0  ]; do e=$((e+1)); echo "Loop $e"; ./case/ietf_system/hostname/test.py -t restconf; done

RESTCONF/NETCONF doesn't matter, it is the same behavior with both.

Dec  9 10:21:15 ix-00-02-00 rousette[17872]: [2024-12-09 10:21:15.823] [rousette] [info] [::1]:50156: POST /restconf/data/infix-test:test/reset 
Dec  9 10:21:15 ix-00-02-00 confd[17849]: libyang[1]: Obsolete schema node "meta" instantiated in data.
Dec  9 10:21:15 ix-00-02-00 confd[17849]: Recursive reader limit 255 reached, possibly because of missing unlocks.
Dec  9 10:21:15 ix-00-02-00 confd[17849]: failed sr_subscription_process_events(), ret:7
Dec  9 10:21:15 ix-00-02-00 finit[1]: Stopping netopeer[17858], sending SIGTERM ...
Dec  9 10:21:15 ix-00-02-00 finit[1]: Stopping statd[17868], sending SIGTERM ...
Dec  9 10:21:15 ix-00-02-00 finit[1]: Stopping rousette[17872], sending SIGTERM ...
Dec  9 10:21:15 ix-00-02-00 finit[1]: Service confd[17849] died, restarting in 2000 msec (1/10)
Dec  9 10:21:15 ix-00-02-00 finit[1]: Starting confd[6308]
Dec  9 10:21:15 ix-00-02-00 sysrepo-plugind[6308]: Connection with CID 10 is dead.
Dec  9 10:21:15 ix-00-02-00 sysrepo-plugind[6308]: Recovering module "sysrepo-plugind" running change subscription of CID 10.
Dec  9 10:21:15 ix-00-02-00 last message buffered 1 times
Dec  9 10:21:15 ix-00-02-00 sysrepo-plugind[6308]: Recovering module "sysrepo-plugind" operational get subscription of CID 10.
Dec  9 10:21:15 ix-00-02-00 confd[6308]: Recovering module "ietf-interfaces" running change subscription of CID 10.
Dec  9 10:21:15 ix-00-02-00 last message buffered 2 times
Dec  9 10:21:15 ix-00-02-00 confd[6308]: Recovering module "ietf-interfaces" startup change subscription of CID 10.
Dec  9 10:21:15 ix-00-02-00 confd[6308]: Recovering module "ietf-interfaces" candidate change subscription of CID 10.
Dec  9 10:21:15 ix-00-02-00 confd[6308]: Recovered a read-lock of CID 10 (sr_module_change_subscribe).
Dec  9 10:21:15 ix-00-02-00 confd[6308]: Recovering module "ietf-keystore" running change subscription of CID 10.
Dec  9 10:21:15 ix-00-02-00 last message buffered 3 times
Dec  9 10:21:15 ix-00-02-00 confd[6308]: ietf_keystore_init failed: Operation succeeded
Dec  9 10:21:15 ix-00-02-00 confd[6308]: Recovering module "ietf-syslog" running change subscription of CID 10.
Dec  9 10:21:15 ix-00-02-00 last message buffered 11 times
Dec  9 10:21:15 ix-00-02-00 confd[6308]: Recovering module "ietf-syslog" startup change subscription of CID 10.
Dec  9 10:21:15 ix-00-02-00 last message buffered 3 times
Dec  9 10:21:15 ix-00-02-00 confd[6308]: Recovered a read-lock of CID 10 (sr_module_change_subscribe).
Dec  9 10:21:15 ix-00-02-00 confd[6308]: Recovering module "ietf-system" running change subscription of CID 10.
Dec  9 10:21:15 ix-00-02-00 last message buffered 22 times
Dec  9 10:21:15 ix-00-02-00 confd[6308]: Connection with CID 12 is dead.
Dec  9 10:21:15 ix-00-02-00 confd[6308]: Recovering module "ietf-system" operational get subscription of CID 12.
Dec  9 10:21:15 ix-00-02-00 confd[6308]: Recovering module "ietf-system" operational get subscription of CID 10.
Dec  9 10:21:15 ix-00-02-00 last message buffered 4 times
Dec  9 10:21:15 ix-00-02-00 confd[6308]: Recovering module "ietf-netconf-acm" running change subscription of CID 10.
Dec  9 10:21:15 ix-00-02-00 confd[6308]: Recovering module "ietf-system" startup change subscription of CID 10.
Dec  9 10:21:15 ix-00-02-00 last message buffered 6 times
Dec  9 10:21:15 ix-00-02-00 confd[6308]: Recovering RPC/action "/ietf-system:system-restart" subscription of CID 10.
Dec  9 10:21:15 ix-00-02-00 confd[6308]: Recovering RPC/action "/ietf-system:system-shutdown" subscription of CID 10.
Dec  9 10:21:15 ix-00-02-00 confd[6308]: Recovering RPC/action "/ietf-system:set-current-datetime" subscription of CID 10.
Dec  9 10:21:15 ix-00-02-00 confd[6308]: Recovering module "infix-dhcp-client" running change subscription of CID 10.
Dec  9 10:21:15 ix-00-02-00 last message buffered 2 times
Dec  9 10:21:15 ix-00-02-00 confd[6308]: Recovering module "infix-dhcp-client" startup change subscription of CID 10.
Dec  9 10:21:15 ix-00-02-00 confd[6308]: Recovering module "infix-dhcp-client" candidate change subscription of CID 10.
Dec  9 10:21:15 ix-00-02-00 confd[6308]: Recovering RPC/action "/infix-factory-default:factory-default" subscription of CID 10.
Dec  9 10:21:15 ix-00-02-00 confd[6308]: Recovering RPC/action "/ietf-factory-default:factory-reset" subscription of CID 10.
Dec  9 10:21:15 ix-00-02-00 confd[6308]: Recovering module "ietf-routing" running change subscription of CID 10.
Dec  9 10:21:15 ix-00-02-00 last message buffered 2 times
Dec  9 10:21:15 ix-00-02-00 confd[6308]: Recovering module "ietf-routing" startup change subscription of CID 10.
Dec  9 10:21:15 ix-00-02-00 confd[6308]: Recovering module "infix-meta" running change subscription of CID 10.
Dec  9 10:21:15 ix-00-02-00 confd[6308]: Recovering RPC/action "/infix-system:install-bundle" subscription of CID 10.
Dec  9 10:21:15 ix-00-02-00 confd[6308]: Recovering module "infix-services" running change subscription of CID 10.
Dec  9 10:21:15 ix-00-02-00 last message buffered 17 times
Dec  9 10:21:15 ix-00-02-00 confd[6308]: Recovering module "infix-services" startup change subscription of CID 10.
Dec  9 10:21:15 ix-00-02-00 last message buffered 5 times
Dec  9 10:21:15 ix-00-02-00 confd[6308]: Recovering module "ieee802-dot1ab-lldp" running change subscription of CID 10.
Dec  9 10:21:15 ix-00-02-00 last message buffered 2 times
Dec  9 10:21:15 ix-00-02-00 confd[6308]: Recovering module "ieee802-dot1ab-lldp" startup change subscription of CID 10.
Dec  9 10:21:15 ix-00-02-00 confd[6308]: Recovering module "ietf-keystore" startup change subscription of CID 10.
Dec  9 10:21:15 ix-00-02-00 confd[6308]: Recovering module "ietf-hardware" running change subscription of CID 10.
Dec  9 10:21:15 ix-00-02-00 last message buffered 2 times
Dec  9 10:21:15 ix-00-02-00 confd[6308]: Recovering module "ietf-hardware" startup change subscription of CID 10.
Dec  9 10:21:15 ix-00-02-00 confd[6308]: Recovering module "ietf-hardware" candidate change subscription of CID 10.
Dec  9 10:21:18 ix-00-02-00 finit[1]: Stopping rousette[17872], sending SIGKILL ...
@mattiaswal mattiaswal added bug Something isn't working triage Pending investigation & classification (CCB) labels Dec 9, 2024
@mattiaswal mattiaswal changed the title A lot of reconfigurations cause confd to crashes A lot of reconfigurations cause confd to crash Dec 9, 2024
@mattiaswal
Copy link
Contributor Author

This seems to be fixed with later CESNET packages, i am using latest (devel:2012-12-20) version of libyang,sysrepo,netopeer2, libnetconf2, rousette, libyang-cpp, sysrepo-cpp and the bug can not be seen.

@mattiaswal mattiaswal self-assigned this Dec 20, 2024
@mattiaswal
Copy link
Contributor Author

This seems to be fixed with later CESNET packages, i am using latest (devel:2012-12-20) version of libyang,sysrepo,netopeer2, libnetconf2, rousette, libyang-cpp, sysrepo-cpp and the bug can not be seen.

That is a lie, not fixed. I think this is something we are responsible to. Start checking that all sr_get_data objects
get freed correctly.

@troglobit troglobit added this to the Infix v25.02 milestone Jan 23, 2025
@troglobit troglobit removed the triage Pending investigation & classification (CCB) label Jan 23, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mattiaswal mattiaswal

Labels
bug Something isn't working
Projects
Infix & C:o
Status: No status
Milestone
Infix v25.02
Development

No branches or pull requests
2 participants
@troglobit @mattiaswal