Potential data loss - Some postgres containers are silently omitted

[mspanc]

I have found out that rancher-backup is silently omitting some of my postgres containers

2018-01-30 00:00:10,342 - INFO - Found 'a-vault/postgres' to do dumping
2018-01-30 00:00:10,355 - INFO - Found 'b-vault/postgres' to do dumping
2018-01-30 00:00:10,367 - INFO - Found 'c-vault/postgres' to do dumping
2018-01-30 00:00:10,380 - INFO - Found 'd-vault/postgres' to do dumping
2018-01-30 00:00:10,395 - INFO - Found 'e-vault/postgres' to do dumping
2018-01-30 00:00:10,411 - INFO - Found 'f-vault/postgres' to do dumping
2018-01-30 00:00:10,428 - INFO - Found 'g-vault/postgres' to do dumping
2018-01-30 00:00:10,439 - INFO - Found 'h-vault/postgres' to do dumping
2018-01-30 00:00:10,452 - INFO - Found 'i-journal/postgres' to do dumping

There are certainly missing postgres containers, they are set up using the same template as all listed above.

I am using image webcenter/rancher-backup:2.0.1-1 with rancher 1.6.10.

My suspicion is that you're allowing cattle API to return paginated results. From my other projects I know that if you don't add limit=0 to the URL being called, you will not get full results. Default pagination limit is set quite high so unless you're running large Rancher instance you can not notice that behaviour.

It seems that you construct the URL to get services in the following way

 rancherService = Rancher(settings['rancher']['api']['url'][:-2] + "v2-beta", settings['rancher']['api']['key'], settings['rancher']['api']['secret'])

I roughly reviewed all of the underlying code and I hadn't seen any place where pagination is applied.

[disaster37]

hi,

under the wook, we use https://github.com/rancher/gdapi-python. There are no mention about pagination when get datas.

Can you look your service/container that are not backuped in log:
INFO - Found '...' to do dumpin

Can you past the dockerfile about container that not backuped (replace with XXX sensible data like password)

In ultime way you can put rancher-backup on debug mode via Environment variable DEBUG=true and attach the result. Don't forget to indicate what service is not backuped.

[mspanc]

I know there's no pagination in the docs, but I encountered this personally when I was writing another service using Rancher API. I had the same issue - a large amount of services that were truncated by default.

Certain services are not found tor sure and they do not appear in log "INFO - Found ... to do dumpin".

The container not being backuped is just official postgres:9.6.6. It is the same as other that are backuped.

Please believe me it is an pagination issue as I went through similar issue once.

If I go to my rancher and open https://rancher.myapp.com/v2-beta/containers?limit=0 I just get way more results than if I access https://rancher.myapp.com/v2-beta/containers. (I have hundreds of services in the cluster).

[rtyshyk]

Agree that it's pagination issue, have the same... only 100 first services are found and backuped.

[rtyshyk]

I tried to fix.. but no luck, even traversing paginator while listServices.pagination['next'] != None I have no all service as well as limit=0.
https://rancher.myapp.com/v2-beta/containers?limit=0 -> 232 items through the browser, but 108 using python client...
@mspanc did you fixed it somehow?

[mspanc]

@rtyshyk nope, I moved away from Rancher in my projects, I found it to be too unstable for production