Excessive memory usage from ytdl-sub

[LECOQQ]

Hey,

I've been playing w/ ytdl-sub for some days now, on my homelab, w/ a proxmox hypervisor on it, running ubuntu lts 24.04 as a vm containerizing ytdl-sub w/ docker compose.

I've noticed that when downloading videos, the RAM usage of my docker container (and thus my node, and thus my proxmox) will go up for each video downloaded. The thing is, it will not stop going up till it overflow from what is physically present in my homelab (32 G). In doing so, it will crash, killing my node and needing me to restart it manually, then restart the docker and the download.
I've found a way to circumvent the issue by fixing a limit on my container (16 G for instance). When downloading, it will do the same behavior then at 16G it will not go up and the downloads will keep going.
After the downloads, even after killing the docker container, my proxmox node will stick show that i'm currently using the total amount of RAM that I limited earlier (16G+).

It doesn't seems right to me, and this might be a RAM management issue inside the docker. I think that this should be natively taken into account, or atleast documented somewhere, though I don't know if it's a Proxmox issue or not.

Quentin

[jmbannon]

Can you post your config/subscription @LECOQQ ? I know yt-dlp can sometimes eat quite a bit of memory when scraping large channels for the first time

[Yankees4life]

I don't know why but it seems like the Linux version of ytdlp has this nasty memory leak. It certainly doesn't act this way when I'm using windows version of ytdlp. Might be an issue the developers may need to sort on their end...

[LECOQQ]

Sorry for the late reply.
I've added the docker compose. For the subscription configuration, nothing fancy as I've had troubles dealing w/ it. I've taken the generic conf provided and changed the url to meet a channel I wanted to download.
I've tried it on a ubuntu computer and on a ubuntu vm running through a proxmox hypervisor on my homelab. Both have this memory leak that increases after downloading a video.

ytdl-sub-docker-compose.txt

[mikemilligram]

Having the same issue (I think)
I haven't looked into it all that much, but all I know is since adding the container to my docker VM on proxmox, it keeps crashing my node. Had to remove it for now unfortunately, but I'd love to get this resolved!

[jmbannon]

I think what I'll try to do is chunk downloads in the background, then garbage collect to avoid any yt-dlp mem leaks.

In the meantime, isn't there way to limit containers' memory? I suggest giving ytdl-sub only 4gb RAM max. This person runs ytdl-sub in a container with only 1gb of memory: #1051

[LECOQQ]

Yes there is indeed a way. It is in the docker compose i've shared earlier.
We need to put the lines below into the docker compose file:

deploy:
  resources:
    limits:
      cpus: '4'
      memory: 16g

This is working fine for me after those modifications.