6.0.28 now requires root for mongodb

[jtackaberry]

I run the controller as a non-root user, and this has been working fine since I originally deployed it earlier this year, including with 6.0.23, but as of 6.0.28, mongodb refuses to start:

2020-10-20T17:36:20.790+0000 I CONTROL  [main] ***** SERVER RESTARTED *****
2020-10-20T17:36:20.794+0000 I CONTROL  [main] ERROR: Cannot write pid file to /usr/lib/unifi/run/mongod.pid: Permission denied

/usr/lib/unifi/run is a symlink to /var/run/unifi.

On 6.0.23:

$ ls -ld /var/run/unifi
drwxrwsrwx 3 root 2000 4096 Oct 20 17:37 /var/run/unifi

And on 6.0.28:

$ ls -ld /var/run/unifi/
drwxr-xr-x 2 root root 4096 Oct 20 17:41 /var/run/unifi/

So this requires mongodb to run as root.

I am running the container under Kubernetes, with a securityContext stanza defining a non-root uid/gid:

      securityContext:
        runAsUser: 10011
        runAsGroup: 10011
        supplementalGroups: [10011]

[jacobalberty]

What cpu are you running under?

[jtackaberry]

linux/amd64

[jacobalberty]

I'm really not sure why it worked before but not now, but this is part of what I'm working on with #362 . If you are not married to those ids then then the userid 999 and groupid 999 are known to work (i'm using them right now)

Your permissions however do not line up with mine at all.

unifi@68389b0d1f1d:/unifi$ ls -ld /var/run/unifi
drwxr-x--- 3 unifi unifi 4096 Oct 20 18:05 /var/run/unifi

[jacobalberty]

Ok I have a theory on the permissions not lining up. /var/run/unifi is a volume. I think you may have an old volume that from the old image (new image uses distro mongodb old one uses the main mongodb packages) there must be a difference in those packages.

for docker using docker-compose i can clear all volumes with docker-compose down -v (and for trouble shooting i added a docker volume prune and docker system prune for good measure) I am not sure what the equivalent for kubernetes is

[jtackaberry]

My apologies for failing to mention it: /var/run/unifi is indeed a volume, but it's a k8s emptyDir which gets recreated each time the pod is deployed and so is fresh on each new invocation.

The confusing part for me is why /var/run/unifi isn't getting appropriately mounted with 6.0.28 while it works fine with 6.0.23. I'll have to dig in deeper to see what's happening there.

[SubtleMuffin]

Same issue here.. had to roll back to deb2b0ab ... 6.0.22

[jacobalberty]

Ok, once the pr in #375 finishes running on travis mongodbfix should be live on docker hub, give it a try if it fixes it then i'll merge that pr back in. Give it about 10 more minutes on the build.

[jacobalberty]

Alright, give jacobalberty/unifi:mongodbfix a try and let me know if that fixes it. It'll only apply to amd64 platforms if it does but I'm hoping to fully fix permissions outside of root in #373 anyway

[SubtleMuffin]

Not working. Returns 404.

[jacobalberty]

@SubtleMuffin the other option is jacobalberty/unifi:beta-noroot, you'll need to supply a PKGURL as an environment variable, https://dl.ui.com/unifi/6.0.28/unifi_sysvinit_all.deb is the url for 6.0.28 it will actively try to change ownership of the directory in question. It's a little further out from making it into master though.

The big caveat with beta-noroot is it does not work if the user does not exist in the passwd file right now, its something i'm still working on a way around.

[jtackaberry]

I'm afraid jacobalberty/unifi:mongodbfix exhibits the same behavior as 6.0.28. What's really baking my noodle is that with 6.0.28 I can't volume-mount anything at /run/unifi. I was using an emptyDir, and then I tried a PersistentVolumeClaim which is what I use for /unifi (which is also a volume specified in the Dockerfile, and which continues to work), but no dice.

I will need more time to dig further into this. There's a puzzle here. There's clearly something about how either the kubelet or containerd work that I don't understand.

@jacobalberty one question, in the Dockerfile you have:

    RUNDIR=/var/run/unifi \
    RUNDIR2=/usr/lib/unifi/run \
[...]
 VOLUME ["/unifi", "${RUNDIR}", "${RUNDIR2}"]

But /usr/lib/unifi/run symlinks to /var/run/unifi (and since /var/run symlinks to /run these both indirectly point to /run/unifi). What was the motivation for the recent addition of ${RUNDIR2} to the volume list, given the symlink?

[jtackaberry]

FWIW, removing "${RUNDIR2}" from the volumes list in the Dockerfile fixes the issue for me. Though it's not yet clear to me why.

[jacobalberty]

I'll have to go through the commit log and see when/why I added it. It'll be a few hours before I get a chance to. It's been there for a while iirc. Interesting that removing it fixes the issue. Sounds like the easiest fix then just need to check the log and see if there's some side effect to removing it. It's possible it just predates the symlink. Ubiquiti in the past has shuffled directories around and symlinks and volumes have been added to keep it working in a container.

…

On Wed, Oct 21, 2020, 9:49 AM Jason Tackaberry ***@***.***> wrote: FWIW, removing "${RUNDIR2}" from the volumes list in the Dockerfile fixes the issue for me. Though it's not yet clear to me why. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#371 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAPDNAU7EXEK7CCGGCT3PATSL3YG7ANCNFSM4SYVN2ZA> .

[jtackaberry]

I'll have to go through the commit log and see when/why I added it. It'll be a few hours before I get a chance to. It's been there for a while iirc.

The addition of ${RUNDIR2} was committed 7 days ago.

[jacobalberty]

Ohh I see now that's a recent addition. I was working on removing runas_uid0 that should have been committed into beta where I was stripping out the gosu stuff. I'm not sure why it got into master I'll remove it as soon as I get to a computer.

…

On Wed, Oct 21, 2020, 9:52 AM Jacob Alberty ***@***.***> wrote: I'll have to go through the commit log and see when/why I added it. It'll be a few hours before I get a chance to. It's been there for a while iirc. Interesting that removing it fixes the issue. Sounds like the easiest fix then just need to check the log and see if there's some side effect to removing it. It's possible it just predates the symlink. Ubiquiti in the past has shuffled directories around and symlinks and volumes have been added to keep it working in a container. On Wed, Oct 21, 2020, 9:49 AM Jason Tackaberry ***@***.***> wrote: > FWIW, removing "${RUNDIR2}" from the volumes list in the Dockerfile > fixes the issue for me. Though it's not yet clear to me why. > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <#371 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AAPDNAU7EXEK7CCGGCT3PATSL3YG7ANCNFSM4SYVN2ZA> > . >

[jacobalberty]

travis-ci is backed up a bit, the latest and stable-5 tags have that patch reverted now, i'm still waiting on 6.0.28 to build.

[jtackaberry]

I got notified that the new 6.0.28 build is up on DockerHub. Installed and it works. 👍

[jacobalberty]

Excellent. closing issue then.