Give users SSH access to builders #6

FrostyX · 2023-09-16T15:56:59Z

xsuchy · 2023-09-16T16:30:42Z

user-ssh-builders/README.md

+Obviously, Copr resources aren't unlimited so we need to allocate them
+wisely.
+
+- Only 1 running instance per user


This number should be configurable. I can imagine that for internal Copr we can make this number higher.

Also, if we allow Packit to provide this to its users (in the future), all the Packit users will share this limit. So, I would have a configurable instance limit and, ideally, a user-level limit as well.

xsuchy · 2023-09-16T16:33:19Z

user-ssh-builders/README.md

+rules, instance type selection, etc., and a button to actually spawn
+the instance. The instance would be fresh and it would be up to the
+user to reproduce the failure manually. Once the builder spawns, the
+page will show instructions on how to connect to it.


I like this one. "Resubmit and keep for ssh"

~~I like this one as well.~~ Edit: changed my mind. The consequence is that we can't guarantee the same kind of VM (in AWS or on HW?).

xsuchy · 2023-09-16T16:35:37Z

user-ssh-builders/README.md

+able to specify at least an architecture. But that is IMHO not good
+enough. For example we have x86_64 builders our own hypervisors and in
+Amazon AWS. Historically, we had issues that appeared only on builders
+from one of them.


I do not care about issues that happens only in some pool. It may happen, but I guess this is very rare. Let them choose only architecture. Or to be precise one chroot.

xsuchy · 2023-09-16T16:37:09Z

user-ssh-builders/README.md

+but try it in an anonymous window, it doesn't work.
+
+Anyway, Copr already fetches some user information from FAS so I think
+we can assume we will be able to fetch the SSH keys as well.


+1
And if it means that they need to logoff and login to make this feature start working, that it is cheap price.

I requested this against noggin/fasjson: fedora-infra/fasjson#580

xsuchy · 2023-09-16T16:38:43Z

user-ssh-builders/README.md

+
+- It is not possible to override a status of an already finished build
+- Users cannot take a build from the queue (maybe not even theirs) and
+  process it


nod we will know at the beginnig that this build fail and we will give ssh to user. In this case we have to ignore the result.

praiskup · 2023-09-25T13:16:27Z

user-ssh-builders/README.md

+      to 30 hours at most. That means the lifespan needs to be at
+      least 30 hours.
+    - I propose 48 hours because we want to give users enough time for
+      debugging after the build finishes


I would give 1/2 or 1 hour max by default, with the possibility to prolong to 2 days as you propose.

+1 for having a really limited lifespan by default. Most of the people won't need much. Half an hour looks fine to me. Not sure if you have a median of the build time or some other metric we can compare this to.

praiskup · 2023-09-25T13:16:59Z

user-ssh-builders/README.md

+      debugging after the build finishes
+    - Alternatively, we can do a 24-hours lifespan and add an "extend"
+      button to Copr WebUI or to add a `copr-builder-extend` command
+      to the instance.


Ah, here -> but I think 24 hours by default is too much.

praiskup · 2023-09-25T13:21:34Z

user-ssh-builders/README.md

+  instance is already running. We don't want to kill it and spawn a
+  new one. We probably need to print it was not possible to spawn a
+  new instance into the builder-live/backend log
+- Where to print the instructions on how to connect to the builder?


The backend.log (backend is responsible for (not)closing the resalloc ticket)?

praiskup · 2023-09-25T13:23:31Z

user-ssh-builders/README.md

+  new instance into the builder-live/backend log
+- Where to print the instructions on how to connect to the builder?
+- If the build fails in multiple chroots, which one of those builders
+  should be kept alive? We can't keep them all.


If we allow user to have N such machines, the first N will be kept -> the rest of them would just dump to backend log that "You already have N machines ready for debugging, please release them first.".

praiskup · 2023-09-25T13:25:02Z

user-ssh-builders/README.md

+  should be kept alive? We can't keep them all.
+- Inability to select an instance type in advance so it may be hard
+  to debug builds where e.g. Amazon AWS Spot instance is needed
+  and nothing else.


Hmm, we have the exact failing machine in hand, what am I missing?

praiskup · 2023-09-25T13:28:10Z

user-ssh-builders/README.md

+
+Since the workflow starts from a failed build, we could create a MOTD
+on the builder showing instructions on how to reproduce the particular
+build using `copr-rpmbuild`.


I faced many times a "why the heck the build succeeds??" situation, too. Could we have:

--keep-builder-for-debugging=WHEN (failed|succeeded defaults to failed)

praiskup · 2023-09-25T13:40:32Z

user-ssh-builders/README.md

+
+Alternativelly, we could allow users to specify a public URL to their
+SSH key, upload it, or copy-paste it to some Copr form, but I don't
+like any of these options.


Agreed. We should also observe possibilities for LDAP for the Red Hat Internal Copr. I think we could make a compromise here if the automatic getting of pub keys is too complicated.

praiskup · 2023-09-25T13:42:05Z

user-ssh-builders/README.md

+could change the password after logging-in, upload their SSH key,
+whatever they want.
+
+Personally, I prefer `Option 2 - Passwords`.


It would have to be stored in some private per-user log, or in *_private database table :-/ I personally prefer Option 1.

True, I didn't think of that.
Agreed that we should go with Option 1 if it is doable.
👍

praiskup · 2023-09-25T13:45:34Z

user-ssh-builders/README.md

+
+We need to make sure that:
+
+- It is not possible to override a status of an already finished build


Can you elaborate? This seems to be "for free"... (either we started a new VM to reproduce previous failure, or we get the machine post-build - depending on the chosen approach).

praiskup · 2023-09-25T13:46:27Z

user-ssh-builders/README.md

+Personally, I prefer `Option 2 - Passwords`.
+
+
+## Security


We should make sure that the resalloc ticket ID is recycled immediately after user finished the debugging.

Hm, actually no - resalloc may re-use the resource with another ticket; the simple solution is to use a randomly generated unique --sandbox while taking the ticket.

praiskup · 2023-09-25T13:47:14Z

Awesome proposal, thank you @FrostyX moving this forward!

lachmanfrantisek

This looks really handy!

As mentioned in today's meeting, it might be worth syncing with the Testing Farm folks since they have a very similar feature.

lachmanfrantisek · 2023-10-16T11:59:01Z

user-ssh-builders/README.md

+Obviously, Copr resources aren't unlimited so we need to allocate them
+wisely.
+
+- Only 1 running instance per user


Also, if we allow Packit to provide this to its users (in the future), all the Packit users will share this limit. So, I would have a configurable instance limit and, ideally, a user-level limit as well.

lachmanfrantisek · 2023-10-16T12:00:40Z

user-ssh-builders/README.md

+      to 30 hours at most. That means the lifespan needs to be at
+      least 30 hours.
+    - I propose 48 hours because we want to give users enough time for
+      debugging after the build finishes


+1 for having a really limited lifespan by default. Most of the people won't need much. Half an hour looks fine to me. Not sure if you have a median of the build time or some other metric we can compare this to.

lachmanfrantisek · 2023-10-16T12:07:57Z

user-ssh-builders/README.md

+Alternativelly, we could allow users to specify a public URL to their
+SSH key, upload it, or copy-paste it to some Copr form, but I don't
+like any of these options.


Being able to set custom SSH key(s) would allow Packit to provide this to its users. We can easily let either people configure the public key or take it from GitHub. The workflow I see here:

Someone opens a PR.

Build is triggered.

Build fails, in the results (or as a comment), Packit suggests the user to try this feature.

User triggers this (e.g. via /packit rerun-with-ssh or some other comment)

Packit uses the Copr Python client to use this feature and set the user's public SSH key (it should be available via GitHub API).

Packit adds a comment with the instructions on how to connect.

TomasTomecek · 2023-10-17T19:08:27Z

user-ssh-builders/README.md

+config file on copr-frontend.
+
+
+## Auth


I wonder if PAM plugin (sssd?) would work to auth using kerberos: we use it both in Fedora and internally.

TomasTomecek · 2023-10-17T19:09:36Z

user-ssh-builders/README.md

+internal Copr by security initiatives.
+
+
+## Security


do you store some secrets on builders?

Fix fedora-copr#2364 See fedora-copr/debate#6

Fix #2364 See fedora-copr/debate#6

FrostyX changed the title ~~Allow users to SSH access builders~~ Give users SSH access to builders Sep 16, 2023

Allow users to SSH access builders

f7b2d2b

See fedora-copr/copr#2364

FrostyX force-pushed the obtain-builders branch from 6eff144 to f7b2d2b Compare September 16, 2023 15:59

xsuchy reviewed Sep 16, 2023

View reviewed changes

praiskup reviewed Sep 25, 2023

View reviewed changes

praiskup mentioned this pull request Sep 25, 2023

Ability to fetch FAS user's pub key? fedora-infra/fasjson#580

Open

praiskup reviewed Sep 25, 2023

View reviewed changes

praiskup mentioned this pull request Sep 25, 2023

Debugging for users - obtain a builder, and give the user SSH there fedora-copr/copr#2364

Closed

Follow-up after team meeting

371a433

FrostyX force-pushed the obtain-builders branch from 08dc1de to 371a433 Compare October 11, 2023 14:26

lachmanfrantisek reviewed Oct 16, 2023

View reviewed changes

TomasTomecek reviewed Oct 17, 2023

View reviewed changes

user-ssh-builders/README.md

internal Copr by security initiatives.

## Security

Copy link

TomasTomecek Oct 17, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do you store some secrets on builders?

FrostyX merged commit 6650cd1 into fedora-copr:main Nov 2, 2023

FrostyX mentioned this pull request Nov 2, 2023

Rebuild and allow user SSH fedora-copr/copr#2977

Merged

FrostyX added a commit to FrostyX/copr that referenced this pull request Dec 21, 2023

frontend, backend, common, rpmbuild: allow user SSH to builders

9fa616e

Fix fedora-copr#2364 See fedora-copr/debate#6

FrostyX added a commit to FrostyX/copr that referenced this pull request Jan 2, 2024

frontend, backend, common, rpmbuild: allow user SSH to builders

88d3be7

Fix fedora-copr#2364 See fedora-copr/debate#6

FrostyX added a commit to FrostyX/copr that referenced this pull request Jan 2, 2024

frontend, backend, common, rpmbuild: allow user SSH to builders

3a3e38d

Fix fedora-copr#2364 See fedora-copr/debate#6

FrostyX added a commit to FrostyX/copr that referenced this pull request Jan 2, 2024

frontend, backend, common, rpmbuild: allow user SSH to builders

2927c4a

Fix fedora-copr#2364 See fedora-copr/debate#6

FrostyX added a commit to FrostyX/copr that referenced this pull request Jan 2, 2024

frontend, backend, common, rpmbuild: allow user SSH to builders

36c6c3a

Fix fedora-copr#2364 See fedora-copr/debate#6

FrostyX added a commit to FrostyX/copr that referenced this pull request Jan 18, 2024

frontend, backend, common, rpmbuild: allow user SSH to builders

3747699

Fix fedora-copr#2364 See fedora-copr/debate#6

FrostyX added a commit to FrostyX/copr that referenced this pull request Jan 26, 2024

frontend, backend, common, rpmbuild: allow user SSH to builders

c5d0ca5

Fix fedora-copr#2364 See fedora-copr/debate#6

FrostyX added a commit to fedora-copr/copr that referenced this pull request Jan 28, 2024

frontend, backend, common, rpmbuild: allow user SSH to builders

c11e59f

Fix #2364 See fedora-copr/debate#6

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Give users SSH access to builders #6

Give users SSH access to builders #6

FrostyX commented Sep 16, 2023

xsuchy Sep 16, 2023

lachmanfrantisek Oct 16, 2023

xsuchy Sep 16, 2023

praiskup Sep 25, 2023 •

edited

Loading

xsuchy Sep 16, 2023

xsuchy Sep 16, 2023

praiskup Sep 25, 2023

FrostyX Oct 11, 2023

xsuchy Sep 16, 2023

praiskup Sep 25, 2023

lachmanfrantisek Oct 16, 2023

praiskup Sep 25, 2023

praiskup Sep 25, 2023

praiskup Sep 25, 2023

praiskup Sep 25, 2023

praiskup Sep 25, 2023

praiskup Sep 25, 2023

praiskup Sep 25, 2023

FrostyX Oct 11, 2023

praiskup Sep 25, 2023

praiskup Sep 25, 2023

praiskup Sep 25, 2023

praiskup commented Sep 25, 2023

lachmanfrantisek left a comment •

edited

Loading

lachmanfrantisek Oct 16, 2023

lachmanfrantisek Oct 16, 2023

lachmanfrantisek Oct 16, 2023

TomasTomecek Oct 17, 2023

TomasTomecek Oct 17, 2023


		We need to make sure that:

		- It is not possible to override a status of an already finished build

Give users SSH access to builders #6

Give users SSH access to builders #6

Conversation

FrostyX commented Sep 16, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

praiskup Sep 25, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

praiskup commented Sep 25, 2023

lachmanfrantisek left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

praiskup Sep 25, 2023 •

edited

Loading

lachmanfrantisek left a comment •

edited

Loading