abc-clone doesn't provide a useful fail message & fails quietly when the ssh key doesnt have the correct permissions

[lwasser]

When you run abc-clone and things don't work you have no idea why. i just ran this on the hub and got the following. i have no idea why it didn't clone because those repos all do exist.

jovyan@jupyter-earth-2dlab:~/fall-2020-abc-ea-bootcamp$ abc-clone bootcamp-2020-08-loops
Loading configuration from config.yml
Could not clone or update the following repos:
 bootcamp-2020-08-loops-adamancer
 bootcamp-2020-08-loops-amap989
 bootcamp-2020-08-loops-bri33

the same command works on my computer locally. could this be an ssh issue? if so why doesn't it tell me why it failed.

and i just also thought about the fact that the user may not have permissions on the hub - that could be why but regardless i think it's important to provide a useful message when things don't work

---

i spent some time looking at abc-clone and i see many points of failure here in addition to the above

there are no tests and checks to ensure

• that the roster exists
• that the repo exists
• when clone doesn't work you have no idea why
• if ssh or whatever authentication we plan to setup is correct

we should refactor this function to ensure there are checks throughout and then tests that make sure it works as we think it does. the clone failing quietly is a bit frustrating because i'm not sure what to work on to fix it. we need really good user messages to guide ...

[lwasser]

nope - not a permissions issue i think as i made the account an owner... will have to keep troubleshooting this

i tried this
ssh -T [email protected]

to test my ssh connection and i got this:

jovyan@jupyter-earth-2dlab:~$ ssh -T [email protected]
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Permissions 0660 for '/home/jovyan/.ssh/id_rsa' are too open.
It is required that your private key files are NOT accessible by others.
This private key will be ignored.
Load key "/home/jovyan/.ssh/id_rsa": bad permissions
[email protected]: Permission denied (publickey).

making progress - i'll have to add this to the hub docs... there are a lot of quirks associated with setting upp ssh on the hub including using VI to copy the ssh key and also the permissions for the key itself

chmod 400 /home/jovyan/.ssh/id_rsa

fixed it

found the solution here: https://stackoverflow.com/questions/29933918/ssh-key-permissions-0644-for-id-rsa-pub-are-too-open-on-mac

[lwasser]

ok great this seems to be working now... so the take away here is that we really need to have more tests setup to ensure things are working as we need them to. the messages a user gets are not helpful and there arent' enough checks right now.

[lwasser]

UPDATE: this worked temporarily but just not i had to rerun chmod to change permissions so i could use abc-clone on the hub. this is odd as why would permissions change like that after logging out?

[lwasser]

@kcranston here is my issue - in a weird place btu related to why abc-clone fails quietly sometimes. the permissions. change using chmod does fix the issue but i periodically have to run it again.

[kcranston]

There was some discussion about this on jupyter gitter a while back. Others have had the same problem. No solution proposed. Will keep digging.

[lwasser]

Ahhh eric has helped us a big – in that thread he suggests a lifecycle hook that applies the permissions? Maybe that is the solution? I’m not exactly sure what that looks like but it would be good to create a hub that h as much less setup for nbgrader. So in other words setting up some of the nbgrader stuff in the docker image ahead of time and maybe like gitpuller, looking for that file to ensure permissions are correct upon pod launch?

[lwasser]

i am going to bump up this issue as a future milestone item. Abc-clone works in the notebook. but notice below that it fails without a helpful message if it can't access ssh! It should ask the user to setup ssh.

This is also a problem on the hub because of the chmod issue but we can build that into the notebook for the time being

[chanelrichardson]

Has any solution been found to this issue?

[lwasser]

hey @ccthegreat313 what a great username! are you having specific issues. we did address this and closed the issue back in march. i think you should get better messaging when things fail.

[eculler]

I am getting this same message: Could not clone or update the following repos

..but only when I try to use abc-clone to update repos.

It works to download the first time, and the ssh key file permissions seem to be fine (ssh -T [email protected] says it authenticates) It's not clear to me why abc-clone can't pull from github; when I pull individual assignments using git it works fine and doesn't require additional authentication.

I also tried updating to the development branch, with no change.

[kcranston]

Do you get the error for all of the student repos, or only a subset?

[lwasser]

this may be a different issue. @eculler can you print out what is writes below "Could not clone or update the following repos?" so please add the entire error message that you get here. i think we need to identify in this message what the point of failure is because it could be coming from multiple locations.

[lwasser]

heya @kcranston elsa can provide more detail but it's definitely doing this for ALL of the students not just a subset.
And @eculler is also able to push feedback to the repos so she has already pushed feedback but now needs to update a few repos locally.
https://github.com/earthlab/abc-classroom/blob/main/abcclassroom/clone.py#L169

[eculler]

I get the error for all of the repos. However, if I remove a cloned repo, it successfully clones it again - it’s only the pull that doesn’t work.

…

On Thu, Sep 30, 2021 at 10:42 AM Karen Cranston ***@***.***> wrote: Do you get the error for all of the student repos, or only a subset? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#315 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA2OEKHE4N7KAEAQZVFFMJDUESHO5ANCNFSM4S2KP32A> .

[lwasser]

ok great - @eculler another question i'm seeing skip_existing in there as a boolean . i could be really off here but what happens if you set that to True ? we definitely need better messaging here. I just am noticing that that flag has no default value from what i can see in the function. i could be totally off base i'm just curious if adding that flag helps or does nothing.

https://github.com/earthlab/abc-classroom/blob/main/abcclassroom/clone.py#L35

[lwasser]

it would be helpful to have a message in that function that tells the user what is happening like "That repo already exists, i'm pulling updates now" ... it would help us better identify where the point of failure is too!

[lwasser]

pinging @nkorinek once you are done with the lessons updates you may be able to work with karen on this.

[eculler]

hm, it’s not accepting an explicit true argument when I tried —skip-existing=True?

…

On Thu, Sep 30, 2021 at 12:47 PM Leah Wasser ***@***.***> wrote: ok great - @eculler <https://github.com/eculler> another question i'm seeing skip_existing in there as a boolean . i could be really off here but what happens if you set that to True ? we definitely need better messaging here. I just am noticing that that flag has no default value from what i can see in the function. i could be totally off base i'm just curious if adding that flag helps or does nothing. https://github.com/earthlab/abc-classroom/blob/main/abcclassroom/clone.py#L35 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#315 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA2OEKEGBQGSZGPAQMSZR23UESWDVANCNFSM4S2KP32A> .

[kcranston]

The syntax is just --skip-existing when you want to skip (i.e. True) and no flag for false.

It would be helpful to see the whole output from abc-clone. The earlier fix for this issue added output for various fail states, so I am curious to see where we might be missing something.

[lwasser]

ok update on what we just discussed in our dev meeting! @eculler @nkorinek is going to suggest that you install this package from github. we realized that we haven't done a release since Feb 2021 yet we've done a good bit of development work during that time. I'm hoping we may have fixed this but we want to test that first. if it's still buggy we will dig deeper into what is going on!

[lwasser]

What needs to happen here is we need more useful feedback around git and github calls when they fail. there are probably multiple points of failure here but the error output is the same. Having better feedback would allow us and the user to better solve this problem.

So this issue needs to be addressed but we may not get there by march 2022.

[lwasser]

this relates to #444 . it would be ideal if someone could use verbose setting to provide more feedback which would help with troubleshooting and maintenance. So this is a bigger issue.