Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

less.exe often hangs inside tmux as Git's pager when quitting and immediately typing something #4812

Closed
dscho opened this issue Feb 13, 2024 · 19 comments

Comments

@dscho
Copy link
Member

dscho commented Feb 13, 2024

This is a somewhat special scenario, yet I am worried that it can hit users in many other scenarios, too. The problem seems to be yet another deadlock in the MSYS2/Cygwin runtime.

Repro

Note: This repro is not consistent, it often works, but sometimes does not.

In a tmux session, I ran git diff/git log, which calls (MINGW) git.exe that then spawns (MSYS) less.exe. Then I press Q and immediately after that ⬆️. Now the process hangs. Actually, there are two processes for less, I assume one is the real one and the other one exists only to handle signals? The latter is the one hanging, and it has the following threads:

  Id   Target Id                     Frame
  1    Thread 32924.0x1354 "less"    0x00007ffe247afec4 in ntdll!ZwWaitForMultipleObjects ()
   from /c/WINDOWS/SYSTEM32/ntdll.dll
  2    Thread 32924.0x1608 "sig"     0x00007ffe247af434 in ntdll!ZwReadFile () from /c/WINDOWS/SYSTEM32/ntdll.dll
  3    Thread 32924.0x398c "consm"   0x00007ffe247afec4 in ntdll!ZwWaitForMultipleObjects ()
   from /c/WINDOWS/SYSTEM32/ntdll.dll
  4    Thread 32924.0x8d10 "conssel" 0x00007ffe247af3f4 in ntdll!ZwWaitForSingleObject ()
   from /c/WINDOWS/SYSTEM32/ntdll.dll
  5    Thread 32924.0x4fb0 "pipesel" 0x00007ffe247af3f4 in ntdll!ZwWaitForSingleObject ()
   from /c/WINDOWS/SYSTEM32/ntdll.dll
* 6    Thread 32924.0x861c           0x00007ffe247b3061 in ntdll!DbgBreakPoint () from /c/WINDOWS/SYSTEM32/ntdll.dll

Thread 6 is probably just a side effect of attaching with gdb. Here are the stacktraces:

Stacktrace of thread 1 ("less")
#0  0x00007ffe247afec4 in ntdll!ZwWaitForMultipleObjects () from /c/WINDOWS/SYSTEM32/ntdll.dll
(gdb) bt
#0  0x00007ffe247afec4 in ntdll!ZwWaitForMultipleObjects () from /c/WINDOWS/SYSTEM32/ntdll.dll
#1  0x00007ffe21c1f6f9 in WaitForMultipleObjectsEx () from /c/WINDOWS/System32/KERNELBASE.dll
#2  0x00007ffe21c1f5fe in WaitForMultipleObjects () from /c/WINDOWS/System32/KERNELBASE.dll
#3  0x0000000210045cd1 in cygwait (object=<optimized out>, timeout=timeout@entry=0x0, mask=mask@entry=5)
    at /usr/src/MSYS2-packages/msys2-runtime/src/msys2-runtime/winsup/cygwin/cygwait.cc:75
#4  0x000000021010d130 in cygwait (mask=5, howlong=4294967295, h=<optimized out>)
    at /usr/src/MSYS2-packages/msys2-runtime/src/msys2-runtime/winsup/cygwin/local_includes/cygwait.h:45
#5  cygwait (howlong=4294967295, h=<optimized out>)
    at /usr/src/MSYS2-packages/msys2-runtime/src/msys2-runtime/winsup/cygwin/local_includes/cygwait.h:51
#6  fhandler_console::read (this=0x800008dd8, pv=0x7ffffc9cf, buflen=@0x7ffffc950: 1)
    at /usr/src/MSYS2-packages/msys2-runtime/src/msys2-runtime/winsup/cygwin/fhandler/console.cc:1023
#7  0x00000002100d0aad in read (fd=3, ptr=0x7ffffc9cf, len=<optimized out>)
    at /usr/src/MSYS2-packages/msys2-runtime/src/msys2-runtime/winsup/cygwin/local_includes/dtable.h:64
#8  0x000000021019396b in _sigfe () from /usr/bin/msys-2.0.dll
#9  0x00000001004167b1 in ?? ()
#10 0x000000010041c869 in ?? ()
#11 0x0000000100407cc2 in ?? ()
#12 0x00000001004086bc in ?? ()
#13 0x0000000100408c05 in ?? ()
#14 0x000000010041e538 in ?? ()
#15 0x0000000210047ed1 in dll_crt0_1 ()
    at /usr/src/MSYS2-packages/msys2-runtime/src/msys2-runtime/winsup/cygwin/dcrt0.cc:1012
#16 0x0000000210045ac3 in _cygtls::call2 (this=0x7ffffce00, func=0x210046da0 <dll_crt0_1(void*)>, arg=0x0,
    buf=buf@entry=0x7ffffcdf0) at /usr/src/MSYS2-packages/msys2-runtime/src/msys2-runtime/winsup/cygwin/cygtls.cc:41
#17 0x0000000210045b74 in _cygtls::call (func=<optimized out>, arg=<optimized out>)
    at /usr/src/MSYS2-packages/msys2-runtime/src/msys2-runtime/winsup/cygwin/cygtls.cc:28
#18 0x0000000000000000 in ?? ()
Stacktrace of thread 2 ("sig")
#0  0x00007ffe247af434 in ntdll!ZwReadFile () from /c/WINDOWS/SYSTEM32/ntdll.dll
#1  0x00007ffe21be623b in ReadFile () from /c/WINDOWS/System32/KERNELBASE.dll
#2  0x00000002100c0081 in wait_sig ()
    at /usr/src/MSYS2-packages/msys2-runtime/src/msys2-runtime/winsup/cygwin/sigproc.cc:1331
#3  0x0000000210044980 in cygthread::callfunc (this=this@entry=0x2102318c0 <threads>,
    issimplestub=issimplestub@entry=false)
    at /usr/src/MSYS2-packages/msys2-runtime/src/msys2-runtime/winsup/cygwin/cygthread.cc:48
#4  0x0000000210044f19 in cygthread::stub (arg=arg@entry=0x2102318c0 <threads>)
    at /usr/src/MSYS2-packages/msys2-runtime/src/msys2-runtime/winsup/cygwin/cygthread.cc:91
#5  0x0000000210045ac3 in _cygtls::call2 (this=0x10bce00, func=0x210044eb0 <cygthread::stub(void*)>,
    arg=0x2102318c0 <threads>, buf=buf@entry=0x10bcd20)
    at /usr/src/MSYS2-packages/msys2-runtime/src/msys2-runtime/winsup/cygwin/cygtls.cc:41
#6  0x0000000210045b74 in _cygtls::call (func=<optimized out>, arg=<optimized out>)
    at /usr/src/MSYS2-packages/msys2-runtime/src/msys2-runtime/winsup/cygwin/cygtls.cc:28
#7  0x00007ffe2366257d in KERNEL32!BaseThreadInitThunk () from /c/WINDOWS/System32/KERNEL32.DLL
#8  0x00007ffe2476aa58 in ntdll!RtlUserThreadStart () from /c/WINDOWS/SYSTEM32/ntdll.dll
#9  0x0000000000000000 in ?? ()
Stacktrace of thread 3 ("consm")
#0  0x00007ffe247afec4 in ntdll!ZwWaitForMultipleObjects () from /c/WINDOWS/SYSTEM32/ntdll.dll
#1  0x00007ffe21c1f6f9 in WaitForMultipleObjectsEx () from /c/WINDOWS/System32/KERNELBASE.dll
#2  0x00007ffe21c1f5fe in WaitForMultipleObjects () from /c/WINDOWS/System32/KERNELBASE.dll
#3  0x0000000210045cd1 in cygwait (object=object@entry=0x0, timeout=timeout@entry=0x2d8caf8, mask=mask@entry=5)
    at /usr/src/MSYS2-packages/msys2-runtime/src/msys2-runtime/winsup/cygwin/cygwait.cc:75
#4  0x00000002100ff928 in cygwait (mask=5, howlong=40, h=0x0)
    at /usr/src/MSYS2-packages/msys2-runtime/src/msys2-runtime/winsup/cygwin/local_includes/cygwait.h:45
#5  cygwait (howlong=40, h=0x0)
    at /usr/src/MSYS2-packages/msys2-runtime/src/msys2-runtime/winsup/cygwin/local_includes/cygwait.h:51
#6  cygwait (howlong=40)
    at /usr/src/MSYS2-packages/msys2-runtime/src/msys2-runtime/winsup/cygwin/local_includes/cygwait.h:57
#7  fhandler_console::cons_master_thread (p=0x2d8cbb0, ttyp=0x1a3000000)
    at /usr/src/MSYS2-packages/msys2-runtime/src/msys2-runtime/winsup/cygwin/fhandler/console.cc:569
#8  0x000000021010c4da in cons_master_thread (arg=<optimized out>)
    at /usr/src/MSYS2-packages/msys2-runtime/src/msys2-runtime/winsup/cygwin/fhandler/console.cc:215
#9  0x0000000210044980 in cygthread::callfunc (this=this@entry=0x210231918 <threads+88>,
    issimplestub=issimplestub@entry=false)
    at /usr/src/MSYS2-packages/msys2-runtime/src/msys2-runtime/winsup/cygwin/cygthread.cc:48
#10 0x0000000210044f19 in cygthread::stub (arg=arg@entry=0x210231918 <threads+88>)
    at /usr/src/MSYS2-packages/msys2-runtime/src/msys2-runtime/winsup/cygwin/cygthread.cc:91
#11 0x0000000210045ac3 in _cygtls::call2 (this=0x2d8ce00, func=0x210044eb0 <cygthread::stub(void*)>,
    arg=0x210231918 <threads+88>, buf=buf@entry=0x2d8cd20)
    at /usr/src/MSYS2-packages/msys2-runtime/src/msys2-runtime/winsup/cygwin/cygtls.cc:41
#12 0x0000000210045b74 in _cygtls::call (func=<optimized out>, arg=<optimized out>)
    at /usr/src/MSYS2-packages/msys2-runtime/src/msys2-runtime/winsup/cygwin/cygtls.cc:28
#13 0x00007ffe2366257d in KERNEL32!BaseThreadInitThunk () from /c/WINDOWS/System32/KERNEL32.DLL
#14 0x00007ffe2476aa58 in ntdll!RtlUserThreadStart () from /c/WINDOWS/SYSTEM32/ntdll.dll
#15 0x0000000000000000 in ?? ()
Stacktrace of thread 4 ("conssel")
#0  0x00007ffe247af3f4 in ntdll!ZwWaitForSingleObject () from /c/WINDOWS/SYSTEM32/ntdll.dll
#1  0x00007ffe21bf3bfe in WaitForSingleObjectEx () from /c/WINDOWS/System32/KERNELBASE.dll
#2  0x0000000210044f54 in cygthread::stub (arg=arg@entry=0x210231970 <threads+176>)
    at /usr/src/MSYS2-packages/msys2-runtime/src/msys2-runtime/winsup/cygwin/cygthread.cc:112
#3  0x0000000210045ac3 in _cygtls::call2 (this=0x2f8ce00, func=0x210044eb0 <cygthread::stub(void*)>,
    arg=0x210231970 <threads+176>, buf=buf@entry=0x2f8cd20)
    at /usr/src/MSYS2-packages/msys2-runtime/src/msys2-runtime/winsup/cygwin/cygtls.cc:41
#4  0x0000000210045b74 in _cygtls::call (func=<optimized out>, arg=<optimized out>)
    at /usr/src/MSYS2-packages/msys2-runtime/src/msys2-runtime/winsup/cygwin/cygtls.cc:28
#5  0x00007ffe2366257d in KERNEL32!BaseThreadInitThunk () from /c/WINDOWS/System32/KERNEL32.DLL
#6  0x00007ffe2476aa58 in ntdll!RtlUserThreadStart () from /c/WINDOWS/SYSTEM32/ntdll.dll
#7  0x0000000000000000 in ?? ()
Stacktrace of thread 5 ("pipesel")
#0  0x00007ffe247af3f4 in ntdll!ZwWaitForSingleObject () from /c/WINDOWS/SYSTEM32/ntdll.dll
#1  0x00007ffe21bf3bfe in WaitForSingleObjectEx () from /c/WINDOWS/System32/KERNELBASE.dll
#2  0x0000000210044f54 in cygthread::stub (arg=arg@entry=0x2102319c8 <threads+264>)
    at /usr/src/MSYS2-packages/msys2-runtime/src/msys2-runtime/winsup/cygwin/cygthread.cc:112
#3  0x0000000210045ac3 in _cygtls::call2 (this=0x318ce00, func=0x210044eb0 <cygthread::stub(void*)>,
    arg=0x2102319c8 <threads+264>, buf=buf@entry=0x318cd20)
    at /usr/src/MSYS2-packages/msys2-runtime/src/msys2-runtime/winsup/cygwin/cygtls.cc:41
#4  0x0000000210045b74 in _cygtls::call (func=<optimized out>, arg=<optimized out>)
    at /usr/src/MSYS2-packages/msys2-runtime/src/msys2-runtime/winsup/cygwin/cygtls.cc:28
#5  0x00007ffe2366257d in KERNEL32!BaseThreadInitThunk () from /c/WINDOWS/System32/KERNEL32.DLL
#6  0x00007ffe2476aa58 in ntdll!RtlUserThreadStart () from /c/WINDOWS/SYSTEM32/ntdll.dll
#7  0x0000000000000000 in ?? ()

This is with MSYS2 runtime 3.4.10-b7ef037e.x86_64, i.e. built from git-for-windows/msys2-runtime@b7ef037, i.e. there seems to be a deadlock between this cygwait(40) and this cygwait(get_handle(), timeout).

@tyan0
Copy link

tyan0 commented Feb 13, 2024

I'm now trying to reproduce the problem, however, I cannot so far.
By any chance, does https://cygwin.com/pipermail/cygwin-patches/2024q1/012624.html fix the problem?

@tyan0
Copy link

tyan0 commented Feb 13, 2024

In a tmux session, I ran git diff/git log, which calls (MINGW) git.exe that then spawns (MSYS) less.exe. Then I press Q and immediately after that ⬆️. Now the process hangs. Actually, there are two processes for less, I assume one is the real one and the other one exists only to handle signals? The latter is the one hanging, and it has the following threads:

This is weird. One is real process and the other seems to be a stub process. However, if the less is MSYS2 less, stub process should be exited immediately after spawning the real process.
(1) Are you sure less is MSYS2 version (I mean, where ldd reports msys-2.0.dll)?
(2) Could you please check process tree of shell->git->less?

@dscho
Copy link
Member Author

dscho commented Feb 13, 2024

@tyan0 thank you for having a look!

(1) Are you sure less is MSYS2 version (I mean, where ldd reports msys-2.0.dll)?

Yes, I am positive that it is the MSYS2 version of less.exe:

$ ldd /usr/bin/less.exe
        ntdll.dll => /c/WINDOWS/SYSTEM32/ntdll.dll (0x7ffe24710000)
        KERNEL32.DLL => /c/WINDOWS/System32/KERNEL32.DLL (0x7ffe23650000)
        KERNELBASE.dll => /c/WINDOWS/System32/KERNELBASE.dll (0x7ffe21bc0000)
        msys-ncursesw6.dll => /usr/bin/msys-ncursesw6.dll (0x5fcb10000)
        msys-pcre2-8-0.dll => /usr/bin/msys-pcre2-8-0.dll (0x5d1c20000)
        msys-2.0.dll => /usr/bin/msys-2.0.dll (0x210040000)

(2) Could you please check process tree of shell->git->less?

Here goes:

$ ps -W | grep less
I   17209       1   17209      39580  cons1       4096 13:28:14 /usr/bin/less

$ wmic process where processid=39580 get commandline,parentprocessid,processid
CommandLine  ParentProcessId  ProcessId
less         43832            39580

$ wmic process where processid=43832 get commandline,parentprocessid,processid
CommandLine                          ParentProcessId  ProcessId
git branch --sort=-committerdate -l  37056            43832

$ wmic process where processid=37056 get commandline,parentprocessid,processid
CommandLine                                      ParentProcessId  ProcessId
C:\git-sdk-64\mingw64\bin\git.exe list-branches  29792            37056

$ wmic process where processid=29792 get commandline,parentprocessid,processid
CommandLine                     ParentProcessId  ProcessId
C:\git-sdk-64\usr\bin\bash.exe  15836            29792

$ wmic process where processid=15836 get commandline,parentprocessid,processid
CommandLine                     ParentProcessId  ProcessId
C:\git-sdk-64\usr\bin\bash.exe  14100            15836

$ wmic process where processid=14100 get commandline,parentprocessid,processid
No Instance(s) Available.

$ ps -W | grep -e 43832 -e 37056 -e 29792 -e 15836 -e 39580
     1446    1231    1446      15836  pty5        4096   Feb 11 /usr/bin/bash
    95328       0       0      29792  ?              0 13:28:14 C:\git-sdk-64\usr\bin\bash.exe
    17208    1446   17208      37056  pty5        4096 13:28:13 /mingw64/bin/git
   109368       0       0      43832  ?              0 13:28:14 C:\git-sdk-64\mingw64\libexec\git-core\git.exe
I   17209       1   17209      39580  cons1       4096 13:28:14 /usr/bin/less

I should probably mention that I still run with disable_pcon.

One thing I just noticed is that less runs in cons1, not in pty5 as the other processes. Another thing I noticed is that I in front of that less line, something I see for the first time and have currently no clue as to what it means.

@tyan0
Copy link

tyan0 commented Feb 14, 2024

Thanks for the additional information. I still fail to reproduce the issue both with and without pseudo console.

$ ps -W | grep -e 43832 -e 37056 -e 29792 -e 15836 -e 39580
     1446    1231    1446      15836  pty5        4096   Feb 11 /usr/bin/bash
    95328       0       0      29792  ?              0 13:28:14 C:\git-sdk-64\usr\bin\bash.exe
    17208    1446   17208      37056  pty5        4096 13:28:13 /mingw64/bin/git
   109368       0       0      43832  ?              0 13:28:14 C:\git-sdk-64\mingw64\libexec\git-core\git.exe
I   17209       1   17209      39580  cons1       4096 13:28:14 /usr/bin/less

In this case, only one less process seems to exist. Does the stub process alive only when the hang occurs?

I should probably mention that I still run with disable_pcon.
One thing I just noticed is that less runs in cons1, not in pty5 as the other processes. Another thing I noticed is that I in front of that less line, something I see for the first time and have currently no clue as to what it means.

This occurs only when pseudo console is enabled. If it is disabled, both less and git should run in pty5.

@tyan0
Copy link

tyan0 commented Feb 14, 2024

The stack trace looks like a normal less state. It might not be a deadlock, just keystrokes getting lost somewhere.

@tyan0
Copy link

tyan0 commented Feb 15, 2024

Could you please take a log using attached patch? for_dscho_log.patch
This patch adds log function for the input pipe switching history between cygwin program and non-cygwin program.

Steps

(1) Apply the patch and build runtime.
(2) Set environment variable LOGPTYSW to a log file name like /tmp/hang.log(Full cygpath).
(3) Start terminal with LOGPTYSW set (mintty, screen, tmux, etc.).
(4) Make the issue happen.
(5) While program is hanging, copy the log file to another file.

After all, please send me the copied log file. Thanks in advance.

@tyan0
Copy link

tyan0 commented Feb 15, 2024

The log patch revised. Please use this new one.
for_dscho_log.patch

[Edited]

@dscho
Copy link
Member Author

dscho commented Feb 15, 2024

Thank you @tyan0. I asked GitHub Actions to build the artifact over here: https://github.com/dscho/msys2-runtime/actions/runs/7923333552/job/21632838417.

@tyan0
Copy link

tyan0 commented Feb 17, 2024

Please also apply this additional patch.

diff --git a/winsup/cygwin/fhandler/pty.cc b/winsup/cygwin/fhandler/pty.cc
index 9d7ef3c9d..8ed435f67 100644
--- a/winsup/cygwin/fhandler/pty.cc
+++ b/winsup/cygwin/fhandler/pty.cc
@@ -1244,6 +1244,7 @@ fhandler_pty_slave::reset_switch_to_nat_pipe (void)
       return;
     }
   /* Clean up nat pipe state */
+  LOGPTYSW ("pty%d reset to to_cyg\n", get_minor ());
   get_ttyp ()->pty_input_state = tty::to_cyg;
   get_ttyp ()->nat_pipe_owner_pid = 0;
   get_ttyp ()->switch_to_nat_pipe = false;

Is the issue still not logged?

@dscho
Copy link
Member Author

dscho commented Feb 17, 2024

Please also apply this additional patch.

Thank you! Force-pushed and building.

Is the issue still not logged?

I have not had much time to work in my Git for Windows SDK, therefore the issue has not happened again yet. But it is logged, the log file is already ~64kB and growing, eagerly awaiting a hang.

As soon as the build is finished, I will replace the MSYS2 runtime and continue monitoring.

@dscho
Copy link
Member Author

dscho commented Feb 18, 2024

I should probably mention that I still run with disable_pcon.
One thing I just noticed is that less runs in cons1, not in pty5 as the other processes. Another thing I noticed is that I in front of that less line, something I see for the first time and have currently no clue as to what it means.

This occurs only when pseudo console is enabled. If it is disabled, both less and git should run in pty5.

Sorry, I missed this comment before.

In my case, disable_pcon is not set via an environment variable. And it is actually not set at all, because I somehow managed to forget that 8e89fffcfb0884da1398dd55f0d0cc57294549ec was dropped in the rebase of Git for Windows' fork onto MSYS2 runtime v3.4.6.

@dscho
Copy link
Member Author

dscho commented Feb 19, 2024

FWIW I am now running without disable_pcon and the log file is getting quite large, 1.2G at time of writing. I may have to abandon the experiment soon. But then, I haven't seen anything like this hang, either. Maybe it was simply a misunderstanding between different processes in the same process tree, running with different ideas as to disable_pcon?

@tyan0
Copy link

tyan0 commented Feb 19, 2024

The log taken while no issue happened is not worth enough. You can delete it and renew the logging. Why on the earth the issue cannot be reproduced...

@dscho
Copy link
Member Author

dscho commented Feb 19, 2024

The log taken while no issue happened is not worth enough. You can delete it and renew the logging.

D'oh, why didn't I think of deleting it ;-)

Why on the earth the issue cannot be reproduced...

I think it really might have something to do with the mix of disable_pcon and enable_pcon processes that are separated by MINGW processes in the process tree: The bash.exe with disable_pcon, from which was spawned the git.exe (not an MSYS process at all), which spawned less.exe with implicit enable_pcon (because the MSYS=disable_pcon variable was somehow lost between calling and running the bash.exe process).

@Geoffrey-A
Copy link

@dscho you may be interested in msys2/msys2-runtime#185. It’s a bit long, but the summary is that the default enable_pcon has caused me issues with less and man, but also more surprisingly with C’s execlp function. The latter point doesn’t seem to have been reproduced by others, though.

A smaller point, but to be aware of, is that while the Git for Windows installer will set disable_pcon by means of a config file, PortableGit will not, so users would observe different behaviours for a same GFW version depending on the installer they choose.

As a final note, I had also made the suggestion that msys2 should take from the GFW the feature of having the commit hash in the dll version, so I was glad to see you submitted PR 192 to msys-runtime (even if that is not the main object of the change, it’s a nice bonus side effect).

In general, thank you for your hard work!

@dscho
Copy link
Member Author

dscho commented Feb 23, 2024

The problem is that enable_pcon seems to solve more problems than it causes, by now.

@dscho
Copy link
Member Author

dscho commented Feb 23, 2024

The log taken while no issue happened is not worth enough. You can delete it and renew the logging.

D'oh, why didn't I think of deleting it ;-)

@tyan0 FWIW I have not experienced any hang in the meantime, and have just deleted a 6.1G log file to make space for more.

@dscho
Copy link
Member Author

dscho commented Feb 25, 2024

Still no problem, so I'll just call it and say that enable_pcon vs disable_pcon was the issue, and this ticket can be closed because we've got bigger fish to feed.

@dscho dscho closed this as not planned Won't fix, can't repro, duplicate, stale Feb 25, 2024
@dscho
Copy link
Member Author

dscho commented Feb 27, 2024

@tyan0 it's happening again, but this time I paid close attention to what I did, and it was that I pressed Ctrl+C while git log was running. The thing is, git.exe is a MINGW process in my case, and it spawns less.exe which is an MSYS program. So I think the problem might be related to what you hinted at a few weeks ago: when the child process goes away unplanned, there is still something waiting in the MSYS2 runtime (or for that matter, the Cygwin runtime), and since it is waiting for something that already is gone for good, it is hanging. Rings a bell?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants