Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

clone via HTTP fails #427

Closed
nomas2000 opened this issue Nov 14, 2023 · 3 comments · Fixed by #428
Closed

clone via HTTP fails #427

nomas2000 opened this issue Nov 14, 2023 · 3 comments · Fixed by #428
Labels
bug Something isn't working

Comments

@nomas2000
Copy link

soft-serve running on AlmaLinux9 (ie. redhat)
a large repo (that I want to mirror) works via SSH, but "fails" via HTTP
... other (smaller?) repos seem to work via HTTP

Attached (log) file shows launching fresh container and then:
installing soft-serve, importing a big-repo, and 'git clone http://...' fails with 500-err
... rerun the steps to reproduce the err ...

Did not see helpful err-msgs w/ *DEBUG* enabled (logs below are from previous week)

git clone http://build-soft/linux-stable.git TRASH
Cloning into 'TRASH'...
fatal: remote-curl: bad line length character: 500
fatal: the remote end hung up unexpectedly
Nov 07 12:08:03 build-soft soft[37635]: 2023-11-07 12:08:03 DEBU request method=GET uri=/linux-stable.git/info/refs?service=git-upload-pack addr=172.x.y.z:42488
Nov 07 12:08:03 build-soft soft[37635]: 2023-11-07 12:08:03 DEBU response status="200 OK" bytes="125 B" time=11.245363ms
Nov 07 12:08:03 build-soft soft[37635]: 2023-11-07 12:08:03 DEBU request method=POST uri=/linux-stable.git/git-upload-pack addr=172.x.y.z:42488
Nov 07 12:08:03 build-soft soft[37635]: 2023-11-07 12:08:03 DEBU response status="200 OK" bytes="255 kB" time=165.572508ms
Nov 07 12:08:03 build-soft soft[37635]: 2023-11-07 12:08:03 DEBU request method=POST uri=/linux-stable.git/git-upload-pack addr=172.x.y.z:42488
Nov 07 12:08:03 build-soft soft[37635]: 2023-11-07 12:08:03 ERRO http: http: superfluous response.WriteHeader call from github.com/charmbracelet/soft-serve/pkg/web.(*logWriter).WriteHeader (logging.go:40)
Nov 07 12:08:03 build-soft soft[37635]: 2023-11-07 12:08:03 DEBU response status="500 Internal Server Error" bytes="31 B" time=10.200038ms

soft-serve-alma9-container-large-repo.txt
@aymanbagabas aymanbagabas added the bug Something isn't working label Nov 14, 2023
aymanbagabas added a commit that referenced this issue Nov 15, 2023
@aymanbagabas
fix(http): times out on large repositories
cdf8928 
This is due to having a _set_ value of Read/Write http server timeout
values, and a faulty git gzip request handler.

Replace the read/write timeouts with idle timeout which is only used
when keep-alive is enabled. That is the case by default.

Fix git by properly handling gzip and buffered git service responses.

Improve git http handler logging

Fixes: #427
@aymanbagabas aymanbagabas mentioned this issue Nov 15, 2023
Merged
aymanbagabas added a commit that referenced this issue Nov 15, 2023
@aymanbagabas
fix(http): times out on large repositories
1b07a36 
This was due to having a _set_ value of Read/Write http server timeout
values, and a faulty git gzip request handler. The server drops the
connection if there wasn't any read/write within 10 seconds.

Replace the read/write timeouts with idle timeout which will reset the
counter to _either_ read/write within 10 seconds. Idle timeout is only
used when keep-alive is enabled. That is the case by default.

Fix git by properly handling gzip and buffered git service responses.

Improve git http handler logging

Fixes: #427
aymanbagabas added a commit that referenced this issue Nov 15, 2023
@aymanbagabas
fix(http): times out on large repositories
ca1c164 
This was due to having a _set_ value of Read/Write http server timeout
values, and a faulty git gzip request handler. The server drops the
connection if there wasn't any read/write within 10 seconds.

Replace the read/write timeouts with idle timeout which will reset the
counter to _either_ read/write within 10 seconds. Idle timeout is only
used when keep-alive is enabled. That is the case by default.

Fix git by properly handling gzip and buffered git service responses.

Improve git http handler logging

Fixes: #427
@aymanbagabas
Copy link
Member

Hi @nomas2000, thank you very much for reporting this issue. After some debugging, I found the culprit. Soft Serve HTTP server has a read timeout of 10 seconds, and it closes the connection since the repo is taking too long to read. Nevertheless, it was also handling gzip incorrectly which is the default compression git use with large repositories.

This now should be fixed in #428, feel free to test it out and report back 🙂

@nomas2000
Copy link
Author

If you fixed it, release it.

aymanbagabas added a commit that referenced this issue Nov 16, 2023
@aymanbagabas
fix(http): times out on large repositories (#428)
e0148ca 
This was due to having a _set_ value of Read/Write http server timeout
values, and a faulty git gzip request handler. The server drops the
connection if there wasn't any read/write within 10 seconds.

Replace the read/write timeouts with idle timeout which will reset the
counter to _either_ read/write within 10 seconds. Idle timeout is only
used when keep-alive is enabled. That is the case by default.

Fix git by properly handling gzip and buffered git service responses.

Improve git http handler logging

Fixes: #427
@aymanbagabas
Copy link
Member

Released #428 in https://github.com/charmbracelet/soft-serve/releases/tag/v0.7.2

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix(http): times out on large repositories charmbracelet/soft-serve
2 participants
@aymanbagabas @nomas2000