Network error (TransientError), entering retry loop. #129
Comments
|
Hi @Stashq, could you possibly let us know the output of running |
|
When I run Ping between W&B server and MinIO works correctly by the way. |
|
Hey @Stashq, apologies for the delay here. Network error probably means your machine (or pod/node) isn't able to connect to the server. Do you have any VPN/Proxy configured on your end? |
|
Hi @Stashq, I wanted to follow up on this request. Please let us know if we can be of further assistance or if your issue has been resolved. |
|
@anmolmann thanks a lot for your reply. I'm not able to check the proxy solution in the next 7 days, but I'll definitely come back to this very soon and inform you about the result. |
|
Thanks for the update, @Stashq! Please feel free to write in again and share your results with us. |
|
@Stashq the key does not exist is likely indication the minio credentials are either incorrect or not encoded properly. When putting credentials into the URL you must be sure they are url encoded to escape any special characters. |
|
Another thing to try that isn't well supported but should work is setting the |
|
@vanpelt thanks for your answer. My minio access and secret keys were suggested by minio and consist only of numbers and letters. I guess the error message "key does not exist" refers to non-existing objects in minio storage. I tried setting I set bucket read/write access as public, but the error remains. This are the logs from wandb-server: |
|
Hey @Stashq , apologies for the delay here. Checking in to see if you were able to resolve the above network errors for your local deployment? |
|
Hi @Stashq , since we have not heard back from you we are going to close this request. If you would like to re-open the conversation, please let us know! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Introduction
I am trying to set up W&B server using an external MySQL database and MinIO storage. This is my docker-compose file:
After I start MinIO, I manually create access and secret keys.
Problem description
Whenever I try to log to W&B, error with folowing message occures:
wandb: Network error (TransientError), entering retry loop.Simple metrics are passed to W&B server, but I can not log artifacts like pytorch models. At the end the program stuck in the loop showing this message. When I check the wandb server logs I see an error: "key does not exist".
wandb-service_1 | {"level":"ERROR","time":"2023-10-17T12:21:48.756074416Z","info":{"program":"gorilla","source":"mnt/ramdisk/core/services/gorilla/pkg/observability/gerr/reporting.go:193","pid":57},"data":{"authUser":"username","userID":4,"operationName":"UpsertBucket","projectID":3,"cacheKey":"project:3:approx_run_count"},"message":"key does not exist","dd.trace_id":"12901086739434798565","error":{"kind":"*errors.errorString","message":"key does not exist"}}What works correctly
wandb server startversion, everything works correctly, so the problem has to be outside of my python code.W&B logs
All logs are listed below:
MinIO check
I've also checked MinIO http logs using
mc admin tracecommand. After W&B process finishes, I get this error:W&B verify
Command
wandb verifygives me the same error:Question
Does anyone have an idea why this error occurs? Is it a problem of the wrong W&B or MinIO setup? Has anybody seen that error before? Thx in advance.
The text was updated successfully, but these errors were encountered: