Wandb server : User setting error - Application Error Unable to reach the backend api. If this perists, check the system logs.

[AJ-ajay-AY]

Hi Team,

I am facing an issue while setting up Wandb server on a AWS EC2 machine for a team trial .

Installation and server start:
command : pip install wandb
version : version 0.15.3
start command: wandb server start
using on default port

System details
server and os : AWS Ec2 , os Amazon linux 2
network: private vpc

I opened up localhost:8080 , created a user and went over to System Settings to set up license keys, S3 storage and MysqlDb . I tried all possible options for frontend host : http://10.5.240.147:8080 and http://127.0.0.1:8080 . I always keep getting this error

Application Error
Unable to reach the backend api. If this perists, check the system logs

I tried resolving by looking at resolved GitHub issues - No luck

[luisbergua]

Hi @AJ-ajay-AY, thanks for reporting this! Would it be possible to share the Debug Bundle of your instance so I can have a look at it and see why this error is raising? An admin of the instance can get it from the /system-admin page -> top right corner W&B icon -> Debug Bundle. Feel free to send it to my email [email protected]

[AJ-ajay-AY]

Sent on your mail id .

[AJ-ajay-AY]

Hi @AJ-ajay-AY, thanks for reporting this! Would it be possible to share the Debug Bundle of your instance so I can have a look at it and see why this error is raising? An admin of the instance can get it from the /system-admin page -> top right corner W&B icon -> Debug Bundle. Feel free to send it to my email [email protected]

Hope you got the email .

[twmht]

I got the same errors, where can i check the system log?

[twmht]

from docker logs

*** Running /etc/my_init.d/01_enable-services.sh...
*** Enabling production mode
*** Running /etc/my_init.d/02_load-settings.sh...
*** Loading settings...
2023/06/02 03:30:57 Created default user
2023/06/02 03:30:57 Generating new session key for auth...
2023/06/02 03:30:57 Generating new certificate and key for auth...
*** Booting runit daemon...
*** Runit started as PID 39
*** Setting up mysql database...
*** Starting wandb servers...
*** Configuring minio...
Bucket created successfully `local/local-files`.
mc: <ERROR> Unable to set 'notify_redis address=127.0.0.1:6379 format=access queue_limit=10000 queue_dir=/etc/minio/events key=filemetadata' to server. Access Denied. (Prefix access is denied: .minio.sys/config/config.json).
mc: <ERROR> Unable to enable notification on the specified bucket. Access Denied.
*** Migrating database...
Initialized random go rng seed: to fix set env var TEST_GO_RNG_SEED=6021073221992367906
Invoking megabinary "migrate": GOMAXPROCS=4
2023/06/02 03:31:22 Running squash migration from version 0 to 171...
2023/06/02 03:33:30 Squash migration successful. Continuing with normal migrations...
172/u add_preview_friendly_url_col_to_access_tokens_table (1.153767434s)
173/u move_preview_friendly_url_to_new_table (2.952600179s)
174/u increase_auth_id_length_limit (3.835195723s)
175/u increase_session_auth_id_length_limit (4.428989913s)
176/u add_tags (6.175668582s)
177/u glue_task_arguments (7.924602462s)
178/u glue_task_types (9.14043414s)
179/u update_preview_friendly_urls (10.173196421s)
180/u increase_task_id_width (11.680714578s)
181/u glue_run_errors (12.946736323s)
182/u glue_run_name_idx (13.813316795s)
183/u automatic_upgrades_columns (13.800288108s)
184/u dismissed_banners (12.800906167s)
185/u scrubbing_state_no_created_at (13.034356265s)
186/u update_kms_key_type (13.048328207s)
187/u teams_config (12.300977726s)
188/u email_invites (11.701192466s)
189/u launch_agents_created_by (11.843370379s)
Loading custom chart definitions...
panic: mismatching uids

goroutine 1 [running]:
github.com/wandb/core/services/gorilla/cmd.(*migrateCommander).MainCmd(0xc000f9f620, {0xc000f694c0, 0x2, 0x2})
        /mnt/ramdisk/core/services/gorilla/cmd/migrate.go:314 +0xe4e
main.main()
        /mnt/ramdisk/core/services/gorilla/cmd/megabinary/main.go:72 +0x452
*** All services started
*** Access W&B at http://localhost:8080
queue_dir=/etc/minio/events key=filemetadata' to server. Access Denied. (Prefix access is denied: .minio.sys/config/config.json).

[AJ-ajay-AY]

Any progress or movement on this @luisbergua ?

[luisbergua]

Hi @AJ-ajay-AY, apologies for the delay on this! Would you mind sending the bundle again? It seems to be an invalid zip file and I'm unable to expand it so cannot check the logs. @twmht you can get the Debug Bundle from the /system-admin page -> top right corner W&B icon -> Debug Bundle. Would you mind sending it to [email protected]?

[luisbergua]

Hi @AJ-ajay-AY! Could you please share the Bundle again? I wasn't able to open the one that I received as the zip file was corrupted

[luisbergua]

Hi @AJ-ajay-AY , since we have not heard back from you we are going to close this request. If you would like to re-open the conversation, please let us know!