-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathslurm-5049492.out
102 lines (100 loc) · 11.5 KB
/
slurm-5049492.out
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
💡 Learn about RayTune at https://docs.ultralytics.com/integrations/ray-tune
Requirement already satisfied: ray[tune] in /home/jpfinley/.conda/envs/cent7/2020.11-py38/yolov8/lib/python3.8/site-packages (2.9.2)
Requirement already satisfied: click>=7.0 in /home/jpfinley/.conda/envs/cent7/2020.11-py38/yolov8/lib/python3.8/site-packages (from ray[tune]) (8.1.7)
Requirement already satisfied: filelock in /home/jpfinley/.conda/envs/cent7/2020.11-py38/yolov8/lib/python3.8/site-packages (from ray[tune]) (3.13.1)
Requirement already satisfied: jsonschema in /home/jpfinley/.conda/envs/cent7/2020.11-py38/yolov8/lib/python3.8/site-packages (from ray[tune]) (4.20.0)
Requirement already satisfied: msgpack<2.0.0,>=1.0.0 in /home/jpfinley/.conda/envs/cent7/2020.11-py38/yolov8/lib/python3.8/site-packages (from ray[tune]) (1.0.7)
Requirement already satisfied: packaging in /home/jpfinley/.conda/envs/cent7/2020.11-py38/yolov8/lib/python3.8/site-packages (from ray[tune]) (23.1)
Requirement already satisfied: protobuf!=3.19.5,>=3.15.3 in /home/jpfinley/.conda/envs/cent7/2020.11-py38/yolov8/lib/python3.8/site-packages (from ray[tune]) (4.25.1)
Requirement already satisfied: pyyaml in /home/jpfinley/.conda/envs/cent7/2020.11-py38/yolov8/lib/python3.8/site-packages (from ray[tune]) (6.0.1)
Requirement already satisfied: aiosignal in /home/jpfinley/.conda/envs/cent7/2020.11-py38/yolov8/lib/python3.8/site-packages (from ray[tune]) (1.3.1)
Requirement already satisfied: frozenlist in /home/jpfinley/.conda/envs/cent7/2020.11-py38/yolov8/lib/python3.8/site-packages (from ray[tune]) (1.4.0)
Requirement already satisfied: requests in /home/jpfinley/.conda/envs/cent7/2020.11-py38/yolov8/lib/python3.8/site-packages (from ray[tune]) (2.31.0)
Requirement already satisfied: pandas in /home/jpfinley/.conda/envs/cent7/2020.11-py38/yolov8/lib/python3.8/site-packages (from ray[tune]) (2.0.3)
Requirement already satisfied: tensorboardX>=1.9 in /home/jpfinley/.conda/envs/cent7/2020.11-py38/yolov8/lib/python3.8/site-packages (from ray[tune]) (2.6.2.2)
Requirement already satisfied: pyarrow>=6.0.1 in /home/jpfinley/.conda/envs/cent7/2020.11-py38/yolov8/lib/python3.8/site-packages (from ray[tune]) (14.0.1)
Requirement already satisfied: fsspec in /home/jpfinley/.conda/envs/cent7/2020.11-py38/yolov8/lib/python3.8/site-packages (from ray[tune]) (2023.12.2)
Requirement already satisfied: numpy>=1.16.6 in /home/jpfinley/.conda/envs/cent7/2020.11-py38/yolov8/lib/python3.8/site-packages (from pyarrow>=6.0.1->ray[tune]) (1.24.3)
Requirement already satisfied: attrs>=22.2.0 in /home/jpfinley/.conda/envs/cent7/2020.11-py38/yolov8/lib/python3.8/site-packages (from jsonschema->ray[tune]) (23.1.0)
Requirement already satisfied: importlib-resources>=1.4.0 in /home/jpfinley/.conda/envs/cent7/2020.11-py38/yolov8/lib/python3.8/site-packages (from jsonschema->ray[tune]) (6.1.1)
Requirement already satisfied: jsonschema-specifications>=2023.03.6 in /home/jpfinley/.conda/envs/cent7/2020.11-py38/yolov8/lib/python3.8/site-packages (from jsonschema->ray[tune]) (2023.11.2)
Requirement already satisfied: pkgutil-resolve-name>=1.3.10 in /home/jpfinley/.conda/envs/cent7/2020.11-py38/yolov8/lib/python3.8/site-packages (from jsonschema->ray[tune]) (1.3.10)
Requirement already satisfied: referencing>=0.28.4 in /home/jpfinley/.conda/envs/cent7/2020.11-py38/yolov8/lib/python3.8/site-packages (from jsonschema->ray[tune]) (0.32.0)
Requirement already satisfied: rpds-py>=0.7.1 in /home/jpfinley/.conda/envs/cent7/2020.11-py38/yolov8/lib/python3.8/site-packages (from jsonschema->ray[tune]) (0.13.2)
Requirement already satisfied: python-dateutil>=2.8.2 in /home/jpfinley/.conda/envs/cent7/2020.11-py38/yolov8/lib/python3.8/site-packages (from pandas->ray[tune]) (2.8.2)
Requirement already satisfied: pytz>=2020.1 in /home/jpfinley/.conda/envs/cent7/2020.11-py38/yolov8/lib/python3.8/site-packages (from pandas->ray[tune]) (2023.3.post1)
Requirement already satisfied: tzdata>=2022.1 in /home/jpfinley/.conda/envs/cent7/2020.11-py38/yolov8/lib/python3.8/site-packages (from pandas->ray[tune]) (2023.3)
Requirement already satisfied: charset-normalizer<4,>=2 in /home/jpfinley/.conda/envs/cent7/2020.11-py38/yolov8/lib/python3.8/site-packages (from requests->ray[tune]) (3.3.2)
Requirement already satisfied: idna<4,>=2.5 in /home/jpfinley/.conda/envs/cent7/2020.11-py38/yolov8/lib/python3.8/site-packages (from requests->ray[tune]) (3.6)
Requirement already satisfied: urllib3<3,>=1.21.1 in /home/jpfinley/.local/lib/python3.8/site-packages (from requests->ray[tune]) (2.1.0)
Requirement already satisfied: certifi>=2017.4.17 in /home/jpfinley/.conda/envs/cent7/2020.11-py38/yolov8/lib/python3.8/site-packages (from requests->ray[tune]) (2023.11.17)
Requirement already satisfied: zipp>=3.1.0 in /home/jpfinley/.conda/envs/cent7/2020.11-py38/yolov8/lib/python3.8/site-packages (from importlib-resources>=1.4.0->jsonschema->ray[tune]) (3.17.0)
Requirement already satisfied: six>=1.5 in /home/jpfinley/.conda/envs/cent7/2020.11-py38/yolov8/lib/python3.8/site-packages (from python-dateutil>=2.8.2->pandas->ray[tune]) (1.16.0)
2024-03-06 13:32:04,255 INFO worker.py:1724 -- Started a local Ray instance.
WARNING ⚠️ search space not provided, using default search space.
2024-03-06 13:32:26,088 INFO tune.py:592 -- [output] This will use the new output engine with verbosity 1. To disable the new output and use the legacy output engine, set the environment variable RAY_AIR_NEW_OUTPUT=0. For more information, please see https://github.com/ray-project/ray/issues/36949
2024-03-06 13:32:26,141 INFO wandb.py:307 -- Already logged into W&B.
[2024-03-06 13:32:26,357 E 34617 39021] logging.cc:97: Unhandled exception: St12system_error. what(): Resource temporarily unavailable
[2024-03-06 13:32:27,355 E 34617 39021] logging.cc:104: Stack trace:
/home/jpfinley/.conda/envs/cent7/2020.11-py38/yolov8/lib/python3.8/site-packages/ray/_raylet.so(+0x10051da) [0x2afff50051da] ray::operator<<()
/home/jpfinley/.conda/envs/cent7/2020.11-py38/yolov8/lib/python3.8/site-packages/ray/_raylet.so(+0x1007918) [0x2afff5007918] ray::TerminateHandler()
/home/jpfinley/.conda/envs/cent7/2020.11-py38/yolov8/bin/../lib/libstdc++.so.6(+0xb135a) [0x2aff34dbc35a] __cxxabiv1::__terminate()
/home/jpfinley/.conda/envs/cent7/2020.11-py38/yolov8/bin/../lib/libstdc++.so.6(+0xb13c5) [0x2aff34dbc3c5]
/home/jpfinley/.conda/envs/cent7/2020.11-py38/yolov8/bin/../lib/libstdc++.so.6(+0xb1658) [0x2aff34dbc658]
/home/jpfinley/.conda/envs/cent7/2020.11-py38/yolov8/lib/python3.8/site-packages/pyarrow/libarrow.so.1400(+0x57ffad) [0x2affeb3c6fad] std::__throw_system_error()
/home/jpfinley/.conda/envs/cent7/2020.11-py38/yolov8/lib/python3.8/site-packages/pyarrow/libarrow.so.1400(+0x20b5a59) [0x2affecefca59]
/home/jpfinley/.conda/envs/cent7/2020.11-py38/yolov8/lib/python3.8/site-packages/pyarrow/libarrow.so.1400(_ZN5arrow8internal10ThreadPool21LaunchWorkersUnlockedEi+0x9f) [0x2affebf68a7f] arrow::internal::ThreadPool::LaunchWorkersUnlocked()
/home/jpfinley/.conda/envs/cent7/2020.11-py38/yolov8/lib/python3.8/site-packages/pyarrow/libarrow.so.1400(_ZN5arrow8internal10ThreadPool9SpawnRealENS0_9TaskHintsENS0_6FnOnceIFvvEEENS_9StopTokenEONS3_IFvRKNS_6StatusEEEE+0x30d) [0x2affebf6a0ed] arrow::internal::ThreadPool::SpawnReal()
/home/jpfinley/.conda/envs/cent7/2020.11-py38/yolov8/lib/python3.8/site-packages/pyarrow/libarrow.so.1400(_ZN5arrow2fs9CopyFilesERKSt10shared_ptrINS0_10FileSystemEERKNS0_12FileSelectorES5_RKSsRKNS_2io9IOContextElb+0xa43) [0x2affeb593313] arrow::fs::CopyFiles()
/home/jpfinley/.conda/envs/cent7/2020.11-py38/yolov8/lib/python3.8/site-packages/pyarrow/_fs.cpython-38-x86_64-linux-gnu.so(+0x41d77) [0x2afff5ac3d77] __pyx_pw_7pyarrow_3_fs_5_copy_files_selector()
python(_PyEval_EvalFrameDefault+0x399) [0x4d84a9] _PyEval_EvalFrameDefault
python(_PyEval_EvalCodeWithName+0x2f1) [0x4d70d1] _PyEval_EvalCodeWithName
python(_PyFunction_Vectorcall+0x19c) [0x4e823c] _PyFunction_Vectorcall
python(PyObject_Call+0x34e) [0x4f76ce] PyObject_Call
python(_PyEval_EvalFrameDefault+0x2073) [0x4da183] _PyEval_EvalFrameDefault
python(_PyEval_EvalCodeWithName+0x2f1) [0x4d70d1] _PyEval_EvalCodeWithName
python(_PyFunction_Vectorcall+0x19c) [0x4e823c] _PyFunction_Vectorcall
python(_PyEval_EvalFrameDefault+0x1153) [0x4d9263] _PyEval_EvalFrameDefault
python(_PyEval_EvalCodeWithName+0x2f1) [0x4d70d1] _PyEval_EvalCodeWithName
python(_PyFunction_Vectorcall+0x19c) [0x4e823c] _PyFunction_Vectorcall
python(_PyEval_EvalFrameDefault+0x1153) [0x4d9263] _PyEval_EvalFrameDefault
python(_PyEval_EvalCodeWithName+0x2f1) [0x4d70d1] _PyEval_EvalCodeWithName
python(_PyFunction_Vectorcall+0x19c) [0x4e823c] _PyFunction_Vectorcall
python(PyObject_Call+0x34e) [0x4f76ce] PyObject_Call
python(_PyEval_EvalFrameDefault+0x2073) [0x4da183] _PyEval_EvalFrameDefault
python(_PyEval_EvalCodeWithName+0x2f1) [0x4d70d1] _PyEval_EvalCodeWithName
python(_PyFunction_Vectorcall+0x19c) [0x4e823c] _PyFunction_Vectorcall
python(PyObject_Call+0x24a) [0x4f75ca] PyObject_Call
python(_PyEval_EvalFrameDefault+0x2073) [0x4da183] _PyEval_EvalFrameDefault
python(_PyFunction_Vectorcall+0x106) [0x4e81a6] _PyFunction_Vectorcall
python(_PyEval_EvalFrameDefault+0x6b2) [0x4d87c2] _PyEval_EvalFrameDefault
python(_PyFunction_Vectorcall+0x106) [0x4e81a6] _PyFunction_Vectorcall
python(_PyEval_EvalFrameDefault+0x6b2) [0x4d87c2] _PyEval_EvalFrameDefault
python(_PyFunction_Vectorcall+0x106) [0x4e81a6] _PyFunction_Vectorcall
python() [0x4f5154] method_vectorcall
python(PyObject_Call+0x24a) [0x4f75ca] PyObject_Call
python() [0x5bc9d7] t_bootstrap
python() [0x5bc984] pythread_wrapper
/usr/lib64/libpthread.so.0(+0x7ea5) [0x2aff156a3ea5] start_thread
/usr/lib64/libc.so.6(clone+0x6d) [0x2aff160bfb0d] clone
*** SIGABRT received at time=1709749947 on cpu 37 ***
PC: @ 0x2aff15ff7387 (unknown) raise
@ 0x2aff156ab630 3536 (unknown)
@ 0x2aff34dbc35a 383780336 __cxxabiv1::__terminate()
@ 0x2aff34dbc580 (unknown) (unknown)
[2024-03-06 13:32:27,355 E 34617 39021] logging.cc:361: *** SIGABRT received at time=1709749947 on cpu 37 ***
[2024-03-06 13:32:27,355 E 34617 39021] logging.cc:361: PC: @ 0x2aff15ff7387 (unknown) raise
[2024-03-06 13:32:27,355 E 34617 39021] logging.cc:361: @ 0x2aff156ab630 3536 (unknown)
[2024-03-06 13:32:27,355 E 34617 39021] logging.cc:361: @ 0x2aff34dbc35a 383780336 __cxxabiv1::__terminate()
[2024-03-06 13:32:27,355 E 34617 39021] logging.cc:361: @ 0x2aff34dbc580 (unknown) (unknown)
Fatal Python error: Aborted
Stack (most recent call first):
File "/home/jpfinley/.conda/envs/cent7/2020.11-py38/yolov8/lib/python3.8/site-packages/pyarrow/fs.py", line 269 in copy_files
File "/home/jpfinley/.conda/envs/cent7/2020.11-py38/yolov8/lib/python3.8/site-packages/ray/train/_internal/storage.py", line 110 in _pyarrow_fs_copy_files
File "/home/jpfinley/.conda/envs/cent7/2020.11-py38/yolov8/lib/python3.8/site-packages/ray/train/_internal/storage.py", line 228 in _upload_to_uri_with_exclude_fsspec
File "/home/jpfinley/.conda/envs/cent7/2020.11-py38/yolov8/lib/python3.8/site-packages/ray/train/_internal/storage.py", line 215 in _upload_to_fs_path
File "/home/jpfinley/.conda/envs/cent7/2020.11-py38/yolov8/lib/python3.8/site-packages/ray/train/_internal/syncer.py", line 136 in entrypoint
File "/home/jpfinley/.conda/envs/cent7/2020.11-py38/yolov8/lib/python3.8/threading.py", line 870 in run
File "/home/jpfinley/.conda/envs/cent7/2020.11-py38/yolov8/lib/python3.8/threading.py", line 932 in _bootstrap_inner
File "/home/jpfinley/.conda/envs/cent7/2020.11-py38/yolov8/lib/python3.8/threading.py", line 890 in _bootstrap
/var/spool/slurm/job5049492/slurm_script: line 5: 34617 Aborted python raytune.py