AutoTabPFNClassifier using CUDA silently fails due to SIGSEGV in cuda malloc

[shaywinter]

Using AutoTabPFNClassifier with a Cuda device (RTX 3090, 24GB) works ok as long as the tuning time remains short (~<=300).
increasing it to 500-1000 Sec fails with a core dump, loading it in gdb:

Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007fbc1c6e58d6 in c10::cuda::CUDACachingAllocator::Native::DeviceCachingAllocator::malloc(signed char, unsigned long, CUstream_st*) ()

Disabling memory optimizations of TabPFN slightly increases max possible tuning time (1000), but it issue still happens with longer tuning times (2000).

[LeoGrin]

Hey @shaywinter ! Thanks for the report :)
I cannot reproduce this error on my machine. Could you please try running this on CPU with the same input and see if you get a more informative error?
If you can share more about your dataset (ideally the dataset itself if public, otherwise some characteristics) I'll be happy to investigate.