OpenML outage because of the TU/e cyberattack

[joaquinvanschoren]

OpenML went down on Sunday January 12th, 2025, due to a cyber-attack on the Eindhoven University of Technology, which even reached the international press. The OpenML servers were not affected or breached. However, the university took down the entire network out of caution, meaning that the OpenML servers were not accessible, not even to the OpenML system admins. While OpenML is fully containerized, has redundancy against failing servers, and multiple backups, everything was hosted in data centers under the university network, meaning that we could not get access to any of our servers. Below is a list of updates and plans to avoid this from happening in the future.

📢 Updates:

• Sunday January 19th: We gained internal access to our servers again and started setting up a new OpenML instance on a different network to restore access.
• Monday January 20th: We are deploying a read-only instance, which should be available very soon.
• Tuesday January 21th: We have deployed a read-only instance on http://145.38.195.79/. Note that this server has limited functionality (see below). ❤️ Shout out to @PGijsbers and @josvandervelde!
• Wednesday January 22th: The DNS is updated. https://openml.org is accessible again and points to the read-only server. The Python API can be used as usual (R and other APIs should work again as well). There are still SSL issues with https://www.openml.org and https://api.openml.org (used in sklearn.fetch_openml). We'll fix that as soon as possible.
• Thursday January 23rd: Certificates for the subdomains now work.
• Friday January 24th: The production server is accessible again. Most functionality is restored. Remaining known issues are listed below.
• Saturday January 25th: We released openml-python 0.15.1, which allows you to skip downloading parquet files. Update with python -m pip install --upgrade openml.

🚧 Known remaining issues:

• Some links (starting with https://openml1.win.tue.nl) are still blocked and won't be unblocked before Monday. This means that .pq versions of datasets are not available.

💻 Use cases:
sklearn.fetch_openml: Works :)

from sklearn.datasets import fetch_openml
adult = fetch_openml("adult", version=2)

🐍 OpenML Python:

If you are using openml-python, make sure to upgrade to the latest version (released January 25th, 2025).
With 0.15.1 installed, set environment variable OPENML_SKIP_PARQUET, i.e., export OPENML_SKIP_PARQUET=true for Linux/MacOS and set OPENML_SKIP_PARQUET=true for Windows, to avoid to attempt download parquet files. We recommend to use the feature only until the MinIO production server is operational again.
Without the environment variable set, openml-python is completely functional, but slower.

If you have to use OpenML 0.15.0 or earlier, you will experience the following issues:

get_dataset: Works, but slow, because it will first try to download .pq version of datasets. This will likely be fixed by Monday.
A workaround is to patch requests like this:

import openml
import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

# Patch requests so it doesn't wait or retry
retry_strategy = Retry(
    total=1,            # Only one retry
    backoff_factor=0.1, # Delay between retries
    status_forcelist=[429, 500, 502, 503, 504],  # Retry for specific status codes
)
adapter = HTTPAdapter(max_retries=retry_strategy)
session = requests.Session()
session.mount("http://", adapter)
session.mount("https://", adapter)
openml.config.requests_session = session

# Get datasets as usual
dataset = openml.datasets.get_dataset(1590, download_data=True)

Download datasets with dataset.get_data results in an error due to a bug in the fallback logic.

Note

Workarounds specifying the ip-based url (http://145.38.195.79/) are no longer needed, and may lead to errors.
In case you experience issues, make sure to try again with a clear cache. If the problem persists, please reach out to us.

Functionality of the read-only server (no longer relevant):

• Datasets and tasks can be searched and downloaded from the website
• The REST API and Python API can be used (see instructions below)
• Flows and runs are not yet accessible (this is a lot of data that we can't easily host on the temporary server)
• Read-only. It's not possible to upload new resources (and login is disabled) since we don't have synchronization with the production server.

Future plans:

• We will set up protocols to very quickly deploy a read-only server in another network to quickly fall back to that server in case anything happens to the university network.
• We will use the lessons learned from this to make sure we always have the OpenML data available outside of the TU/e network.
• We will set up a page with status information and unify communication to better inform the community.

We apologize for the outage and for any inconvenience this may have caused. This was an unprecedented series of events that we did not foresee and had very little control over. We will be better prepared if this ever happens again.

The OpenML team

[brianjester]

I was using a scikit-learn example that stopped working, and used the example on this page to create a workaround, which is provided below:

Code I was attempting to run from scikit-learn:

from sklearn.datasets import fetch_openml
bike_sharing = fetch_openml("Bike_Sharing_Demand", version=2, as_frame=True)
df = bike_sharing.frame

Workaround based on example on this page that now downloads:

import openml
openml.config.server = "http://145.38.195.79/api/v1/xml" # Point to the read-only server
bike_sharing = openml.datasets.get_dataset(42712, download_data=True) # version 2 of the dataset
df, *_ = bike_sharing.get_data()

[lesteve]

@joaquinvanschoren honestly there is no rush (we can definitely wait a few more days), but a question out of curiosity and as a scikit-learn maintainer, I was hoping for a work-around in scikit-learn similar to this but this does not work:

from sklearn.datasets import fetch_openml

from sklearn.datasets import _openml
# Point to the read-only server
_openml._OPENML_PREFIX = "http://145.38.195.79/"

bike_sharing = fetch_openml("Bike_Sharing_Demand", version=2, as_frame=True)
df = bike_sharing.frame

I get a HTTPError 400:

HTTPError: HTTP Error 400: Bad Request

Here is the curl info:

 ❯ curl 'http://145.38.195.79/data/v1/download/22044627'
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>InvalidBucketName</Code><Message>The specified bucket is not valid.</Message><Resource>/v1/download/22044627</Resource><RequestId>181CEF3D5DF63BBE</RequestId><HostId>dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8</HostId></Error>%   

Maybe it's the scikit-learn code that has some assumptions, or relies on something that is not yet working in the read-only server, maybe it's something else, not sure ...

[PGijsbers]

It looks like scikit-learn uses hard-coded path to determine where to fetch the data: https://github.com/scikit-learn/scikit-learn/blob/fef470195d4fcb689bb80c8858d1fcb116e027bc/sklearn/datasets/_openml.py#L40

The correct path on the read-only server is provided by the dataset description url and on the readonly server (currently) looks something like "http://145.38.195.79/data/datasets/0001/1212/dataset_1212.arff" for dataset 1212. However, don't try that right now as I am trying to reconfigure our proxy to work a bit better with minio (it currently doesn't allow for listing calls which breaks the download_all_files functionality of openml-python).

[lesteve]

OK I guess this is an opportunity for scikit-learn code to be improved then 😉 I opened scikit-learn/scikit-learn#30699 to track this (not urgent at all I guess you have bigger fish to fry right now 😉)

[fouodo]

Should R users wait until the primary server is back online?

[sebffischer]

@joaquinvanschoren When I change the server as suggested (and don't pass any API key), I get:

odt(42279)
INFO  [15:16:40.753] Retrieving JSON {url: `http://145.38.195.79/api/v1/json/data/42279`, authenticated: `FALSE`}
Error in curl::curl_fetch_disk(add_auth_string(url, api_key = api_key),  : 
  SSL peer certificate or SSH remote key was not OK: [145.38.195.79] SSL: no alternative certificate subject name matches target ipv4 address '145.38.195.79'

[PGijsbers]

The workaround with the IP adres is no longer needed since we managed to update the domain and we also added certificates. As a consequence, the workaround with the IP adres broke. It very likely works out of the box now (it does for Python), make sure to clear your cache if you continue to see references to the ip adres. If you still experience an issue, let us know and we'll look into it.

[sebffischer]

okay thanks

[sebffischer]

Should the data splits already work?

I get

! Error downloading 'https://openml.org/api_splits/get/359983/Task_359983_splits.arff' (http code: 412, oml code: NA, message: 'failed to perform action generate_folds. Evaluation Engine result send to EMAIL_API_LOG account.'

[joaquinvanschoren]

The production server is back. This should work now. Can you please try again?

[joaquinvanschoren]

For those who want to use image datasets (e.g. MetaAlbum), the cache paths are a bit weird at the moment, but should be normal again after the DNS updates. For now, here's a usage example:

https://colab.research.google.com/drive/1aEZmcmSFFLPFhW7lVAtHGDsbIpqktu5H?usp=sharing

[martinlschumann]

I am getting privacy errors with www.
It seems that the SSL certificate is only valid for https://openml.org and not https://www.openml.org
E.g. the following from https://openml.org/apis (REST API) does not work:
curl -X 'GET' 'https://www.openml.org/api/v1/json/data/3' -H 'accept: application/json'

[joaquinvanschoren]

Confirmed, thanks. We'll fix it as soon as possible.

[PGijsbers]

We believe the issues with the certification for subdomains are resolved now.

[imatiach-msft]

It says in the edited discussion above that the DNS was updated, but I am still seeing errors when calling the fetch_openml api without specifying the server directly in builds (as in, not using server=...):

Wednesday January 22th: The DNS is updated. https://www.openml.org/ is accessible again and points to the read-only server. The Python API can be used as usual (R and other APIs should work again as well). There is still a small issue with the search on the website.

See: https://github.com/microsoft/responsible-ai-toolbox/actions/runs/12795480111/job/36020797462?pr=2594

[imatiach-msft]

That makes a lot more sense now, thank you! In that case, I think I will just wait for the scikit-learn APIs to work again next week, and for now I will force-merge PRs if needed. If a release will be required from scikit-learn, then I will hopefully be able to update to the latest version of scikit-learn. If worst comes to worst, I can try to modify the notebook to download the dataset in a different way or from a different source next week once this becomes more clear.

[hengtanoms]

I'm still having issue when trying to calling fetch_openml:

X, y = fetch_openml("mnist_784", version=1, return_X_y=True, parser="auto", cache=False)

here is the exception:

File /opt/venv/lib/python3.9/site-packages/sklearn/datasets/_openml.py:488, in _load_arff_response(url, data_home, parser, output_type, openml_columns_info, feature_names_to_select, target_names_to_select, shape, md5_checksum, n_retries, delay)
    485     actual_md5_checksum = md5.hexdigest()
    487 if actual_md5_checksum != md5_checksum:
--> 488     raise ValueError(
    489         f"md5 checksum of local file for {url} does not match description: "
    490         f"expected: {md5_checksum} but got {actual_md5_checksum}. "
    491         "Downloaded file could have been modified / corrupted, clean cache "
    492         "and retry..."
    493     )
    495 gzip_file = _open_openml_url(url, data_home, n_retries=n_retries, delay=delay)
    496 with closing(gzip_file):

ValueError: md5 checksum of local file for data/v1/download/52667 does not match description: expected: 0298d579eb1b86163de7723944c7e495 but got fe38182e173709620a43065ffd82bc6b. Downloaded file could have been modified / corrupted, clean cache and retry...

I've checked the xml file https://openml.org/api/v1/data/554 it stated the checksum value is 0298d579eb1b86163de7723944c7e495 but the actual download file checksum is fe38182e173709620a43065ffd82bc6b.

Any idea how we can resolve this? I wasn't able to openml.datasets.get_dataset because I don't have permission to install the openml module (restriction set by online tutorlal jupyter site i'm using), I only allow to use fetch_openml

[joaquinvanschoren]

Hi, see the messages above about fetch_openml. It's currently not yet working again, sorry...
Are you not allowed to run '!pip install openml' in your jupyter notebook?
Can you download the dataset directly (e.g. with curl or wget)? There are .arff and .pq download links in https://openml.org/api/v1/data/554.

[joaquinvanschoren]

fetch_openml is working now 🎉

[hengtanoms]

@joaquinvanschoren very grateful thanks for you help. I can confirm that fetch_openml is working as expected now. Thanks again to you your team for all your hard work for getting opeml server working again :) .

[AI-Ahmed]

Hello @joaquinvanschoren,
On colab, I run the following snippet:

#  Satellite dataset https://www.openml.org/d/182
# Switch to the read-only server
openml.config.server = "http://145.38.195.79/api/v1/xml"

# Set the cache dir (for colab)
openml.config.set_root_cache_directory(os.path.expanduser('/content/cache'))

dataset = openml.datasets.get_dataset(182, download_data=True, download_all_files=True) # version 2 of the dataset

I get:

FileNotFoundError: [Errno 2] No such file or directory: '/content/cache/79/195/38/145/datasets/182/description.xml'

SSLError: HTTPSConnectionPool(host='145.38.195.79', port=443): Max retries exceeded with url: /api/v1/xml/data/182 (Caused by SSLError(SSLCertVerificationError(1, "[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: IP address mismatch, certificate is not valid for '145.38.195.79'. (_ssl.c:1006)")))

[AI-Ahmed]

The reason is clear, but I was focusing on reading the comments that contain scripts rather than reading the updates. The today's update declared that there is no need to use server configuration anymore...

Workarounds specifying the ip-based url (http://145.38.195.79/) are no longer needed, and may lead to errors.

[TheRazorace]

Hi everyone and thanks for the support. I am now able to download datasets but still unable to load runs if they are not cached:

Example:
run = openml.runs.get_run(1838227, ignore_cache=True)

Error:
SSLError: HTTPSConnectionPool(host='www.openml.org', port=443): Max retries exceeded with url: /api/v1/xml/run/1838227 (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)')))

[joaquinvanschoren]

This works again now.

[PGijsbers]

If you are using openml-python, make sure to upgrade to the latest version (released minutes ago, January 25th, 2025). You can upgrade with python -m pip install --upgrade openml.

With 0.15.1 installed, set environment variable OPENML_SKIP_PARQUET, i.e.,

Linux/MacOS: export OPENML_SKIP_PARQUET=true
Windows: set OPENML_SKIP_PARQUET=true

This stops openml-python from attempting to download parquet files. We recommend to use the feature only until the MinIO production server is operational again.
Without the environment variable set, openml-python is completely functional, but slower.

[TheRazorace]

Hi, just wanted to mention that in Linux, I still have problems with the get_data() function even with the above configuration.

Example:
dataset = openml.datasets.get_dataset(11)
X, y, _, _ = dataset.get_data(dataset_format="dataframe")

Error:
openml.exceptions.OpenMLHashException: Checksum of downloaded file is unequal to the expected checksum 76938608d472f620c170cef9c8c1fa65 when downloading https://openml.org/datasets/0000/0011/dataset_11.arff. Raised when downloading dataset 11.

If I don't use the configuration, even though the dataset is cached, I am still not able to get it.

Error: raise Exception(f"File: {data_file}") from e
Exception: File: /apollo/ida/.cache/openml/org/openml/www/datasets/11/dataset_11.pq

I can do it in MacOS, though, without problems. Is there a reason?

It is not urgent, just reporting it. Thanks