Update CometMLTracker to allow re-using experiment

[Lothiraldan]

What does this PR do?

Update CometMLTracker to use new comet_ml.start function to create Experiments, this way end-users can create online, offline experiments, append data to an existing experiment and it also automatically re-use a running experiment if one is present rather than creating a new one.

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

[BenjaminBossan]

Thanks for this update. Since you mention it's a new feature, could it happen that this will break existing scripts for users that use an older version? If yes, how about adding a check and raising a helpful error message that users should upgrade the comet package?

[Lothiraldan]

@BenjaminBossan Good point, but I'm not sure where it would be best to add the helpful error message.

I tried adding it where we check if Comet is installed https://github.com/comet-ml/accelerate/blob/comet-ml-tracker-update/src/accelerate/utils/imports.py#L251 but the downside is that the message would be shown for all users that have comet installed even if they don't use the tracker.

I tried adding it in the Tracker itself https://github.com/comet-ml/accelerate/blob/comet-ml-tracker-update/src/accelerate/utils/imports.py#L251 but that would breaks if the user use log_with="all" as the Tracker will be picked but raise an Exception when being created:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Cell In[7], line 2
      1 accelerator = Accelerator(log_with="all")
----> 2 accelerator.init_trackers(
      3     project_name="comet-example-accelerate-notebook", config=hyper_params
      4 )
      5 device = accelerator.device
      8 class Net(nn.Module):

File ~/.../accelerate/src/accelerate/accelerator.py:723, in Accelerator.on_main_process.<locals>._inner(*args, **kwargs)
    722 def _inner(*args, **kwargs):
--> 723     return PartialState().on_main_process(function)(*args, **kwargs)

File ~/.../accelerate/src/accelerate/accelerator.py:2703, in Accelerator.init_trackers(self, project_name, config, init_kwargs)
   2699             self.trackers.append(
   2700                 tracker_init(project_name, self.logging_dir, **init_kwargs.get(str(tracker), {}))
   2701             )
   2702         else:
-> 2703             self.trackers.append(tracker_init(project_name, **init_kwargs.get(str(tracker), {})))
   2704 if config is not None:
   2705     for tracker in self.trackers:

File ~/.../accelerate/src/accelerate/tracking.py:82, in on_main_process.<locals>.execute_on_main_process(self, *args, **kwargs)
     79 @wraps(function)
     80 def execute_on_main_process(self, *args, **kwargs):
     81     if getattr(self, "main_process_only", False):
---> 82         return PartialState().on_main_process(function)(self, *args, **kwargs)
     83     else:
     84         return function(self, *args, **kwargs)

File ~/.../accelerate/src/accelerate/tracking.py:426, in CometMLTracker.__init__(self, run_name, **kwargs)
    424 print("COMET VERSION", comet_version)
    425 if compare_versions(comet_version, "<", "4.41.0"):
--> 426     raise RuntimeError("comet_ml is installed but is too old, please update with `pip install -U comet_ml>=3.41.0`")
    428 from comet_ml import start
    430 self.writer = start(project_name=run_name, **kwargs)

RuntimeError: comet_ml is installed but is too old, please update with `pip install -U comet_ml>=3.41.0`

Do you have a preference or another idea when to show the error message?

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[muellerzr]

Thanks for looking into bringing such a nice feature! Let's be careful to not set limits in our integration dependencies, we tend to keep these very open unless for good reason. As an alternative, let's do some version guarding

[muellerzr]

The way we should do this is via an import check. Do we know if it's 3.47.6 that introduced this? Then based on the version either use Experiment/end or start/flush

[Lothiraldan]

We released the last fixes to comet_ml.start in the version 3.41.0 back in April 2024 (https://pypi.org/project/comet-ml/3.41.0/). I have considered using Experiment/start based on the package version and while the new API is 99% compatible, there is new accepted by start (like the experiment name or tags). That could make documenting the callback slightly more complicated.
I would personally prefer to set the minimum version of comet to 3.41.0 as it would make it clearer for end-user what parameters they can pass. But it's only a small preference, I'm also fine with having a different behavior based on the comet version.

[muellerzr]

< 1 year is a bit too soon to just drop IMO. Let's please do both. Especially as 50% of your pypi downloads still are < 3.4.1