Implement conversion from CutSet to HuggingFace dataset #1398
Conversation
So far, conversion from CutSet containing MonoCut and single-source audio to HuggingFace dataset.
lhotse/cut/set.py
Outdated
| def has_one_audio_source(self) -> bool: | ||
| return all(len(cut.recording.sources) == 1 for cut in self) | ||
|
|
||
| def _convert_cuts_info_to_hf(self) -> Tuple[Dict[str, Any], Dict[str, Any]]: |
There was a problem hiding this comment.
Since this file is pretty large and the functionality introduced is fairly well isolated, I suggest to change every method into a function (except the main HF export API) and move them into a separate file lhotse/hf.py. Keep the HF datasets imports local like they are. The body of to_huggingface_dataset(self) should import and call def export_cuts_to_hf(cuts) from that file.
lhotse/cut/set.py
Outdated
| Converts cut supervisions into a dictionary compatible with HuggingFace datasets format. | ||
|
|
||
| :param has_speaker: Whether the supervisions have speaker information. | ||
| :param has_language: Whether the supervisions have language information. |
There was a problem hiding this comment.
Shouldn't these be auto-deducted?
domklement
force-pushed
the
feature/export-to-huggingface-dataset
branch
from
b4a5756 to
6247d99
Compare
domklement
force-pushed
the
feature/export-to-huggingface-dataset
branch
from
6247d99 to
aa981c7
Compare
lhotse/cut/set.py
Outdated
| Converts a CutSet to a HuggingFace Dataset. Currently, only MonoCut with one recording source is supported. | ||
| Other cut types will be supported in the future. | ||
|
|
||
| More detailed description is in lhotse/hf.py |
There was a problem hiding this comment.
One last comment: can you copy (duplicate) the documentation from export_to_hf here? It will be more user friendly for people using interactive suggestions (notebooks, IDEs)
There was a problem hiding this comment.
That's a good point. It's done.
|
Thanks!!! |
…h#1398) * Implement conversion from CutSet to HuggingFace dataset So far, conversion from CutSet containing MonoCut and single-source audio to HuggingFace dataset. * Refactor * Add docs to set.py --------- Co-authored-by: Piotr Żelasko <[email protected]>
This PR implements a simple conversion from a CutSet containing MonoCuts and single-source Recording to a HuggingFace dataset.
CutSet.to_huggingface_dataset: None -> DataSetconverts the cutset into one of two formats, depending on whether all the cuts contain only one supervision or multiple of them. The formats are described in the method's docstring.