From 5156e9e2ff8de2cb1b46800eb3b06883777778ab Mon Sep 17 00:00:00 2001 From: JSchmie Date: Fri, 20 Dec 2024 14:00:41 +0000 Subject: [PATCH] Updated Readme --- README.md | 136 +++++++++++++++++++++++++++++------------------------- 1 file changed, 73 insertions(+), 63 deletions(-) diff --git a/README.md b/README.md index 476e9bb..a090dc8 100644 --- a/README.md +++ b/README.md @@ -1,25 +1,20 @@ # `ScrAIbe: Streamlined Conversation Recording with Automated Intelligence Based Environment` πŸŽ™οΈπŸ§  -Welcome to `ScrAIbe`, a state-of-the-art, [PyTorch](https://pytorch.org/) based multilingual speech-to-text framework designed to generate fully automated transcriptions. +Welcome to `ScrAIbe`, a state-of-the-art, [PyTorch](https://pytorch.org/) based multilingual speech-to-text framework designed to generate fully automated transcriptions. Beyond transcription, ScrAIbe supports advanced functions such as speaker diarization and speaker recognition. πŸš€ Designed as a comprehensive AI toolkit, it uses multiple powerful AI models: - **[Whisper](https://github.com/openai/whisper)**: A general-purpose speech recognition model. -- **[WhisperX](https://github.com/m-bain/whisperX)**: A faster, quantized version of Whisper for enhanced performance on CPU. ⚑ +- **[FasterWhisper](https://github.com/guillaumekln/faster-whisper)**: An optimized version of Whisper for enhanced performance and flexibility. - **[Pyannote-Audio](https://github.com/pyannote/pyannote-audio)**: An open-source toolkit for speaker diarization. πŸ—£οΈ The framework utilizes a PyanNet-inspired pipeline, with the `Pyannote` library for speaker diarization and `VoxCeleb` for speaker embedding. -During post-diarization, each audio segment is processed by the OpenAI `Whisper` model in a transformer encoder-decoder structure. Initially, a CNN mitigates noise and enhances speech. Before transcription, `VoxLingua` identifies the language segment, facilitating Whisper's role in both transcription and text translation. 🌍✨ +During post-diarization, each audio segment is processed by the OpenAI `Whisper` model or its optimized counterpart `FasterWhisper`. Initially, a CNN mitigates noise and enhances speech. Before transcription, `VoxLingua` identifies the language segment, facilitating Whisper's role in both transcription and text translation. 🌍✨ -The following graphic illustrates the whole pipeline: - -
- - -
+--- ## Getting Started πŸš€ @@ -27,13 +22,15 @@ The following graphic illustrates the whole pipeline: Before installing ScrAIbe, ensure you have the following prerequisites: -- **Python**: Version 3.9 or later. +- **Python**: Version 3.11 or later. - **PyTorch**: Version 2.0 or later. -- **CUDA**: A compatible version with your PyTorch Version if you want to use GPU acceleration. +- **CUDA**: A compatible version with your PyTorch version if you want to use GPU acceleration. + +**Note:** PyTorch should be automatically installed with the pip installer. However, if you encounter any issues, consider installing it manually by following the instructions on the [PyTorch website](https://pytorch.org/get-started/locally/). -**Note:** PyTorch should be automatically installed with the pip installer. However, if you encounter any issues, you should consider installing it manually by following the instructions on the [PyTorch website](https://pytorch.org/get-started/locally/). +--- -### Install ScrAIbe +### Install ScrAIbe Install ScrAIbe on your local machine with ease using PyPI. @@ -41,27 +38,38 @@ Install ScrAIbe on your local machine with ease using PyPI. pip install scraibe ``` -If you want to install the development version, you can do so by installing it from GitHub: +For the development version, install directly from GitHub: ```bash pip install git+https://github.com/JSchmie/ScrAIbe.git@develop ``` -or from PyPI using our latest pre-release: +Alternatively, use the latest pre-release: ```bash pip install --pre scraibe ``` -Get started with ScrAIbe today and experience seamless, automated transcription and diarization. +--- + +### Overview of the Pipeline + +Below are images illustrating the ScrAIbe pipeline. Each step, from speaker diarization to transcription, is meticulously handled by our AI models. + +
+ Pipeline Illustration - Dark Mode + Pipeline Illustration - Light Mode +
+ +--- ## Usage -We've developed ScrAIbe with several access points to cater to diverse user needs. +ScrAIbe provides flexible access points to cater to various user needs. ### Python Usage -Gain full control over the functionalities as well as process customization. +Customize and control functionalities programmatically. ```python from scraibe import Scraibe @@ -73,97 +81,99 @@ text = model.autotranscribe("audio.wav") print(f"Transcription: \n{text}") ``` -The `Scraibe` class ensures the models are properly loaded. You can customize the models with various keywords: +#### Customization Options -- **Whisper Models**: Use the `whisper_model` keyword to specify models like `tiny`, `base`, `small`, `medium`, or `large` (`large-v2`, `large-v3`) depending on your accuracy and speed needs. -- **Pyannote Diarization Model**: Use the `dia_model` keyword to change the diarization model. -- **WhisperX**: Set the `whisper_type` to `"whisperX"` for enhanced performance on CPU and use their enhanced models. (Model names are the same) -- **Keyword Arguments**: A variety of different `kwargs` are available: - - `use_auth_token`: Pass a Hugging Face token to the Pyannote backend if you want to use one of the models hosted on their Hugging Face. - - `verbose`: Enable this to add an additional level of verbosity. - - In general, you should be able to input any `kwargs` that you can input in the original Whisper (WhisperX) and Pyannote Python APIs. +- **Whisper Models**: Use the `whisper_model` keyword for models like `tiny`, `base`, `small`, `medium`, or `large` (`large-v2`, `large-v3`, `large-v3-turbo`). +- **Pyannote Diarization Model**: Use the `dia_model` keyword for alternative diarization models. +- **FasterWhisper**: Set `whisper_type` to `"faster-whisper"` for optimized performance. -As input, `autotranscribe` accepts every format compatible with [FFmpeg](https://ffmpeg.org/ffmpeg-formats.html). Examples include `.mp4`, `.mp3`, `.wav`, `.ogg`, `.flac`, and many more. +#### Keyword Arguments -To further control the pipeline of `ScrAIbe`, you can pass almost any keyword argument that is accepted by `Whisper` or `Pyannote`. For more options, refer to the documentation of these frameworks, as their keywords are likely to work here as well. +- `use_auth_token`: Pass a Hugging Face token to access specific models. +- `verbose`: Enable detailed logs. -Here are some examples regarding `diarization` (which relies on the `pyannote` pipeline): +Compatible formats include `.mp4`, `.mp3`, `.wav`, `.ogg`, `.flac`, and more. -- `num_speakers`: Number of speakers in the audio file -- `min_speakers`: Minimum number of speakers in the audio file -- `max_speakers`: Maximum number of speakers in the audio file +```python +text = model.autotranscribe("audio.wav", language="german", num_speakers=2) +``` -Then there are arguments for the transcription process, which uses the "Whisper" model: +--- -- `language`: Specify the language ([list of supported languages](https://github.com/openai/whisper/blob/main/language-breakdown.svg)) -- `task`: Can be either `transcribe` or `translate`. If `translate` is selected, the transcribed audio will be translated to English. +### Command-line Usage -For example: +Run ScrAIbe via CLI: -```python -text = model.autotranscribe("audio.wav", language="german", num_speakers = 2) +```bash +scraibe -f "audio.wav" --language "german" --num_speakers 2 ``` -`Scraibe` also contains the option to just do a transcription: +Get the full list of options: -```python -transcription = model.transcribe("audio.wav") -``` +```bash +scraibe -h +``` -or just do a diarization: +--- -```python -diarization = model.diarization("audio.wav") -``` +## Gradio App 🌐 + +The Gradio App has transitioned to ScrAIbe-WebUI! Run the model via a user-friendly interface in your browser. Learn more at the [ScrAIbe-WebUI GitHub repository](https://github.com/JSchmie/ScrAIbe-WebUI). -Start exploring the powerful features of ScrAIbe and customize it to fit your specific transcription and diarization needs! +--- -### Command-line usage +## Docker Container 🐳 -Next to the Pyhton interface, you can also run ScrAIbe using the command-line interface: +For CLI-based operations, use the `hadr0n/scraibe` Docker container. Pull the image: ```bash -scraibe -f "audio.wav" --language "german" --num_speakers 2 +docker pull hadr0n/scraibe ``` -For the full list of options, run: +Run CLI commands inside the container: ```bash -scraibe -h +docker exec -it scraibe -f /data/audio.wav --language german --num_speakers 2 ``` -This will display a comprehensive list of all command-line options, allowing you to tailor ScrAIbe’s functionality to your specific needs. +Mount your local directory: -## Gradio App 🌐 +```bash +docker run -d --name scraibe-container -v $(pwd):/data hadr0n/scraibe +``` -The Gradio App is now part of ScrAIbe-WebUI! This user-friendly interface enables you to run the model without any coding knowledge. You can easily run the app in your browser and upload your audio files, or make the framework available on your network and run it on your local machine. πŸš€ +List all CLI options: -All functionalities previously available in the Gradio App are now part of the ScrAIbe-WebUI. For more information and detailed instructions, visit the [ScrAIbe-WebUI GitHub repository](https://github.com/JSchmie/ScrAIbe-WebUI). +```bash +docker exec -it scraibe-container scraibe -h +``` -## Docker Container 🐳 +--- -ScrAIbe's Docker containers have also moved to ScrAIbe-WebUI! This option is especially useful if you want to run the model on a server or if you would like to use the GPU without dealing with CUDA. +## Documentation πŸ“š -All Docker container functionalities are now part of ScrAIbe-WebUI. For more information and detailed instructions on how to use the Docker containers, please visit the [ScrAIbe-WebUI GitHub repository](https://github.com/JSchmie/ScrAIbe-WebUI). +Explore detailed guides and advanced tips on our [documentation page](https://jschmie.github.io/ScrAIbe/). --- -With these changes, ScrAIbe focuses on its core functionalities while the enhanced Gradio App and related Docker containers are now part of ScrAIbe-WebUI. Enjoy a more streamlined and powerful transcription experience! πŸŽ‰ +## Releases -## Documentation πŸ“š +For the latest updates, including bug fixes, dependency changes, and new features, refer to the [ScrAIbe Release Notes](https://github.com/JSchmie/ScrAIbe/releases). Stay informed about improvements like the transition to `FasterWhisper`, enhanced Torch configuration, and new CLI features. -For comprehensive guides, detailed instructions, and advanced usage tips, visit our [documentation page](https://jschmie.github.io/ScrAIbe/). Here, you will find everything you need to make the most out of ScrAIbe. +--- ### Contributions 🀝 We warmly welcome contributions from the community! Whether you’re fixing bugs, adding new features, or improving documentation, your help is invaluable. Please see our [Contributing Guidelines](./CONTRIBUTING.md) for more information on how to get involved and make your mark on ScrAIbe-WebUI. +--- ### License πŸ“œ ScrAIbe-WebUI is proudly open source and licensed under the GPL-3.0 license. This promotes a collaborative and transparent development process. For more details, see the [LICENSE](./LICENSE) file in this repository. +--- + ## Acknowledgments Special thanks go to the [KIDA](https://www.kida-bmel.de/) project and the [BMEL (Bundesministerium fΓΌr ErnΓ€hrung und Landwirtschaft)](https://www.bmel.de/EN/Home/home_node.html), especially to the AI Consultancy Team.