v1.0.0 - 🚀 Major Update: Full Overhaul with Pipeline Design, Caching, and Standardized Workflow

This release marks a significant overhaul of the project, introducing a comprehensive pipeline-based design strategy, caching enhancements, and improved file tracking. Along with these features, we’ve standardized workflows with a robust Makefile and restructured configuration handling, leading to breaking changes in the setup and execution process.

Key Highlights

1. Complete Shift to a Pipeline Design Strategy

• Each processing operation (trim, merge, fade, etc.) is now encapsulated in clearly defined pipeline steps.
• Pipelines are configured and executed through a functional chaining model, ensuring modularity, maintainability, and reusability.

2. Introduction of Downloader Proxy with Caching

• A new DownloaderProxy system enables transparent caching of downloaded files to minimize redundant downloads and improve re-run efficiency.
• Supports both YouTube and S3 downloads with integrated cache management.

3. Enhanced File Tracking with New Dataclass

• A redesigned PipelineData class to:
  • Track downloaded, intermediate, and the final output files separately.
  • Seamlessly integrate with pipeline steps and reduce manual file management.

4. Standardized Makefile for Consistency

• All scripts and workflows now use a standardized Makefile for better usability and automation.
• Makefile includes new commands for running audio, video, or both pipelines, and for testing, cleaning, and building services.

5. Comprehensive Testing Suite with Pytest

• Added a full testing suite using pytest, providing:
  • Unit tests for all pipeline steps.
  • Integration tests for complete audio and video pipeline execution.

Breaking Changes

1. Configuration File Migration

• Configuration has moved from .env files to a new config/pipeline_config.json structure.
• Pipelines now fetch settings from this JSON file, allowing for better organization and support for more complex configurations.

2. Updated Startup Commands

• Scripts are no longer directly executed. Instead, you must use the Makefile:
  • make run: Execute the processor with input choice of which pipelines to run.
  • make run-audio: Run the audio pipeline.
  • make run-video: Run the video pipeline.
  • make run-both: Run both pipelines sequentially.
  • make test: Build and run tests with logging.

Migration Steps

1. Update Your Configuration

• Convert your .env settings to a new pipeline_config.json file.
• Follow the example schema provided in config/example_pipeline_config.json.

2. Switch to Makefile Commands

• Replace custom scripts or old manual commands with the Makefile commands outlined above.