Check for tokenizer in downloaded models directory#364

Merged

mergify[bot] merged 8 commits intoinstructlab:mainfrom khaledsulayman:ks-teacher-model-tokenizer

Nov 14, 2024

+93,770-27

Commits on Nov 14, 2024

Check for tokenizer in downloaded models directory
khaledsulayman
committed
Add error logging for failed tokenizer loading
khaledsulayman
committed
Add unit tests for tokenizer creation
khaledsulayman
committed
Increase Exception specificity for invalid model paths
khaledsulayman
committed
include custom safetensor and gguf format checking based on implementation in instructlab.utils
khaledsulayman
committed
add necessary dependencies for validating gguf files
khaledsulayman
committed
hotfix: change error logging for invalid model formats
khaledsulayman
committed
change HF model path to local model path in functional test
khaledsulayman
committed