Code

Add tokenizers support.#1

Code

Merged

LaurentMazare merged 7 commits intomainkyutai-labs/yomikomi:mainfrom tokenizerskyutai-labs/yomikomi:tokenizersCopy head branch name to clipboardDec 4, 2024

+88-17

Commits

Commits on Dec 4, 2024

• Add tokenizers support.

  LaurentMazare

  committed
  53dd806

  Copy full SHA for 53dd806

• Clippy fixes.

  LaurentMazare

  committed
  87ed1e8

  Copy full SHA for 87ed1e8

• Better handling of bos/eos.

  LaurentMazare

  committed
  58b9fe9

  Copy full SHA for 58b9fe9

• Fix the CI.

  LaurentMazare

  committed
  cfd7b75

  Copy full SHA for cfd7b75

• Also bump the dependency version.

  LaurentMazare

  committed
  098a7f1

  Copy full SHA for 098a7f1

• Another CI tweak.

  LaurentMazare

  committed
  a17086c

  Copy full SHA for a17086c

• Ok let's be x86 only for now...

  LaurentMazare

  committed
  d96c898

  Copy full SHA for d96c898