GPTX

Reproducing GPTX from scratch using PyTorch

TODO

• Implement the model architecture

References

1. The architectures
   1. LitGPT: https://github.com/Lightning-AI/litgpt
   2. nanoGPT: https://github.com/karpathy/nanoGPT
   3. llama3 from scratch: https://github.com/naklecha/llama3-from-scratch
   4. LLMs from scratch: https://github.com/rasbt/LLMs-from-scratch
   5. x-transformers: https://github.com/lucidrains/x-transformers
2. minbpe: https://github.com/karpathy/minbpe
3. RLHF
   1. PPO: https://github.com/huggingface/trl/blob/main/trl/trainer/ppo_trainer.py
   2. DPO: https://github.com/huggingface/trl/blob/main/trl/trainer/dpo_trainer.py
4. Use Case
   1. chess GPT: https://adamkarvonen.github.io/machine_learning/2024/01/03/chess-world-models.html