Skip to content

Commit

Permalink
Update chapters/en/Unit 3 - Vision Transformers/KnowledgeDistillation…
Browse files Browse the repository at this point in the history
….mdx

Co-authored-by: Merve Noyan <[email protected]>
  • Loading branch information
asusevski and merveenoyan authored Feb 10, 2024
1 parent 73316e1 commit 255c60f
Showing 1 changed file with 1 addition and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ I am very confident that it *is* Draco Malfoy", this gives you some information
This is precisely the kind of information that gets passed down to our student model under the Knowledge Distillation paradigm.

## Distilling the Knowledge in a Neural Network
In the paper [*Distilling the Knowledge in a Neural Network*](https://arxiv.org/abs/1503.02531), Hinton et al. introduced the training methodology known as Knowledge Distillation,
In the paper [*Distilling the Knowledge in a Neural Network*](https://arxiv.org/abs/1503.02531), Hinton et al. introduced the training methodology known as knowledge distillation,
taking inspiration *insects*, of all places. The analogy being that, just as insects transition from larval to adult forms optimized for different tasks, large-scale machine learning models can
initially be cumbersome, like larvae, for extracting structure from data, but can distill their knowledge into smaller, more efficient models for deployment.

Expand Down

0 comments on commit 255c60f

Please sign in to comment.