How does exllama_hf use less VRAM?

[bartowski1182]

As far as I can find, all it does is replace the logits and the samples/pipeline (though even that part confuses me), so why does it result in lower consumption? Anyone know where I can read up on it?

[matatonic]

There was a comment in the exllama pull request which went into detail, but essentially from what I understood, the top memory gains were made by not fragmenting memory by not dynamically growing the memory (it allocates max sizes at start). There may be other improvements, but that was the big one afaik.

[78Alpha]

That sounds like memory pinning. A feature used a lot in dream booth training. Also helps with speed.