Code

TriePagedAttentionCache#632

Code

Merged

renxida merged 23 commits intonod-ai:mainnod-ai/shark-ai:mainfrom renxida:trie-3renxida/shark-ai:trie-3Copy head branch name to clipboardDec 4, 2024

+943-40

Commits

Commits on Dec 4, 2024

• add unit tests and fix numerous small problems

  renxida

  committed
  a1946b8

  Copy full SHA for a1946b8

• all tests passing

  renxida

  committed
  8437a7f

  Copy full SHA for 8437a7f

• all but publishing working

  renxida

  committed
  24a4313

  Copy full SHA for 24a4313

• naming consistency Page -> Paged

  renxida

  committed
  5c87b2b

  Copy full SHA for 5c87b2b

• passing all tests now

  renxida

  committed
  07d3040

  Copy full SHA for 07d3040

• name and documentation update publish_pages -> publish_pages_for_tokens

  renxida

  committed
  8f83118

  Copy full SHA for 8f83118

• undo accidental edit of messages.py

  renxida

  committed
  5c37666

  Copy full SHA for 5c37666

• remove not-very-useful test case

  renxida

  committed
  802c4e2

  Copy full SHA for 802c4e2

• fix missing rename

  renxida

  committed
  3c7cc6c

  Copy full SHA for 3c7cc6c

• use trie by default

  renxida

  committed
  f59485e

  Copy full SHA for f59485e

• add errmsg for unknown kvcache type

  renxida

  committed
  646c167

  Copy full SHA for 646c167

• fix typoes

  renxida

  committed
  5801d27

  Copy full SHA for 5801d27

• trie doesn't work yet reverting to base for default

  renxida

  committed
  7f6e5a3

  Copy full SHA for 7f6e5a3

• rename cache_type to prefix_sharing_algorithm

  renxida

  committed
  ca29397

  Copy full SHA for ca29397

• add another test case for trie and xfail it

  renxida

  committed
  7abaa7c

  Copy full SHA for 7abaa7c

• fix xpass by fixing a place where i forgot to pass prefix_sharing_algorithm in conftest.py

  renxida

  committed
  1e35634

  Copy full SHA for 1e35634

• organize test cases

  renxida

  committed
  1a419f1

  Copy full SHA for 1a419f1

• add an extend_allocation method to cache allocations and make decode requests use that instead

  renxida

  committed
  85bc5fc

  Copy full SHA for 85bc5fc

• remove xfail because we are now PASSING!

  renxida

  committed
  feb3aa9

  Copy full SHA for feb3aa9

• fix typo; force certain args to be kwargs

  renxida

  committed
  7e07cc2

  Copy full SHA for 7e07cc2

• use a reference counter class to replace int for ref_count in trie node

  renxida

  committed
  d71826f

  Copy full SHA for d71826f

• fixed problem with refcount not having a default

  renxida

  committed
  871b622

  Copy full SHA for 871b622

• use integer ops instead of float with ceil / floor

  renxida

  committed
  c32be80

  Copy full SHA for c32be80