Code

Integrate Flash-Decoding into engine#181

Code

Merged

sunggg merged 30 commits intooctoml:batch-servingoctoml/mlc-llm:batch-servingfrom masahi:flash-decoding-enginemasahi/mlc-llm:flash-decoding-engineCopy head branch name to clipboardFeb 12, 2024

+107-61

Commits

Commits on Jan 29, 2024

• test stub

  masahi

  committed
  30e57a0

  Copy full SHA for 30e57a0

• wip

  masahi

  committed
  6f3429a

  Copy full SHA for 6f3429a

• wip

  masahi

  committed
  97a4366

  Copy full SHA for 97a4366

• wip

  masahi

  committed
  7279cb6

  Copy full SHA for 7279cb6

• compiled

  masahi

  committed
  7348f0e

  Copy full SHA for 7348f0e

• wip

  masahi

  committed
  b692376

  Copy full SHA for b692376

• fix

  masahi

  committed
  1df6cac

  Copy full SHA for 1df6cac

• fix

  masahi

  committed
  8c8872c

  Copy full SHA for 8c8872c

• wip, decode with flash decoding works

  masahi

  committed
  6a8272f

  Copy full SHA for 6a8272f

• all work

  masahi

  committed
  487129c

  Copy full SHA for 487129c

Commits on Jan 30, 2024

• add paged_kv_cache_type option

  masahi

  committed
  8114197

  Copy full SHA for 8114197

• read kv_type from artifact

  masahi

  committed
  2d6c81b

  Copy full SHA for 2d6c81b

• black

  masahi

  committed
  67353b2

  Copy full SHA for 67353b2

• refactor attention backend

  masahi

  committed
  b9e41e1

  Copy full SHA for b9e41e1

• minor clean up

  masahi

  committed
  910e31b

  Copy full SHA for 910e31b

• Integrate flash-decoding into mlc-serve

  masahi

  committed
  ab910f2

  Copy full SHA for ab910f2

• remove --use-vllm-attention

  masahi

  committed
  4c8a75b

  Copy full SHA for 4c8a75b

• wip decode_multi_query integration

  masahi

  committed
  00e1d09

  Copy full SHA for 00e1d09

• temp handling for multi-query logits

  masahi

  committed
  5fbf671

  Copy full SHA for 5fbf671

• remove tmp support for multi-query decode

  masahi

  committed
  2eff7b0

  Copy full SHA for 2eff7b0

• Merge branch 'batch-serving' into flash-decoding-engine

  masahi

  committed
  c51c2a4

  Copy full SHA for c51c2a4

Commits on Jan 31, 2024

• Merge branch 'batch-serving' into flash-decoding-engine

  masahi

  committed
  d7704e2

  Copy full SHA for d7704e2

• typo

  masahi

  committed
  404b305

  Copy full SHA for 404b305

• Merge branch 'batch-serving' into flash-decoding-engine

  masahi

  committed
  d87506c

  Copy full SHA for d87506c

Commits on Feb 1, 2024

• Merge branch 'batch-serving' into flash-decoding-engine

  masahi

  committed
  99af3fb

  Copy full SHA for 99af3fb

• use block size 128 or 64 when possible

  masahi

  committed
  a003965

  Copy full SHA for a003965

Commits on Feb 3, 2024

• Merge branch 'batch-serving' into flash-decoding-engine

  masahi

  committed
  780e244

  Copy full SHA for 780e244

• remove unused var

  masahi

  committed
  56d7a23

  Copy full SHA for 56d7a23

Commits on Feb 8, 2024

• Merge branch 'batch-serving' into flash-decoding-engine

  masahi

  committed
  1b976dc

  Copy full SHA for 1b976dc

• merge fix

  masahi

  committed
  a028c7d

  Copy full SHA for a028c7d