Code

Integrate vllm/ns into llm-on-ray#267

Code

Open

jiafuzha wants to merge 86 commits intointel:mainintel/llm-on-ray:mainfrom jiafuzha:vllm-ns-merged-209-7d49516jiafuzha/llm-on-ray:vllm-ns-merged-209-7d49516Copy head branch name to clipboard

+81,963-72

Commits

Commits on Apr 17, 2024

• add benchmark run script, visualize script

  KepingYan

  committed
  d2d1f20

  Copy full SHA for d2d1f20

Commits on Apr 26, 2024

• upd

  KepingYan

  committed
  88cc01e

  Copy full SHA for 88cc01e

Commits on May 7, 2024

• update multi replicas

  KepingYan

  committed
  083ae60

  Copy full SHA for 083ae60

Commits on May 8, 2024

• use --result-dir to parse results

  KepingYan

  committed
  4c6fa74

  Copy full SHA for 4c6fa74

• fix ci proxy

  KepingYan

  committed
  1b3b13a

  Copy full SHA for 1b3b13a

Commits on May 9, 2024

• add test ci

  KepingYan

  committed
  184e00e

  Copy full SHA for 184e00e

• add license

  KepingYan

  committed
  bd85b7d

  Copy full SHA for bd85b7d

• fix

  KepingYan

  committed
  38c52ed

  Copy full SHA for 38c52ed

• fix

  KepingYan

  committed
  78dc091

  Copy full SHA for 78dc091

Commits on May 10, 2024

• add autoscaling config

  KepingYan

  committed
  7cc0de0

  Copy full SHA for 7cc0de0

• fix ci

  KepingYan

  committed
  e241b25

  Copy full SHA for e241b25

• fix ci

  KepingYan

  committed
  3eb1c08

  Copy full SHA for 3eb1c08

• add package matplotlib

  KepingYan

  committed
  882ff4d

  Copy full SHA for 882ff4d

• verify CI test

  KepingYan

  committed
  21994cd

  Copy full SHA for 21994cd

Commits on May 11, 2024

• verify CI test

  KepingYan

  committed
  d688804

  Copy full SHA for d688804

Commits on May 13, 2024

• create assets folder to place pictures

  KepingYan

  committed
  c8eabbc

  Copy full SHA for c8eabbc

• verify CI test

  KepingYan

  committed
  3905082

  Copy full SHA for 3905082

• support openai autoscaling

  KepingYan

  committed
  97ec06a

  Copy full SHA for 97ec06a

• remove

  KepingYan

  committed
  606f286

  Copy full SHA for 606f286

Commits on May 16, 2024

• integrate vllm and ns

  Show description for 55c1dd1
  jiafuzha

  committed
  55c1dd1

  Copy full SHA for 55c1dd1

Commits on May 17, 2024

• update config file

  KepingYan

  committed
  e709010

  Copy full SHA for e709010

• integrate vllm and ns

  Show description for 5b1bd85
  jiafuzha

  committed
  5b1bd85

  Copy full SHA for 5b1bd85

• integrate vllm and ns

  Show description for eb71ace
  jiafuzha

  committed
  eb71ace

  Copy full SHA for eb71ace

• remove .eggs

  Show description for a969f7f
  jiafuzha

  committed
  a969f7f

  Copy full SHA for a969f7f

• integration adjustment

  Show description for 1b6aba3
  jiafuzha

  committed
  1b6aba3

  Copy full SHA for 1b6aba3

Commits on May 20, 2024

• llm on ray deployed

  Show description for ce3ac61
  jiafuzha

  committed
  ce3ac61

  Copy full SHA for ce3ac61

• llm on ray deployed

  Show description for 213ad89
  jiafuzha

  committed
  213ad89

  Copy full SHA for 213ad89

Commits on May 21, 2024

• llm on ray deployed

  Show description for 9b4884f
  jiafuzha

  committed
  9b4884f

  Copy full SHA for 9b4884f

• more doc

  Show description for 3cb6f64
  jiafuzha

  committed
  3cb6f64

  Copy full SHA for 3cb6f64

• merge with master

  Show description for 3f9ba62
  jiafuzha

  committed
  3f9ba62

  Copy full SHA for 3f9ba62

• more doc for installing vllm ext

  Show description for f6d60be
  jiafuzha

  committed
  f6d60be

  Copy full SHA for f6d60be

• Merge remote-tracking branch 'keping/test_benchmark_script' into vllm-ns-perf-test

  jiafuzha

  committed
  04cddcf

  Copy full SHA for 04cddcf

• Merge remote-tracking branch 'keping/autoscaling_config' into vllm-ns-perf-test

  jiafuzha

  committed
  d0d40dd

  Copy full SHA for d0d40dd

Commits on May 24, 2024

• bug fix

  Show description for 24cc480
  jiafuzha

  committed
  24cc480

  Copy full SHA for 24cc480

Commits on May 27, 2024

• save

  Show description for 295186e
  jiafuzha

  committed
  295186e

  Copy full SHA for 295186e

• add vllm-ext/requirements.txt

  Show description for 875aa89
  jiafuzha

  committed
  875aa89

  Copy full SHA for 875aa89

• add CMakeLists.txt

  Show description for 2a462ea
  jiafuzha

  committed
  2a462ea

  Copy full SHA for 2a462ea

• changed benchmarks

  Show description for a105321
  jiafuzha

  committed
  a105321

  Copy full SHA for a105321

Commits on May 30, 2024

• tuned graph build

  Show description for 6aa0540
  jiafuzha

  committed
  6aa0540

  Copy full SHA for 6aa0540

Commits on May 31, 2024

• graph build time reduced

  Show description for 7d6d3b4
  jiafuzha

  committed
  7d6d3b4

  Copy full SHA for 7d6d3b4

• graph build time reduced

  Show description for 473671e
  jiafuzha

  committed
  473671e

  Copy full SHA for 473671e

Commits on Jun 4, 2024

• configurable perf stats and copy quant config automatically

  Show description for 1a88edd
  jiafuzha

  committed
  1a88edd

  Copy full SHA for 1a88edd

Commits on Jun 5, 2024

• save test script

  Show description for dfd26b0
  jiafuzha

  committed
  dfd26b0

  Copy full SHA for dfd26b0

Commits on Jun 6, 2024

• add max_batched_tokens parameter

  Show description for 65c816f
  jiafuzha

  committed
  65c816f

  Copy full SHA for 65c816f

Commits on Jun 12, 2024

• adjustment and ray-vllm-examples

  Show description for 89936d3
  jiafuzha

  committed
  89936d3

  Copy full SHA for 89936d3

Commits on Jun 17, 2024

• perf tuned and improved by disable mmap for multiple instances

  Show description for 4f088e2
  jiafuzha

  committed
  4f088e2

  Copy full SHA for 4f088e2

Commits on Jun 19, 2024

• remove unnecessary thread sync in kernels

  Show description for 597d83d
  jiafuzha

  committed
  597d83d

  Copy full SHA for 597d83d

Commits on Jun 24, 2024

• merged ns PR 209 7d49516

  Show description for e06e53b
  jiafuzha

  committed
  e06e53b

  Copy full SHA for e06e53b

Commits on Jun 25, 2024

• change order of loop, batch size first, then iteration

  Show description for b093d3f
  jiafuzha

  committed
  b093d3f

  Copy full SHA for b093d3f

Commits on Jun 26, 2024

• modified some examples

  Show description for 423fc10
  jiafuzha

  committed
  423fc10

  Copy full SHA for 423fc10

• add more parameters for vllm-ns test

  Show description for 96657ce
  JoshuaL3000

  committed
  96657ce

  Copy full SHA for 96657ce

• Merge remote-tracking branch 'refs/remotes/origin/vllm-ns-merged-209-7d49516' into vllm-ns-merged-209-7d49516

  jiafuzha

  committed
  2336966

  Copy full SHA for 2336966

• add more parameters for vllm-ns test

  Show description for f1d06d9
  JoshuaL3000

  committed
  f1d06d9

  Copy full SHA for f1d06d9

• add more parameters for vllm-ns test

  Show description for 34664ed
  

  JoshuaL3000

  authored and

  jiafuzha

  committed
  34664ed

  Copy full SHA for 34664ed

Commits on Jun 27, 2024

• merged with master

  Show description for 4782617
  jiafuzha

  committed
  4782617

  Copy full SHA for 4782617

• prevent quantization being messed-up with multiple processes

  Show description for 04b7582
  jiafuzha

  committed
  04b7582

  Copy full SHA for 04b7582

• fix merge error

  Show description for b791a1d
  jiafuzha

  committed
  b791a1d

  Copy full SHA for b791a1d

• rename py to sh

  Show description for 79e5daf
  jiafuzha

  committed
  79e5daf

  Copy full SHA for 79e5daf

• fix formatting issue

  Show description for 2c9b287
  jiafuzha

  committed
  2c9b287

  Copy full SHA for 2c9b287

• fix formatting issue

  Show description for 5ac7907
  jiafuzha

  committed
  5ac7907

  Copy full SHA for 5ac7907

• fix merge error

  Show description for 19fc069
  JoshuaL3000

  committed
  19fc069

  Copy full SHA for 19fc069

• Merge remote-tracking branch 'refs/remotes/origin/vllm-ns-perf-test' into vllm-ns-perf-test

  jiafuzha

  committed
  76fe811

  Copy full SHA for 76fe811

Commits on Jun 28, 2024

• add vllm-ns ci

  Show description for 5760c65
  jiafuzha

  committed
  5760c65

  Copy full SHA for 5760c65

• remove unnecessary logs

  Show description for 30efd3f
  jiafuzha

  committed
  30efd3f

  Copy full SHA for 30efd3f

• remove some debug code

  Show description for 1d9b4e3
  jiafuzha

  committed
  1d9b4e3

  Copy full SHA for 1d9b4e3

• add '--privileged' to docker run

  Show description for a14a146
  jiafuzha

  committed
  a14a146

  Copy full SHA for a14a146

• set unlimited max lock memory for neural speed engine

  Show description for 4f59cb8
  jiafuzha

  committed
  4f59cb8

  Copy full SHA for 4f59cb8

• merged with master

  Show description for 4df4f85
  jiafuzha

  committed
  4df4f85

  Copy full SHA for 4df4f85

Commits on Jul 2, 2024

• llama-3-8B support

  Show description for e781d0b
  jiafuzha

  committed
  e781d0b

  Copy full SHA for e781d0b

Commits on Jul 5, 2024

• extend token length limit to 8192 for mha

  Show description for af7730a
  jiafuzha

  committed
  af7730a

  Copy full SHA for af7730a

• extend token length limit to 8192 for mha

  Show description for a92f019
  jiafuzha

  committed
  a92f019

  Copy full SHA for a92f019

• extend token length limit to 8192 for mha (fix) and support different threads for prompt decoding and next token decoding

  Show description for 77ee207
  jiafuzha

  committed
  77ee207

  Copy full SHA for 77ee207

• extend token length limit to 8192 for mha (fix) and support different threads for prompt decoding and next token decoding

  Show description for 5154887
  jiafuzha

  committed
  5154887

  Copy full SHA for 5154887

Commits on Jul 9, 2024

• add llama3 for plain cpu

  Show description for f8e51a2
  jiafuzha

  committed
  f8e51a2

  Copy full SHA for f8e51a2

Commits on Jul 11, 2024

• benchmark idc simple/medium/complex/verycomplex prompts

  Show description for 4ab3b0a
  jiafuzha

  committed
  4ab3b0a

  Copy full SHA for 4ab3b0a

Commits on Jul 12, 2024

• benchmark idc simple/medium/complex/verycomplex prompts

  Show description for 5476705
  jiafuzha

  committed
  5476705

  Copy full SHA for 5476705

• benchmark idc simple/medium/complex/verycomplex prompts

  Show description for ea02ef3
  jiafuzha

  committed
  ea02ef3

  Copy full SHA for ea02ef3

Commits on Jul 16, 2024

• add inference_engine resource and app_router resource to distinct engine worker and router worker since they have different resource config

  Show description for 7952602
  jiafuzha

  committed
  7952602

  Copy full SHA for 7952602

• Merge remote-tracking branch 'refs/remotes/origin/vllm-ns-merged-209-7d49516' into vllm-ns-merged-209-7d49516

  jiafuzha

  committed
  c5c6a12

  Copy full SHA for c5c6a12

• enhanced benchmark script to support IDC test data

  Show description for 58ad614
  jiafuzha

  committed
  58ad614

  Copy full SHA for 58ad614

• updated ray startup script to add resources for app_router and inference_engine

  Show description for 724eced
  jiafuzha

  committed
  724eced

  Copy full SHA for 724eced

• fix first token latency and next token latency issue in open-ai mode in benchmark script

  Show description for 7a4d7fd
  jiafuzha

  committed
  7a4d7fd

  Copy full SHA for 7a4d7fd

• updated ray startup script to add resources for app_router and inference_engine

  Show description for 5a59427
  jiafuzha

  committed
  5a59427

  Copy full SHA for 5a59427

• addressed some review comments

  Show description for d5694c2
  jiafuzha

  committed
  d5694c2

  Copy full SHA for d5694c2

• fix lint issue

  Show description for 10f0f7c
  jiafuzha

  committed
  10f0f7c

  Copy full SHA for 10f0f7c

Commits on Jul 17, 2024

• address review comment by getting number of threads from ray num-cpus and setting threads for cases not with ray

  Show description for 52c3451
  jiafuzha

  committed
  52c3451

  Copy full SHA for 52c3451