Code

merge ppl.nn.llm#920

Code

Merged

ouonline merged 83 commits intoOpenPPL:masterOpenPPL/ppl.nn:masterfrom ouonline:mergedCopy head branch name to clipboardApr 16, 2024

+26,408-83

Commits

Commits on Sep 1, 2023

• initial import

  openppl-public

  committed
  4d9db83

  Copy full SHA for 4d9db83

• [misc] remove comments

  openppl-public

  committed
  0cbaecb

  Copy full SHA for 0cbaecb

• [fix] pplnn_llm.cc: add missing header

  openppl-public

  committed
  7fadf3f

  Copy full SHA for 7fadf3f

Commits on Sep 4, 2023

• [misc] update build command

  ouonline

  committed
  4a5e29c

  Copy full SHA for 4a5e29c

Commits on Sep 6, 2023

• [doc] update quick start for running model

  

  Alcanderian

  authored and

  ouonline

  committed
  e5bded0

  Copy full SHA for e5bded0

• [opt] alloc cublas workspace in device

  

  Alcanderian

  authored and

  ouonline

  committed
  9d82016

  Copy full SHA for 9d82016

• [opt] only use cublas workspace for sm90

  

  Alcanderian

  authored and

  ouonline

  committed
  77d8fee

  Copy full SHA for 77d8fee

• [doc] add comment

  

  Alcanderian

  authored and

  ouonline

  committed
  e3f1560

  Copy full SHA for e3f1560

Commits on Sep 7, 2023

• [refactor] reafactor rotary position embedding

  

  Alcanderian

  authored and

  ouonline

  committed
  42a8224

  Copy full SHA for 42a8224

• [refactor] refactor parallel ops

  

  Alcanderian

  authored and

  ouonline

  committed
  506cec5

  Copy full SHA for 506cec5

Commits on Sep 8, 2023

• [refactor] remove deprecate code

  Show description for 41afb26
  

  Alcanderian

  authored and

  ouonline

  committed
  41afb26

  Copy full SHA for 41afb26

Commits on Sep 13, 2023

• [doc] ppl.llm system overview

  Alcanderian

  authored
  f84023c

  Copy full SHA for f84023c

• [fix] pic link in system_overview.md

  ouonline

  committed
  b9cec2a

  Copy full SHA for b9cec2a

Commits on Sep 14, 2023

• [fix] fix PPLNN_BUILD_TOOLS

  

  Alcanderian

  authored and

  ouonline

  committed
  76fb017

  Copy full SHA for 76fb017

• [doc] update platform

  

  Alcanderian

  authored and

  ouonline

  committed
  853acef

  Copy full SHA for 853acef

Commits on Sep 26, 2023

• [feature] i8i8 quantize rev0

  Alcanderian

  authored
  cfe8c00

  Copy full SHA for cfe8c00

Commits on Sep 27, 2023

• [feature] add llama benchmark

  

  shengyunrui

  authored and

  ouonline

  committed
  90361a0

  Copy full SHA for 90361a0

Commits on Oct 8, 2023

• [misc] add quant_method ffor benchmark

  

  Alcanderian

  authored and

  ouonline

  committed
  9096f30

  Copy full SHA for 9096f30

• [fix] init allocator in PlainDevice

  ouonline

  committed
  d7ae2bb

  Copy full SHA for d7ae2bb

Commits on Oct 9, 2023

• [doc] add benchmark llama doc

  

  shengyunrui

  authored and

  ouonline

  committed
  58265a1

  Copy full SHA for 58265a1

• [feature] enbale fp6 gemm bias to support internlm

  

  Alcanderian

  authored and

  ouonline

  committed
  f8f3ec2

  Copy full SHA for f8f3ec2

• [fix] orin: use sync allocator for engine device

  ouonline

  committed
  1e1f9ab

  Copy full SHA for 1e1f9ab

Commits on Oct 10, 2023

• [feature] i8i8 gemm/dequant support bias for internlm

  

  Alcanderian

  authored and

  ouonline

  committed
  35ed734

  Copy full SHA for 35ed734

• [opt] use zero-copy on weight quantization

  

  Alcanderian

  authored and

  ouonline

  committed
  3b5c15d

  Copy full SHA for 3b5c15d

Commits on Oct 12, 2023

• [fix] fix compile when nccl disbaled

  

  Alcanderian

  authored and

  ouonline

  committed
  cbf9e56

  Copy full SHA for cbf9e56

• [opt] do not save backup quantized weight and scale

  

  Alcanderian

  authored and

  ouonline

  committed
  656e1ea

  Copy full SHA for 656e1ea

Commits on Oct 16, 2023

• Update build.sh

  Alcanderian

  committed
  ca1714f

  Copy full SHA for ca1714f

Commits on Oct 18, 2023

• [fix] add kv cache shape set, add model config check

  

  shengyunrui

  authored and

  Alcanderian

  committed
  bf99cb3

  Copy full SHA for bf99cb3

Commits on Oct 23, 2023

• [opt] use rowmajor int8 linear

  Alcanderian

  committed
  bde4fcb

  Copy full SHA for bde4fcb

• [opt] support plain mm-policy

  Alcanderian

  committed
  9c11698

  Copy full SHA for 9c11698

Commits on Oct 24, 2023

• [misc] disable cuda by default

  ouonline

  committed
  2c47d92

  Copy full SHA for 2c47d92

Commits on Oct 25, 2023

• [misc] update algo cache api

  Alcanderian

  committed
  a44af9f

  Copy full SHA for a44af9f

Commits on Oct 26, 2023

• [fix] disable cuda benchmark when PPLNN_USE_LLM_CUDA is off

  ouonline

  committed
  a83b20f

  Copy full SHA for a83b20f

Commits on Oct 31, 2023

• [fix] fix benchmark for unaligned input

  Alcanderian

  committed
  b77301b

  Copy full SHA for b77301b

Commits on Nov 6, 2023

• [opt] enable kernel profiling for benchmark_llama

  Alcanderian

  committed
  5a88f04

  Copy full SHA for 5a88f04

Commits on Nov 16, 2023

• add MM_BESTFIT support

  ouonline

  committed
  0effd72

  Copy full SHA for 0effd72

Commits on Dec 14, 2023

• [opt] bump some bugfix and optimizations

  Alcanderian

  committed
  5fd1a0c

  Copy full SHA for 5fd1a0c

• [fix] fix hang when benchmark exit

  Alcanderian

  committed
  b8c0036

  Copy full SHA for b8c0036

• MHCA support flash decoding

  

  yimmmin

  authored and

  Alcanderian

  committed
  92fe27b

  Copy full SHA for 92fe27b

• MHCA support dynamic block size

  

  yimmmin

  authored and

  Alcanderian

  committed
  7766f88

  Copy full SHA for 7766f88

• multi_head_cache_attention

  Alcanderian

  committed
  82786c8

  Copy full SHA for 82786c8

• add chatglm. chatglm2 support 1. add op including gelu, layernorm, rotary2demb ... 2. README mark "nsys do not trace cuda-kernel in nccl multi-thread mode"

  

  shengyunrui

  authored and

  Alcanderian

  committed
  3e577fd

  Copy full SHA for 3e577fd

• mha support 2 dim attention_mask

  

  shengyunrui

  authored and

  Alcanderian

  committed
  15f08f8

  Copy full SHA for 15f08f8

• refactor layernorm & rotary2demb

  Alcanderian

  committed
  625869d

  Copy full SHA for 625869d

• [fix] fix int8 quantize bug when dim not aligned

  Alcanderian

  committed
  f56a122

  Copy full SHA for f56a122

Commits on Dec 15, 2023

• [doc] update docs

  Alcanderian

  committed
  eef9bf9

  Copy full SHA for eef9bf9

Commits on Dec 25, 2023

• [doc] update READMD.md

  Alcanderian

  committed
  4f7702d

  Copy full SHA for 4f7702d

Commits on Dec 27, 2023

• [feature] users can use different devices for input/output and runtime

  ouonline

  committed
  bca4c2e

  Copy full SHA for bca4c2e

Commits on Dec 28, 2023

• [fix] tools build: remove Runtime::GetHostDeviceContext()

  ouonline

  committed
  db0df92

  Copy full SHA for db0df92

Commits on Jan 2, 2024

• [fix] fix decoding_batches in benchmark_llama

  Alcanderian

  committed
  e5755e6

  Copy full SHA for e5755e6

Commits on Jan 3, 2024

• [fix] pplnn_llm.cc: destroy host_device after runtime

  ouonline

  committed
  61a8b16

  Copy full SHA for 61a8b16

• [fix] delete host_device in the corresponding thread env

  ouonline

  committed
  f2015a7

  Copy full SHA for f2015a7

Commits on Jan 5, 2024

• [misc] move infer_func init to CommonInit

  

  xupinjie

  authored and

  Alcanderian

  committed
  d96dae2

  Copy full SHA for d96dae2

• [feature] add GeGLU, SwiGLU, move Reshape into DoExecute

  Alcanderian

  committed
  8af0bd5

  Copy full SHA for 8af0bd5

Commits on Jan 6, 2024

• [doc] add chatglm3

  Alcanderian

  committed
  35c36df

  Copy full SHA for 35c36df

Commits on Jan 17, 2024

• [feature] support pmx model export and import

  

  xupinjie

  authored and

  Alcanderian

  committed
  abc3b9c

  Copy full SHA for abc3b9c

• [feature] check pmx and engine version

  

  xupinjie

  authored and

  Alcanderian

  committed
  41915c8

  Copy full SHA for 41915c8

• [feature] add host device to llm_cuda engine

  Alcanderian

  committed
  4375e12

  Copy full SHA for 4375e12

• [fix] fix ~LlmHostDevice()

  Alcanderian

  committed
  34a7e6a

  Copy full SHA for 34a7e6a

Commits on Jan 18, 2024

• [fix] fix llm_cuda kernel.h

  Alcanderian

  committed
  e66c8fa

  Copy full SHA for e66c8fa

Commits on Jan 19, 2024

• [feature] benchmark support pmx model

  

  xupinjie

  authored and

  Alcanderian

  committed
  e8d9b64

  Copy full SHA for e8d9b64

Commits on Jan 22, 2024

• [fix] fix PPLNN_ENABLE_PMX_MODEL off compile bug

  

  xupinjie

  authored and

  Alcanderian

  committed
  45c1254

  Copy full SHA for 45c1254

Commits on Jan 24, 2024

• [feature] python support init

  ouonline

  committed
  b823e4b

  Copy full SHA for b823e4b

• [kernel] add moe linear op

  

  Eddiewdy

  authored and

  Alcanderian

  committed
  8b64bbb

  Copy full SHA for 8b64bbb

• [feature] add moe_select, moe_reduce, linear

  Show description for aadfd66
  

  Vincent-syr

  authored and

  Alcanderian

  committed
  aadfd66

  Copy full SHA for aadfd66

• [opt] change expert_offset to host tensor

  

  Vincent-syr

  authored and

  Alcanderian

  committed
  236ec44

  Copy full SHA for 236ec44

• [fix] change benchmark input to random number

  

  Vincent-syr

  authored and

  Alcanderian

  committed
  2277361

  Copy full SHA for 2277361

• [refactor] refactor moe ops

  Alcanderian

  committed
  48348b8

  Copy full SHA for 48348b8

• [misc] change mhca api

  Alcanderian

  committed
  5a97a1a

  Copy full SHA for 5a97a1a

Commits on Jan 30, 2024

• [fix] fix rms norm

  Alcanderian

  committed
  5387747

  Copy full SHA for 5387747

Commits on Feb 2, 2024

• [misc] rename MM_BESTFIT => MM_BEST_FIT to be the same as other engines

  ouonline

  committed
  7244a6d

  Copy full SHA for 7244a6d

• [misc] add ppnnn_llm.py

  ouonline

  committed
  079d4a2

  Copy full SHA for 079d4a2

Commits on Feb 6, 2024

• [feature] add Swish op

  

  xupinjie

  authored and

  Alcanderian

  committed
  69004cb

  Copy full SHA for 69004cb

Commits on Feb 28, 2024

• [opt] do not use algo heuristic for sm>=89

  Alcanderian

  committed
  55423df

  Copy full SHA for 55423df

Commits on Mar 5, 2024

• [feature] add Add op

  

  xupinjie

  authored and

  Alcanderian

  committed
  5356412

  Copy full SHA for 5356412

• [feature] resume mha op

  Alcanderian

  committed
  1678262

  Copy full SHA for 1678262

Commits on Mar 11, 2024

• [feature] add new op PositionIndex

  Alcanderian

  committed
  74769a3

  Copy full SHA for 74769a3

• [fix] fix register op

  Alcanderian

  committed
  529c2ce

  Copy full SHA for 529c2ce

Commits on Mar 21, 2024

• [fix] fix debug log

  Alcanderian

  committed
  8787647

  Copy full SHA for 8787647

Commits on Mar 22, 2024

• [feature] export cmake config

  ouonline

  committed
  fc7b41f

  Copy full SHA for fc7b41f

Commits on Apr 15, 2024

• merge ppl.nn.llm

  ouonline

  committed
  6e87bef

  Copy full SHA for 6e87bef

Commits on Apr 16, 2024

• [fix] opencv 4 or later is required to build the classification sample

  ouonline

  committed
  ed3c410

  Copy full SHA for ed3c410

• [fix] put generated pb files into a lib

  ouonline

  committed
  73dbd8d

  Copy full SHA for 73dbd8d