[AutoParallel]:intermediate api supports LoRA #70539
Conversation
|
你的PR提交成功，感谢你对开源项目的贡献! |
blacksheep-Aristotle
force-pushed
the
intermediate_api
branch
from
973cdbf to
3d0b643
Compare
| "llama.layers.*.mlp.up_proj": dist.ColWiseParallel(), | ||
| "llama.layers.*.mlp.down_proj": dist.RowWiseParallel(), | ||
| "lm_head.weight": dist.ColWiseParallel(), | ||
| f"{prefix}llama.layers.*.self_attn.o_proj.lora_B": dist.RowWiseParallel(), |
There was a problem hiding this comment.
because the input of o_proj is RowWiseParallel, so we shall config f"{prefix}llama.layers.*.self_attn.o_proj.lora_A": dist.RowWiseParallel(), rather than q_proj.lora_B?
| gather_output=True | ||
| ), | ||
| "llama.layers.*.self_attn.k_proj": dist.ColWiseParallel( | ||
| f"{prefix}llama.layers.*.self_attn.q_proj.lora_A": dist.ColWiseParallel(), |
There was a problem hiding this comment.
because we shall keep result of lora is ColWiseParallel, so we shall config f"{prefix}llama.layers.*.self_attn.q_proj.lora_B": dist.ColWiseParallel(), rather than q_proj.lora_A?
There was a problem hiding this comment.
same issue of k_proj.lora_A and v_proj.lora_A
There was a problem hiding this comment.
now lora_A is RowWiseParallel when layer is RowWiseParallel.
lora_B is ColWiseParallel when layer is ColWiseParallel
There was a problem hiding this comment.
lora_B is ColWiseParallel when layer is RowWiseParallel
lora_B is ColWiseParallel when layer is ColWiseParallel?
| shard_param_list.add("weight") | ||
| shard_param_list.add("bias") |
There was a problem hiding this comment.
if shard_param_list is not [], should not add weight and bias? consider the plan of shard_param_list is different from weight and bias.
There was a problem hiding this comment.
shard_param_list will add weight or bias only when shard_param_list is []
blacksheep-Aristotle
force-pushed
the
intermediate_api
branch
from
93229ac to
92b162d
Compare
blacksheep-Aristotle
changed the title
PR Category
Auto Parallel
PR Types
New features
Description
中层api支持lora&添加lora 单测
Pcard-67164