Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Refactor to Introduce Backend Abstraction #2011

Closed
Changes from 1 commit
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
50fcfc0
modify parallelization strategy
zhenglongjiepheonix Aug 14, 2024
4114d3b
only support model id in api now
zhenglongjiepheonix Aug 14, 2024
c689402
more comments
zhenglongjiepheonix Aug 15, 2024
252c3b7
more comments
zhenglongjiepheonix Aug 16, 2024
1be77ed
Merge remote-tracking branch 'upstream/main' into longjie/generalize_…
zhenglongjiepheonix Aug 16, 2024
22d6766
address comments
zhenglongjiepheonix Aug 20, 2024
febac9b
remove idle runner
zhenglongjiepheonix Aug 20, 2024
bf99175
fix
zhenglongjiepheonix Aug 20, 2024
4d9d036
format
zhenglongjiepheonix Aug 20, 2024
44a87f4
more comments
zhenglongjiepheonix Aug 26, 2024
513d516
generalize api & add backend abstraction
zhenglongjiepheonix Aug 26, 2024
8335a35
fix
zhenglongjiepheonix Aug 27, 2024
d051217
copyright
zhenglongjiepheonix Aug 27, 2024
6b03855
fix api
zhenglongjiepheonix Aug 28, 2024
6466ccc
Merge remote-tracking branch 'upstream/main' into longjie/add_backend…
zhenglongjiepheonix Aug 29, 2024
b4166ac
move weights intialization inside post process
zhenglongjiepheonix Aug 29, 2024
576104c
seperate meta update and parallel layer construction
zhenglongjiepheonix Aug 30, 2024
8bbc2e9
move weight intialization & binding inside backend
zhenglongjiepheonix Sep 2, 2024
d68df89
add weights tying for nanotron backend
zhenglongjiepheonix Sep 2, 2024
c752e29
fix
zhenglongjiepheonix Sep 3, 2024
82d1cf9
resolve
zhenglongjiepheonix Sep 3, 2024
3a1a195
fix
zhenglongjiepheonix Sep 3, 2024
b5b371f
fix conflict
zhenglongjiepheonix Sep 20, 2024
0ff39bb
address comments
zhenglongjiepheonix Sep 20, 2024
5137f68
address comments
zhenglongjiepheonix Sep 20, 2024
9dd77de
fix
zhenglongjiepheonix Sep 20, 2024
a375b6d
fix
zhenglongjiepheonix Sep 20, 2024
40880a3
fix
zhenglongjiepheonix Sep 20, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix
zhenglongjiepheonix committed Sep 20, 2024
commit a375b6df0cab7363bccf802056ad457ea9b057dc
3 changes: 0 additions & 3 deletions optimum/fx/parallelization/backend/base.py
Original file line number Diff line number Diff line change
@@ -99,7 +99,6 @@ def create_parallel_cross_entropy(
else:
return sharded_cross_entropy_wrapper_fn(process_group=parallel_ctx.tp_group)

@abstractmethod
def pre_process(self, graph_module: GraphModule, ctx: "ParallelExecutionCtx", config: "Config") -> GraphModule:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's a config?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a Config is a data class which records static configurations during the whole process

"""
Mark tie information right before we run passes because dynamo tracing will alter the parameter name while our
@@ -115,15 +114,13 @@ def pre_process(self, graph_module: GraphModule, ctx: "ParallelExecutionCtx", co
param_meta.tied_to = parameter_mp[key]
return graph_module

@abstractmethod
def post_process(self, graph_module: GraphModule, ctx: "ParallelExecutionCtx", config: "Config") -> nn.Module:
"""
This method is called after the parallelization passes have been applied. It is used to perform any backend-specific
post-processing on the graph module.
"""
return graph_module

@abstractmethod
def init_parallelization_pass_pipeline(self) -> PassPipeline:
"""
Ensemble a pass pipeline which contains the following passes: