You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently, the prefix sharing algorithm is configured from config.json, which makes it necessary to edit it every time we export a model before server would work.
config.json should be only used for exported config keys from sharktank.export_paged_llm_v1
server.py should take additional options either from a separate config file or from the commandline arguments.
Ideally we figure out some project-wide standards for configuration management (e.g. json / yml files in ~/.shortfin with cmdline options to override them).
The text was updated successfully, but these errors were encountered:
renxida
changed the title
[shortfin llm] Separate config setup for server
[shortfin llm] Separate server configs and exported model configs
Dec 26, 2024
Currently, the prefix sharing algorithm is configured from config.json, which makes it necessary to edit it every time we export a model before server would work.
config.json should be only used for exported config keys from sharktank.export_paged_llm_v1
server.py should take additional options either from a separate config file or from the commandline arguments.
Ideally we figure out some project-wide standards for configuration management (e.g. json / yml files in ~/.shortfin with cmdline options to override them).
The text was updated successfully, but these errors were encountered: