Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enhance README: Shields to capture the attention of potential contributors #9

Open
wants to merge 98 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
98 commits
Select commit Hold shift + click to select a range
1799633
feat: llama2 7b with qlora
jazelly Apr 20, 2024
20f8cf8
restructure
jazelly Apr 22, 2024
b3f6033
feat: add llama3 and phi3
jazelly Apr 25, 2024
772e8ae
chore: add submodule slowllama
jazelly May 2, 2024
f2c1b89
fix: path issues in slowllama
jazelly May 2, 2024
4999f7d
ignore update
jazelly May 4, 2024
6384fd5
merge
jazelly May 4, 2024
ffb34bd
feat: support finetune llama in gguf
jazelly May 5, 2024
013b653
chore: update README
jazelly May 5, 2024
c0bfdf0
chore: remove train text
jazelly May 5, 2024
7026e02
chore: update readme of slowllama
jazelly May 5, 2024
e356610
fix: submodule mv
jazelly May 5, 2024
e83f191
update submodules
jazelly May 5, 2024
62e6301
fix: submodule
jazelly May 5, 2024
16e8a9e
chore: setup slowllama
jazelly May 5, 2024
8c52b8e
feat: load llama 2
jazelly May 8, 2024
1b80d96
chore: update ignore
jazelly May 8, 2024
9a5e79f
chore: update readme
jazelly May 9, 2024
5b4484f
chore: update readme
jazelly May 9, 2024
7cbfab8
feat: support slowllama
jazelly May 11, 2024
864179f
wip: client
jazelly May 12, 2024
61efbe6
feat: expose as ui and server
jazelly May 24, 2024
b76877f
fix: path resolution
jazelly May 25, 2024
ac807f4
fix: worker thread and worker manager works in unit test
jazelly May 25, 2024
dc46309
main: ui update on logo and footer
jazelly May 25, 2024
d53f96a
workspace-setting: remove workspace pfp
jazelly May 25, 2024
db2a442
main: add upload button to global
jazelly May 25, 2024
599414d
chore: wording in default chat
jazelly May 26, 2024
da13cb7
wip: dataset upload
jazelly May 26, 2024
f01df38
feat: upload dataset by chunk
jazelly May 27, 2024
b209f67
feat: scrape url from hf dataset
jazelly May 28, 2024
af36875
doc: update readme
jazelly May 28, 2024
158ef13
fix: polish scrape hf link
jazelly May 28, 2024
da0d539
remove embed chat widget
jazelly May 28, 2024
10bbf93
test: add coverage for unit-files
jazelly May 28, 2024
623b8af
ui: file view
jazelly May 28, 2024
5a15de4
Update README.md
jazelly May 29, 2024
c59d7ed
wip: dashboard ui
jazelly May 31, 2024
f4b95bd
merge
jazelly May 31, 2024
d796792
ui: dashboard frame finished
jazelly Jun 1, 2024
a9b38d7
refactor: turn server to TS
jazelly Jun 2, 2024
7336498
feat: dashboard ui finished v0.1
jazelly Jun 2, 2024
06cb026
update readme
jazelly Jun 2, 2024
79401e1
ui: finetune panel polish
jazelly Jun 2, 2024
c3cfb14
fix: types usage in dashboard
jazelly Jun 4, 2024
cc8a989
feat: partially finished submit job endpoint
jazelly Jun 4, 2024
5a86517
update readme
jazelly Jun 4, 2024
0a25454
fix: style issue of sidebar icon
jazelly Jun 4, 2024
4248868
refactor: decouple onboarding flow
jazelly Jun 5, 2024
a68d1c3
update readme
jazelly Jun 5, 2024
79ad92d
update readme
jazelly Jun 5, 2024
3135272
wip: websocket between client and trainer
jazelly Jun 5, 2024
bbaaabe
wip: websocket between client and trainer
jazelly Jun 5, 2024
f03d496
fix: leaking keys
jazelly Jun 6, 2024
d51a8db
refactor: decouple user menu
jazelly Jun 6, 2024
99ab631
wip: job logs
jazelly Jun 6, 2024
3ad80f8
update readme
jazelly Jun 6, 2024
c222bb5
wip: relay job request to trainer
jazelly Jun 7, 2024
791bb81
refactor: TS and remove redundant files
jazelly Jun 7, 2024
a8ac2ec
feat: submitting job is connected to trainer
jazelly Jun 7, 2024
880b6a4
feat: support SFT via trainer
jazelly Jun 8, 2024
b8706b2
update readme
jazelly Jun 8, 2024
e8f1ccc
fix: ui issue of dashboard
jazelly Jun 8, 2024
f6f5176
dev: provide watch mode for trainer
jazelly Jun 8, 2024
c5b8546
fix: build issue and pipe trainer logs
jazelly Jun 8, 2024
9ebe3c3
fix: resolve CTAButton in component
jazelly Jun 8, 2024
0a95d8d
fix: sft trainer script bugs
jazelly Jun 8, 2024
993f3d5
feat: sft script can take model and dataset
jazelly Jun 11, 2024
9ddb909
chore: ignore logs
jazelly Jun 11, 2024
4031f3b
feat: support orpo in training script
jazelly Jun 11, 2024
32bfcb1
chore: delete unintended items
jazelly Jun 11, 2024
6d90483
refactor: separate rest and ws
jazelly Jun 11, 2024
739c1a6
fix: submitting job got module resolving bug
jazelly Jun 12, 2024
64b7bfb
build: use npm for frontend and server
jazelly Jun 14, 2024
70541e1
trainer, sft: can log via logger and pipe back to main process
jazelly Jun 14, 2024
bfc2866
feat: client and talk to trainer with ws
jazelly Jun 15, 2024
a6dc7ee
refactor: client submits job via ws to trainer directly
jazelly Jun 15, 2024
2ad63d8
feat: useWebSocket hook
jazelly Jun 15, 2024
5d2f033
chore: lint files
jazelly Jun 15, 2024
f403829
feat: use permalink as header
jazelly Jun 17, 2024
8153548
chore: lint
jazelly Jun 17, 2024
483f46a
chore: update readme with CUDA installation guide
jazelly Jun 17, 2024
af2f782
feat: trainer can actively push back to clients by group identifier
jazelly Jun 17, 2024
150c01b
env: type safely load dotenv and parse them
jazelly Jun 19, 2024
15fce1a
types: ts modals files
jazelly Jun 19, 2024
e588ccd
pipeline: page prototype
jazelly Jun 22, 2024
f09e572
ui: add inference panel with chat container
jazelly Jun 22, 2024
eac5bd9
fix: polish inference panel UI
jazelly Jun 23, 2024
aa91438
ui: added animated loader
jazelly Jun 23, 2024
f614653
ui: chat container polish
jazelly Jun 25, 2024
55bb3dd
feat: settings page is refactored and usable now
jazelly Jun 25, 2024
601782f
fix: redundant settings
jazelly Jun 25, 2024
67dc1b2
ui: redesign phase 1
jazelly Jul 3, 2024
bef40d0
ui: enhanced inference panel and rearranged header and sidebar
jazelly Jul 4, 2024
12570ae
chore: remove redundant files and ui update
jazelly Jul 8, 2024
9f6c9b0
feat: add local files endpoint
jazelly Jul 8, 2024
dd8c134
fix: some import issues
jazelly Jul 8, 2024
d78cdb5
Update README.md
devactivity-team Jul 9, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
324 changes: 164 additions & 160 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,160 +1,164 @@
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class

# C extensions
*.so

# Distribution / packaging
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
share/python-wheels/
*.egg-info/
.installed.cfg
*.egg
MANIFEST

# PyInstaller
# Usually these files are written by a python script from a template
# before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec

# Installer logs
pip-log.txt
pip-delete-this-directory.txt

# Unit test / coverage reports
htmlcov/
.tox/
.nox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*.cover
*.py,cover
.hypothesis/
.pytest_cache/
cover/

# Translations
*.mo
*.pot

# Django stuff:
*.log
local_settings.py
db.sqlite3
db.sqlite3-journal

# Flask stuff:
instance/
.webassets-cache

# Scrapy stuff:
.scrapy

# Sphinx documentation
docs/_build/

# PyBuilder
.pybuilder/
target/

# Jupyter Notebook
.ipynb_checkpoints

# IPython
profile_default/
ipython_config.py

# pyenv
# For a library or package, you might want to ignore these files since the code is
# intended to run in multiple environments; otherwise, check them in:
# .python-version

# pipenv
# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
# However, in case of collaboration, if having platform-specific dependencies or dependencies
# having no cross-platform support, pipenv may install dependencies that don't work, or not
# install all needed dependencies.
#Pipfile.lock

# poetry
# Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
# This is especially recommended for binary packages to ensure reproducibility, and is more
# commonly ignored for libraries.
# https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
#poetry.lock

# pdm
# Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
#pdm.lock
# pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
# in version control.
# https://pdm.fming.dev/#use-with-ide
.pdm.toml

# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
__pypackages__/

# Celery stuff
celerybeat-schedule
celerybeat.pid

# SageMath parsed files
*.sage.py

# Environments
.env
.venv
env/
venv/
ENV/
env.bak/
venv.bak/

# Spyder project settings
.spyderproject
.spyproject

# Rope project settings
.ropeproject

# mkdocs documentation
/site

# mypy
.mypy_cache/
.dmypy.json
dmypy.json

# Pyre type checker
.pyre/

# pytype static type analyzer
.pytype/

# Cython debug symbols
cython_debug/

# PyCharm
# JetBrains specific template is maintained in a separate JetBrains.gitignore that can
# be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
# and can be added to the global gitignore or merged into this file. For a more nuclear
# option (not recommended) you can uncomment the following to ignore the entire idea folder.
#.idea/
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class

# C extensions
*.so

# Distribution / packaging
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
share/python-wheels/
*.egg-info/
.installed.cfg
*.egg
MANIFEST

# PyInstaller
# Usually these files are written by a python script from a template
# before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec

# Installer logs
pip-log.txt
pip-delete-this-directory.txt

# Unit test / coverage reports
htmlcov/
.tox/
.nox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*.cover
*.py,cover
.hypothesis/
.pytest_cache/
cover/

# Translations
*.mo
*.pot

# Django stuff:
*.log
local_settings.py
db.sqlite3
db.sqlite3-journal

# Flask stuff:
instance/
.webassets-cache

# Scrapy stuff:
.scrapy

# Sphinx documentation
docs/_build/

# PyBuilder
.pybuilder/
target/

# Jupyter Notebook
.ipynb_checkpoints

# IPython
profile_default/
ipython_config.py

# pyenv
# For a library or package, you might want to ignore these files since the code is
# intended to run in multiple environments; otherwise, check them in:
# .python-version

# pipenv
# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
# However, in case of collaboration, if having platform-specific dependencies or dependencies
# having no cross-platform support, pipenv may install dependencies that don't work, or not
# install all needed dependencies.
#Pipfile.lock

# poetry
# Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
# This is especially recommended for binary packages to ensure reproducibility, and is more
# commonly ignored for libraries.
# https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
#poetry.lock

# pdm
# Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
#pdm.lock
# pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
# in version control.
# https://pdm.fming.dev/#use-with-ide
.pdm.toml

# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
__pypackages__/

# Celery stuff
celerybeat-schedule
celerybeat.pid

# SageMath parsed files
*.sage.py

# Environments
.env
.venv
env/
venv/
ENV/
env.bak/
venv.bak/
*/.venv

# Spyder project settings
.spyderproject
.spyproject

# Rope project settings
.ropeproject

# mkdocs documentation
/site

# mypy
.mypy_cache/
.dmypy.json
dmypy.json

# Pyre type checker
.pyre/

# pytype static type analyzer
.pytype/

# Cython debug symbols
cython_debug/

# PyCharm
# JetBrains specific template is maintained in a separate JetBrains.gitignore that can
# be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
# and can be added to the global gitignore or merged into this file. For a more nuclear
# option (not recommended) you can uncomment the following to ignore the entire idea folder.
#.idea/

checkpoint_dir/
node_modules/
3 changes: 3 additions & 0 deletions .gitmodules
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
[submodule "mac/slowllama/slowllama"]
path = mac/slowllama/llama2/slowllama
url = [email protected]:jazelly/slowllama.git
5 changes: 5 additions & 0 deletions .vscode/settings.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
{
"editor.accessibilityPageSize": 12,
"editor.formatOnSave": true,
"python.analysis.typeCheckingMode": "off"
}
42 changes: 21 additions & 21 deletions LICENSE
Original file line number Diff line number Diff line change
@@ -1,21 +1,21 @@
MIT License

Copyright (c) 2024 jazelly

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
MIT License
Copyright (c) 2024 Jason Zhang
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
Loading