Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Feature/experiment tracking #98

Open
wants to merge 77 commits into
base: main
Choose a base branch
from
Open

Feature/experiment tracking #98

wants to merge 77 commits into from

Conversation

adamkells
Copy link
Contributor

@adamkells adamkells commented Nov 8, 2024 •
edited
Loading

Description

The purpose of this PR is to add experiment tracking to Healthchain.

Related Issue

#57

Changes Made

A new class ExperimentTracker is created for usage inside the sandbox decorator. This automatically tracks any metadata about experiments run and saves the data to a local sql database.

A few additional bug fixes were made for LLM integration tests which were failing.

Testing

The summarization pipeline example in the cookbook was modified to use the experiment tracker and run locally to confirm that the expected behaviour occurs

Checklist

  • I have read the contributing guidelines
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes

Additional Notes

jenniferjiangkells and others added 30 commits October 10, 2024 16:51
@jenniferjiangkells
Added connector modules
e03111b
@jenniferjiangkells
Fix typo
9fc2f36
@jenniferjiangkells
Added processing of io connectors in pipelines
453b636
@jenniferjiangkells
Refactored CDA related processing in use case to connectors
7427aa8
@jenniferjiangkells
Added tests
ffe36a4
@jenniferjiangkells
Added CdsFhirConnector
c45a018
@adamkells
first pass at adding spacy and hf integrations
5a15cdb
@jenniferjiangkells
Updated use case functions and tests
dad5336
@jenniferjiangkells
WIP connector usage in pipelines and components
ee797da
@jenniferjiangkells
Fix model import name in docs
83d3299
@jenniferjiangkells
Update Bundle validator method to dynamically import nested resource …
32fa8bb 
…types
@jenniferjiangkells
Update CdsFhirConnector input method validations
ad4a4a7
@jenniferjiangkells
Add create method to CdsFhirData
45400fa
@jenniferjiangkells
Fixed CdsResponse should return list of actions
ba6f847
@jenniferjiangkells
Added tests
c5aa473
@jenniferjiangkells
Added pipeline tests
6f6a2f6
@adamkells
fix pyproject
ca73827
@adamkells
adding langchain and modifying document
b931229
@adamkells
added testing
7a3b904
@jenniferjiangkells
Changed .add() -> .add_node() to make more explicit and use conventio…
0a30447 
…n of BaseObject and base.py in modules
@jenniferjiangkells
Update documentation to reflect changes in this PR
1ffa34b
@jenniferjiangkells
Merge branch 'main' of https://github.com/dotimplement/HealthChain in…
c921aba 
…to feature/cda-connector
@adamkells
adding docs
1e057c9
@adamkells
finish docs
86c3b9f
@adamkells
fix test
f1dc664
@adamkells
fix test2
d1798dd
@jenniferjiangkells
WIP
6a38c72
@adamkells
skip transformers test
c01b61d
@adamkells
fix tests
694cd74
@adamkells
adding magicmock for iterable
4b8c91d
adamkells and others added 24 commits November 8, 2024 15:24
@adamkells
fix decorator code deleted by cursor
58c99f5
@jenniferjiangkells
Updated tests for prebuilt pipelines
0b24f70
@jenniferjiangkells
Added tests for pipeline loading method and modelrouter
c26d08e
@jenniferjiangkells
Update test for spacy integration
44e6475
@jenniferjiangkells
Tweak fixture
981e9d5
@jenniferjiangkells
Use Mixin for ModelRouter
1119e8c
@jenniferjiangkells
Clean up __init__ imports
3ee43ba
@adamkells
sql database experiment tracking
ca3cf49
@jenniferjiangkells
Fix resourceType not showing up by explicitly passing it in when call…
a4e5bfa 
…ed in data generator
@jenniferjiangkells
Parse text from DocumentReference in cdsfhir
22a5c49
@jenniferjiangkells
Add delimiter to create multiple cards and basic text cleaner for tem…
1ba3a73 
…plates to card reader
@jenniferjiangkells
Make model loading more explicit and added langchain routing
46453f0
@jenniferjiangkells
Update prebuilt pipeline initialization methods
0bec7a0
@jenniferjiangkells
Update tests
0ae269d
@adamkells
Merge branch 'feature/pipeline-implementations' into feature/experime…
fb9f228 
…nt-tracking
@jenniferjiangkells
Added cookbook
a295166
@jenniferjiangkells
Moved default mapping initialization inside data generator
cba3b17
@jenniferjiangkells
Split .load method to from_model_id and from_local_model and added te…
3977f75 
…mplate path as init option
@jenniferjiangkells
Update tests and docs
9970e0b
@adamkells
merge pipelines
fbeffc4
@adamkells
refining experiment tracking
4f1fca3
@adamkells
refine table structure
362a095
@adamkells
bugfixing the test
c65e9b4
@adamkells
Add docs
b243be2
@adamkells adamkells changed the base branch from feature/pipeline-implementations to main December 4, 2024 10:07
@adamkells adamkells marked this pull request as ready for review December 4, 2024 10:08
jenniferjiangkells
jenniferjiangkells approved these changes Dec 5, 2024
View reviewed changes
Copy link
Member

@jenniferjiangkells jenniferjiangkells left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm. Just out of curiosity do you have an example of what the experiment data looks like in a real example? Maybe can put it in docs.

@adamkells
Copy link
Contributor Author

Screenshot from 2024-12-07 22-42-04
Screenshot from 2024-12-07 22-42-18

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jenniferjiangkells jenniferjiangkells jenniferjiangkells approved these changes

Assignees

@adamkells adamkells

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@adamkells @jenniferjiangkells