Implemented TruncatedOnlineStatistics to collect truncated stats in a distributed way#57

Merged

ChaoPang merged 11 commits intomainfrom bound_numeric_values

Sep 13, 2024

+519-16

Commits on Sep 12, 2024

added patient_id to the pretraining data for debugging
ChaoPang
committed
added RunningStatistics to remove the extreme outliers when calculating the running mean and std
ChaoPang
committed
added the lab value lower/upper bounds to the tokenizer so we can bound the extreme values during tokenization
ChaoPang
committed

Commits on Sep 13, 2024