persist the patient events throughout the generation of the patient sequences for better readability and debugging purposes #24

ChaoPang · 2025-01-07T15:13:58Z

No description provided.

…equences for debugging purposes

…e duration is grater than 24 hours, the visits will be set to inpatient

…of the EHR shot records

…s in ehrshot data

…d to control how far apart two visits are from each other to be considered as separate visits

…_mapping table

…nd_date is set to True

…nding visit window

…e corresponding visit range

… index_date

…sits

…fixed

…f each individual visit is less than 24 hours

…rst token occurs after the index_date

ChaoPang added 2 commits January 7, 2025 09:58

persist the patient events throughout the generation of the patient s…

bca6885

…equences for debugging purposes

set the outpatient visit_start_datetime to be at the start of the day

2f02967

ChaoPang force-pushed the persist_events branch from 980ea54 to 2f02967 Compare January 8, 2025 16:09

ChaoPang added 14 commits January 8, 2025 12:57

explicitly cast datetime of ehr records to time stamps

2c690ff

cast timestamps in extract_features

f336174

infer the inpatient visits based on the duration of the visits. If th…

fa74f0d

…e duration is grater than 24 hours, the visits will be set to inpatient

switched to spark function to filter instead of the sql syntax

d4ed4a6

saved all the temp dataframes when cleaning up the visit information …

6192b1f

…of the EHR shot records

updated the real_visits_folder variable name

b05e00f

fixed a bug in referencing the columns

1d55383

use aliases in joining ehr_shot_data to inferred_inpatient_visits

592b8fd

fixed a bug in creating code for the ehrshot data for non visit events

43e7690

fixed a bug in creating the start and end date times for visit record…

9f3ecab

…s in ehrshot data

started working on fixing inpatient visits

b377590

added spark applications to connect ehrshot visit chronologically

14d14ac

removed unncessary logging

55916fb

added inpatient_hour_diff_threshold and outpatient_hour_diff_threshol…

cc8c856

…d to control how far apart two visits are from each other to be considered as separate visits

ChaoPang force-pushed the persist_events branch from 2bb07cd to cc8c856 Compare January 10, 2025 22:22

ChaoPang added 6 commits January 10, 2025 17:40

exclude visit_occurrence and death from being updated using the visit…

c74c777

…_mapping table

copy vocab tables

0e01237

copy person table over

1e92660

persist visit during AttEventDecorator

00d5ecb

fixed a bug when extract_features spark app is run when bound_visit_e…

ab20d32

…nd_date is set to True

the visits need to be bounded per cohort member

bd26431

ChaoPang force-pushed the persist_events branch from 60c01b4 to bd26431 Compare January 11, 2025 05:15

ChaoPang added 3 commits January 11, 2025 00:20

persist cohort_visit_occurrence to the disk

9645fcf

updated the persistant dataframe paths

493d76d

disconnect the records whose time stamps fall outside of the correspo…

79cdf96

…nding visit window

ChaoPang force-pushed the persist_events branch 2 times, most recently from fa22c9e to a27f97b Compare January 11, 2025 16:17

fix the visit and domain records whose time stamps fall outside of th…

9d0010f

…e corresponding visit range

ChaoPang force-pushed the persist_events branch from a27f97b to 9d0010f Compare January 11, 2025 16:18

ChaoPang added 3 commits January 11, 2025 12:55

fixed the bug in creating the new visit_id

6215db4

added original_visit_id to the ehrshot output

9a89ee6

fixed a bug when bound_visit_end is enable in extract_features.py

c63052e

ChaoPang force-pushed the persist_events branch from 6499cef to 7457aa1 Compare January 13, 2025 04:58

Add an artificial token for the visit in which the prediction is made

0643be0

ChaoPang force-pushed the persist_events branch from 7457aa1 to 0643be0 Compare January 13, 2025 05:01

ChaoPang added 5 commits January 13, 2025 07:07

added placeholder tokens to the output folder for debugging

f6d053b

fixed the placeholder token time stamps

cf1f3a9

filter for visit_occurrence records based on the cohort_member_id and…

e19044f

… index_date

use event_start to construct the visit_end_datetime for artificial vi…

f31ce92

…sits

changed the way to bound the visit_end_datetime

945a2b9

ChaoPang force-pushed the persist_events branch from a4577e7 to 945a2b9 Compare January 13, 2025 14:58

ChaoPang added 3 commits January 13, 2025 11:42

fixed windowing bug in creating visit_rank

8ab7b02

fixed a bug in constructing the visit mappings

5d9dd22

removed a query that does not affect the results

5da6984

ChaoPang force-pushed the persist_events branch from 9b997ab to 5da6984 Compare January 13, 2025 18:44

use hours to infer whether the visit start/end date times need to be …

8540d5d

…fixed

ChaoPang force-pushed the persist_events branch from 3ff94db to 8540d5d Compare January 13, 2025 18:55

ChaoPang added 5 commits January 13, 2025 14:17

if we split inpatient visits into multiple visits, we need to check i…

388dbdd

…f each individual visit is less than 24 hours

changed the day_cutoff default value to 1

b8ba31b

if we split inpatient visits into multiple visits, we need to check i…

19dcd77

…f each individual visit is less than 24 hours

We create placeholder tokens for those inpatient visits, where the fi…

37cdff0

…rst token occurs after the index_date

removed unused imports

7d4ae0b

ChaoPang force-pushed the persist_events branch from 2024929 to 61c2cb8 Compare January 16, 2025 14:43

try to fix the java incompatibility issue with pyspark

4b94b70

ChaoPang force-pushed the persist_events branch from 61c2cb8 to 4b94b70 Compare January 16, 2025 14:49

fixed the unit test

3b8c558

ChaoPang merged commit 66b2a6f into main Jan 16, 2025
4 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

persist the patient events throughout the generation of the patient sequences for better readability and debugging purposes #24

persist the patient events throughout the generation of the patient sequences for better readability and debugging purposes #24

ChaoPang commented Jan 7, 2025

persist the patient events throughout the generation of the patient sequences for better readability and debugging purposes #24

persist the patient events throughout the generation of the patient sequences for better readability and debugging purposes #24

Conversation

ChaoPang commented Jan 7, 2025