Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DataTable is returning proper columns, but with null data #52

Open
m-glisson opened this issue Oct 26, 2023 · 1 comment
Open

DataTable is returning proper columns, but with null data #52

m-glisson opened this issue Oct 26, 2023 · 1 comment

Comments

@m-glisson
Copy link

I have a data set that I unfortunately can't share, however its hosted on S3.

I can load the data in using

delta_table_path = 's3://my/delta/path'
df = DeltaTable(delta_table_path, file_system=fs).to_pandas() 

this comes across with the correct column names, and seemingly the correct row count, however all of the data int the dataset is null which is not the case because we do have this data picked up in spark and generating output tables

Sorry for the vague response, I'm just looking for some advice or if this is a known issue

@jeppe742
Copy link
Owner

Hey @m-glisson
I'm not aware of this issue.
Unfortunately, I hope you understand that it is almost impossible for me to troubleshoot without any more information.
Would you at least be able to provide one of the json files from the _delta_log, or maybe just the schema of the table?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants