SchemaString Key Error

[crisnaX]

While trying to load delta table getting shemaString key error at apply_partial_logs function. This issue due to some imperfections in checkpoint file like metadata doesn't have schemaString in one of the checkpoint file. I need to skip that file and load delta table. For this we need to add error handling functionality.

[jeppe742]

Hey @crisnaX
I'm a bit surprised you even managed to get this error.
According to the delta protocol the schemaString is a required field.
What did you use to create the delta table?

[crisnaX]

We are using databricks streaming tables. pyspark will create delta tables. For the negative testing we send the change in schema or some other violated files to check the capability of handling of our data quality rules. During that time the delta log records are not having this schemaString. When I am using delta lake reader from my vscode, I was getting this error at such particular log file. For that,I have raised a pull request.

[crisnaX]

Just Try and Except added to partial logs,I was able to resolve that issue. You are correct according to delta protocol schema String is required field. But, exceptional cases like schema mismatch this is happening.

[crisnaX]

Any update @jeppe742 ?