fix: failure Loading json columns into an arrow table

[neuromantik33]

Issue Summary

When loading a table with json columns into an Arrow table, the process fails due to nullable json payloads. Since the dataset is sparse, the first row in the result set is often None, leading to py_type being inferred as NoneType. As a result, the column is not converted into a JSON string array, causing the following error: Expected bytes, got a 'dict' object.

Steps to Reproduce

• Load a SQL table containing a json column into an Arrow table.
• Ensure the first row of the result set contains None
• Observe that the Arrow conversion fails with the above error message.

Expected Behavior

• The json column should be properly converted to a json string array, even if the first value in the dataset is NULL.
• The inference mechanism should recognize the json type instead of defaulting to NoneType.

sql_database workaround

A query adapter can be used to explicitly cast JSON columns to TEXT:

def json_to_text(_: Select, table: sa.Table) -> Select:
    """FIXME Workaround to convert all JSON types to text until pyarrow adapter is fixed"""
    columns = [
        sa.cast(col, sqltypes.Text).label(col.name) if isinstance(col.type, sqltypes.JSON) else col
        for col in table.columns
    ]
    return sa.select(*columns)

But seeing as I also use this module in my pg_replication source plugin I preferred to correct it at the source, no pun intended 😉

[netlify]

✅ Deploy Preview for dlt-hub-docs canceled.

Name	Link
🔨 Latest commit	0f57b92
🔍 Latest deploy log	https://app.netlify.com/sites/dlt-hub-docs/deploys/67a55a0dbe50aa00082f3cc2

[rudolfix]

@neuromantik33 FYI we are on it here #2295
@zilto please look at test case in this PR. this is the thing I mentioned in PR. the current idea is that we launch the conversion when we get an exception from arrow, so we prevent looping on Python rows in most cases....

[neuromantik33]

Thanks for tackling this issue, feel free to close this whenever you have something and if you need a beta tester ... 😉