Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Hive binary type doesn't work correctly due to Ion Blob<>bytes conversion #101

Open
cheqianh opened this issue Jan 2, 2023 · 1 comment
Labels
bug Something isn't working

Comments

@cheqianh
Copy link
Contributor

cheqianh commented Jan 2, 2023

Description

ion-hive-serde IonBlob <-> binary conversion currently returns the byte array’s reference in memory. It should returns at least a text representation of the Ion Blob value.

Reproduction

One example when query Ion Blob values through AWS Athena

Schema:

content has a type of binary

CREATE EXTERNAL TABLE test_blob (
    `payload` struct < 
    id: string,
    content: binary >
)
ROW FORMAT SERDE 'com.amazon.ionhiveserde.IonHiveSerDe'
WITH SERDEPROPERTIES (
    'ion.encoding' = 'BINARY',
    'ion.fail_on_overflow' = 'true',
    'ion.path_extractor.case_sensitive' = 'false',
    'ion.timestamp.serialization_offset' = 'Z',
    'ion.serialize_null' = 'OMIT',
    'ion.ignore_malformed' = 'false'
)
STORED AS INPUTFORMAT 'com.amazon.ionhiveserde.formats.IonInputFormat' OUTPUTFORMAT 'com.amazon.ionhiveserde.formats.IonOutputFormat'
LOCATION 's3://test-athena-ion-shorthand/';

File

Below is the Ion file, content has an IonBlob value.

{"payload": 
  { id:"2022-12-08T17:08:11.174Z#5a5b185a-9b13-482c-bb78-fe1894f39000",
    content:    {{e3BhdHRlcm46W3tsTJDKVEO#JD3r942kdkvDNVfdGltZToyMDIyLTEyLTA4VDE3OjA4OjA5LjU2MFosc291cmNlOntjdXN0b21lcl9pZDo2NTc3NjI5NjAwMixza3U6IkIwQk5OTVBTTjgiLHNrdV92ZXJzaW9uOjJ9fV19fV19}},
}}

Query

select * from test_blob

Result

{
    id=2022-12-08T17:08:11.174Z#5a5b185a-9b13-482c-bb78-fe1894f39000, 
    content=[B@4e3f3378, 
}

Notice that content value changed from an Ion Blob to B@4e3f3378, which is a bug in ion-hive-serde.

@cheqianh cheqianh added the bug Something isn't working label Jan 2, 2023
@treyyi
Copy link

treyyi commented Jan 4, 2023

@cheqianh
Hi Eric,
This is Trey who addressed this issue to you.
Please assign this issue to me and I'm going to work on it :)

Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants