arrow ipc copy #2
Conversation
| jobs: | ||
| unitTests: | ||
| name: SQL unit tests | ||
| runs-on: macos-latest |
There was a problem hiding this comment.
no reason, we were build linux and osx in other repo. I thought we will start with one. LMK if you want me to change this to linux.
There was a problem hiding this comment.
i would prefer linux. gives me more confidence on the platform we will run on
| @@ -87,11 +117,11 @@ ArrowIPCWriteInitializeGlobal(ClientContext &context, FunctionData &bind_data, | |||
|
|
|||
| vector<unique_ptr<Expression>> ArrowIPCWriteSelect(CopyToSelectInput &input) { | |||
| vector<unique_ptr<Expression>> result; | |||
| bool any_change = false; | |||
| // bool any_change = false; | |||
|
|
||
| // Create stream decoder and buffer | ||
| auto stream_decoder = make_uniq<BufferingArrowIPCStreamDecoder>(); | ||
| auto consume_result = stream_decoder->Consume(file_buffer.data(), file_size); | ||
| if (std::string(reinterpret_cast<char*>(file_buffer->data()), 6) == ARROW_MAGIC) { |
There was a problem hiding this comment.
can you explain what this is doing? Is the written data wrong and if so can we fix it on the write side?
Is the write side compatible w/ snowflake's arrow or not?
There was a problem hiding this comment.
This is trying to handle both file and stream formats. ipc stream doesn't have magic and footer, where as file format has it.
I will fix the write side to keep it same as snowflake.
Just thinking that the extension should support both formats, atleast on read side. I will do similar fix in arrow_scan_ipc.cpp.
I will add a comment.
No description provided.