Arrow language to use industry standard format for columnar data

[JaroslavTulach]

Investigation of

• Investigate usage of Python libraries #7388

resulted in an observation that Pandas 2.0 support Apache Arrow. If we want smooth, zero copy exchange of Enso Table & Column data with Pandas & other Arrow supporting libraries, then the best way is to store the data in Arrow formats.

To get the best from Truffle we want to create an internal Arrow language to hide the access to Arrow structures behind a simple facade and InteropLibrary.readArrayElement. This would be the expected usage:

var myDate32array = ctx.eval("arrow", "new[date32]").execute(size)

e.g. one could specify the format of data as a text that gets parsed and returns a factory function to create array of given size. The function than returns a TruffleObject that behaves like an array:

@ExportLibrary(InteropLibrary.class)
class ArrayDate32Array {
    private final ByteBuffer buffer;

    @ExportMessage
    void writeArrayElement(Object obj, int index, @CachedLibrary InteropLibrary iop) {
        assert iop.isDate(obj);
        var at = index * 4;
        var time = iop.asDate(obj).epochTimeMillis();
        buffer.putLong(at, time);
    }
 }

When this partially evaluates, we really would like to get to simple few assembly instructions that read the buffer offset, add at and write the four bytes representing time to that address.

Btw. one can use the official Java bindings as an inspiration. To be able to consume "Arrows" from other libraries, we need a way to "cast" a pointer to some Arrow type:

var otherDate32array = ctx.eval("arrow", "cast[date32]").execute(pointer)

The Arrow Language only allocates (or casts) array-like structures_ in proper format suitable for exchange with other Arrow-ready libraries. Apache Arrow only specifies the data layout - it doesn't provide any operations on the data. Such operations are then provided by libraries working on the interchangeable data.

Further references

• the heap language shows how to implement such an internal language
• the full heap language implementation
• Arrow streaming format

[enso-bot]

Hubert Plociniczak reports a new STANDUP for yesterday (2023-12-06):

Progress: Fixed remaining issues for MacOS arm build. Started looking into requirements for Arrow, doing some background reading on specification. It should be finished by 2023-12-22.

Next Day: Next day I will be working on the #7755 task. Continue investigating into Arrow.

[enso-bot]

Hubert Plociniczak reports a new STANDUP for yesterday (2023-12-07):

Progress: Small progress on getting the initial example going. Increased timeout for running launcher command (MacOS appears to consistently fail for no apparent reason on CI #8486). It should be finished by 2023-12-22.

Next Day: Next day I will be working on the #7755 task. Continue with adding support for Arrow.

[enso-bot]

Hubert Plociniczak reports a new STANDUP for the provided date (2023-12-08):

Progress: Continued working on small date examples, dealing with some interop problems. Debugged some connection problems in CI for MacOS builds; turned out some retries were fixed and added to latest sbt (needed a bump, #8498) It should be finished by 2023-12-22.

Next Day: Next day I will be working on the #7755 task. Continue with adding support for Arrow.

[enso-bot]

Hubert Plociniczak reports a new STANDUP for yesterday (2023-12-11):

Progress: Draft PR is up which implements the first half proposed in the ticket. Date32 and Date64 types are supported. It should be finished by 2023-12-22.

Next Day: Next day I will be working on the #7755 task. Continue with adding support for Arrow.

[enso-bot]

Hubert Plociniczak reports a new STANDUP for yesterday (2023-12-12):

Progress: Adding more tests and types, as per specification. It should be finished by 2023-12-22.

Next Day: Next day I will be working on the #7755 task. Continue with adding support for Arrow.

[enso-bot]

Hubert Plociniczak reports a new STANDUP for yesterday (2023-12-13):

Progress: Addressing PR reviews, figuring out how to test casting. Lots of meetings. It should be finished by 2023-12-22.

Next Day: Next day I will be working on the #7755 task. Continue with adding support for Arrow language.

[enso-bot]

Hubert Plociniczak reports a new STANDUP for yesterday (2023-12-14):

Progress: Support for casting for Arrow vectors created in other languages. Added a test illustrating the support using Java's Arrow implementation. It should be finished by 2023-12-22.

Next Day: Next day I will be working on the #7755 task. Continue with adding support for Arrow language.

[enso-bot]

Hubert Plociniczak reports a new STANDUP for the provided date (2023-12-15):

Progress: Added proper nullability support without any copying. PR is mostly ready with the initial support. Added more tests. It should be finished by 2023-12-22.

Next Day: Next day I will be working on the #7755 task. Address review, but also look into ongoing compiler bugs.

[enso-bot]

Hubert Plociniczak reports a new 🔴 DELAY for yesterday (2024-01-09):

Summary: There is 19 days delay in implementation of the Arrow language to use industry standard format for columnar data (#7755) task.
It will cause 0 days delay for the delivery of this weekly plan.

Delay Cause: Didn't address PR feedback before holidays and moved on to other issues in the meantime.

[enso-bot]

Hubert Plociniczak reports a new STANDUP for yesterday (2024-01-09):

Progress: Rebasing against latest develop, fighting with the JPMS. Also trying to confirm optimization in AliasAnalysis via benchmarks. It should be finished by 2024-01-10.

Next Day: Next day I will be working on the #7755 task. Merge PR

[enso-bot]

Hubert Plociniczak reports a new STANDUP for yesterday (2024-01-10):

Progress: Modularized Arrow language project. Investigated UUID optimization. It should be finished by 2024-01-10.

Next Day: Next day I will be working on the #7755 task. Address PR review.

[enso-bot]

Hubert Plociniczak reports a new 🔴 DELAY for yesterday (2024-01-11):

Summary: There is 2 days delay in implementation of the Arrow language to use industry standard format for columnar data (#7755) task.
It will cause 0 days delay for the delivery of this weekly plan.

Delay Cause: Underestimated the amount of work related to PR review. Will delay some work to follow up PRs.

[enso-bot]

Hubert Plociniczak reports a new STANDUP for yesterday (2024-01-11):

Progress: Addressing PR review. PR that introduces lazy UUID generation (#8716). It should be finished by 2024-01-12.

Next Day: Next day I will be working on the #7755 task. Address PR review.

[enso-bot]

Hubert Plociniczak reports a new STANDUP for the provided date (2024-01-12):

Progress: Fighting with CI, last minute improvements to PR. The remaining work will go in followup PRs. Similarly with UUID PR in #8728. Investigated assertion failure in #8595, pushed a workaround unblocking the PR but failures to come up with a minimal reproducible case so far failed. It should be finished by 2024-01-12.

Next Day: Next day I will be working on the #8689 task. Look into GUI/backedn failures reported recently.