Arrow language #8512

hubertp · 2023-12-11T16:11:39Z

Pull Request Description

Initial implementation of the Arrow language. Closes #7755.
Currently supported logical types are

Date (days and milliseconds)
Int (8, 16, 32, 64)

One can currently

allocate a new fixed-length, nullable Arrow vector - new[<name-of-the-type>]
cast an already existing fixed-length Arrow vector from a memory address - cast[<name-of-the-type>]

Closes #7755.

Checklist

Please ensure that the following checklist has been satisfied before submitting the PR:

The documentation has been updated, if necessary.
All code follows the
Scala,
Java,
and
Rust
style guides. In case you are using a language not listed above, follow the Rust style guide.
All code has been tested:
- Unit tests have been written where possible.

JaroslavTulach

I think this is a correct start.

I assume that in the future we want to support "structures" of array elements. E.g. not just an ArrayFixedArrayData32 or 64-bit, but:

ArrayFixedArray of int32, Date32, float, etc. struct. That will probably lead towards abstracting/defining the structure of single array element rather than having special types of array. Possibly.

build.sbt

engine/runtime-language-arrow/src/main/java/org/enso/interpreter/arrow/ArrowContext.java

engine/runtime-language-arrow/src/main/java/org/enso/interpreter/arrow/ArrowLanguage.java

engine/runtime-language-arrow/src/main/java/org/enso/interpreter/arrow/ArrowParser.java

...e-language-arrow/src/main/java/org/enso/interpreter/arrow/runtime/ArrowFixedArrayDate32.java

...e-language-arrow/src/main/java/org/enso/interpreter/arrow/runtime/ArrowFixedArrayDate64.java

engine/runtime-with-instruments/src/main/java/module-info.java

engine/runtime-with-polyglot/src/test/java/org/enso/interpreter/test/VerifyArrowTest.java

JaroslavTulach · 2023-12-13T04:47:19Z

A recommendation for study: https://www.graalvm.org/truffle/javadoc/com/oracle/truffle/api/staticobject/package-summary.html - that's the official Truffle way to represent immutable (e.g. static) objects.

com.oracle.truffle.api.staticobject (GraalVM Truffle Java API Reference)

...arrow/src/main/java/org/enso/interpreter/arrow/runtime/ArrowCastToFixedSizeArrayFactory.java

engine/runtime-with-instruments/src/main/java/module-info.java

...ime-language-arrow/src/main/java/org/enso/interpreter/arrow/node/ArrowCastFixedSizeNode.java

engine/runtime-language-arrow/src/main/java/org/enso/interpreter/arrow/node/ArrowEvalNode.java

engine/runtime-language-arrow/src/main/java/org/enso/interpreter/arrow/ArrowParser.java

Akirathan · 2023-12-19T09:04:25Z

engine/runtime-language-arrow/src/main/java/org/enso/interpreter/arrow/ArrowLanguage.java

+    id = ArrowLanguage.ID,
+    name = "Truffle implementation of Arrow",
+    characterMimeTypes = {ArrowLanguage.MIME},
+    defaultMimeType = ArrowLanguage.MIME,


You have to explicitly specify internal = true. By default internal = false.

The ArrowLanguage shall be an experimental feature accessible via foreign arrow xyz = """ syntax. I am afraid the language needs to be non-internal to be exposed in the foreign set of language (as of #7882).

In any case, it'd be good if ArrowLanguage wasn't accessible by default (throw a parsing error for example). Possibly shield it with

var canArrow = false; assert canArrow = true; if (!canArrow) throw ...

A simple example with continuous fixed-size array storing dates is added. More to follow.

Tests were previously wrong and specialization had a typo, which was misleading. In order to be really comparable (not only up-to miliseconds) had to create instant for ZonedDateTime from seconds and nanoseconds.

Also simplified Date implementation to share more code.

Added an example where an IntVector is created in Java, we get a memory pointer to the buffer, and cast it in our Arrow language. The main trick was to ensure that the buffer created at the specific address has `LITTLE_ENDIAN` order. The test also revealed that we didn't type-adjust the index when writing/reading to our fixed-size int array.

It makes little sense to include the full dependency just to allow for memory-mapped byte buffer which we can construct by hand.

Rather than copying over non-null bitmaps (aka `validityBuffer`) we memory-map it, similarly to the data buffer. The code can be further simplified if we lock the type of elements stored in the ByteBuffer to a single one.

JaroslavTulach

I would be happy if there was a separate arrow-language.jar with own module-info.class. That way the Arrow language was clearly separated from the Enso runtime.

I'd hide the Arrow language behind a flag - otherwise it is hard to find out it is experiemental. Possibly behind -ea for now.

JaroslavTulach · 2024-01-10T12:54:07Z

build.sbt

+      },
+      Test / addModules := Seq("org.enso.interpreter.arrow"),
+      Test / javaOptions ++= testLogProviderOptions ++ Seq(
+        "--add-opens=java.base/java.nio=org.enso.interpreter.arrow",


I am not a fan of --add-opens, but having that in tests only is probably fine.

One can probably use --add-opens=org.enso.interpreter.arrow/org.enso.interpreter.arrow==ALL-UNNAMED - or something like that.

build.sbt

engine/runtime-fat-jar/src/main/java/module-info.java

...arrow/src/main/java/org/enso/interpreter/arrow/runtime/ArrowCastToFixedSizeArrayFactory.java

engine/runtime-language-arrow/src/main/java/org/enso/interpreter/arrow/runtime/NullValue.java

engine/runtime-language-arrow/src/main/java/org/enso/interpreter/arrow/util/MemoryUtil.java

engine/runtime-language-arrow/src/test/java/org/enso/interpreter/arrow/VerifyArrowTest.java

JaroslavTulach · 2024-01-11T07:34:53Z

engine/runtime-language-arrow/src/test/java/org/enso/interpreter/arrow/VerifyArrowTest.java

+      assertNotNull(int32Constr);
+      Value int32Array =
+          int32Constr.newInstance(
+              vector.getDataBufferAddress(),


address and capacity is OK, for now. It'd be handy (for interaction with Python) to also support InteropLibrary.isPointer.

...untime-language-arrow/src/main/java/org/enso/interpreter/arrow/runtime/ByteBufferDirect.java

hubertp · 2024-01-11T13:37:41Z

I would be happy if there was a separate arrow-language.jar with own module-info.class. That way the Arrow language was clearly separated from the Enso runtime.

That's possible already now.

sbt:enso> runtime-language-arrow/package
> jar tf engine/runtime-language-arrow/target/runtime-language-arrow-0.0.0-dev.jar 
...
META-INF/services/com.oracle.truffle.api.provider.TruffleLanguageProvider
module-info.class
...

JaroslavTulach · 2024-01-11T19:43:30Z

I would be happy if there was a separate arrow-language.jar with own module-info.class. That way the Arrow language was clearly separated from the Enso runtime.

That's possible already now.
sbt:enso> runtime-language-arrow/package
> jar tf engine/runtime-language-arrow/target/runtime-language-arrow-0.0.0-dev.jar 
...
META-INF/services/com.oracle.truffle.api.provider.TruffleLanguageProvider
module-info.class
...

The problem is not that it is possible, but that it is impossible to distribute runtime.jar without Arrow!

hubertp · 2024-01-12T09:05:55Z

The problem is not that it is possible, but that it is impossible to distribute runtime.jar without Arrow!

I don't think that is the case anymore. Nothing depends on arrow project.

hubertp · 2024-01-12T09:09:04Z

I think this first step towards supporting Arrow language is sufficient enough. There is obviously more work involved but the longer I delay the merge the harder becomes to keep it in sync with develop.

Akirathan

Excuse the delay in my review. Apart from some minor suggestions, looks good.

engine/runtime-language-arrow/src/test/java/org/enso/interpreter/arrow/VerifyArrowTest.java

Akirathan · 2024-01-12T09:48:56Z

engine/runtime-language-arrow/src/test/java/org/enso/interpreter/arrow/VerifyArrowTest.java

+  private BaseFixedWidthVector allocateFixedLengthVector(
+      BufferAllocator allocator, Object[] testValues, LogicalLayout unit) {
+    var valueCount = 0;
+    switch (unit) {


minor: refactor to return switch expression

engine/runtime-language-arrow/src/main/java/org/enso/interpreter/arrow/ArrowParser.java

...time-language-arrow/src/main/java/org/enso/interpreter/arrow/runtime/ArrowFixedArrayInt.java

engine/runtime-language-arrow/src/test/resources/application-test.conf

build.sbt

Akirathan · 2024-01-12T10:22:51Z

The problem is not that it is possible, but that it is impossible to distribute runtime.jar without Arrow!

@JaroslavTulach What do you mean that it is impossible to distribute runtime.jar without Arrow? Currently, no classes from runtime-language-arrow project get assembled into runtime.jar, neither in any other jar archives that are on module path or on class path.

Or did you mean that you would like to distribute the arrow language, but would like to keep it separate from the org.enso.runtime module?

Co-authored-by: Pavel Marek <[email protected]>

JaroslavTulach reviewed Dec 12, 2023

View reviewed changes

enso-bot bot mentioned this pull request Dec 13, 2023

Invalid Sharing of AST nodes in EpbContext #5087

Closed

2 tasks

hubertp added the CI: Clean build required CI runners will be cleaned before and after this PR is built. label Dec 13, 2023

hubertp force-pushed the wip/hubert/7755-arrow branch from 810e06e to fd7a132 Compare December 13, 2023 15:37

JaroslavTulach reviewed Dec 15, 2023

View reviewed changes

...arrow/src/main/java/org/enso/interpreter/arrow/runtime/ArrowCastToFixedSizeArrayFactory.java Outdated Show resolved Hide resolved

hubertp force-pushed the wip/hubert/7755-arrow branch from d9e640b to 28a06cb Compare December 15, 2023 14:52

hubertp changed the title ~~WIP: First take at implementing Arrow~~ Arrow language Dec 15, 2023

hubertp marked this pull request as ready for review December 15, 2023 21:37

hubertp requested review from 4e6 and Akirathan as code owners December 15, 2023 21:37

Akirathan reviewed Dec 19, 2023

View reviewed changes

radeusgd mentioned this pull request Jan 8, 2024

Make fill_nothing take an empty string #8643

Merged

5 tasks

hubertp added 17 commits January 8, 2024 17:17

First take at implementing Arrow

6755164

A simple example with continuous fixed-size array storing dates is added. More to follow.

Fix implementation for Date64

29b7ce6

Tests were previously wrong and specialization had a typo, which was misleading. In order to be really comparable (not only up-to miliseconds) had to create instant for ZonedDateTime from seconds and nanoseconds.

Fix native image build

2c80897

Add IntX Arrow type

29ed745

Also simplified Date implementation to share more code.

missing file

3a66ce1

Addressing PR comments

fc68ca0

Accept any number for new instance size

fd138b2

Fix native image build

6446bd1

Support nullable vectors

e451806

Drop arrow-memory-core dependency

67ba40d

It makes little sense to include the full dependency just to allow for memory-mapped byte buffer which we can construct by hand.

fmt

5147628

Refactored tests and added Int64 tests

c13efb0

Memory-mapped non-null bitmaps

f52cbd3

Rather than copying over non-null bitmaps (aka `validityBuffer`) we memory-map it, similarly to the data buffer. The code can be further simplified if we lock the type of elements stored in the ByteBuffer to a single one.

changelog

7381959

Remove (currently) unsupported types

88c80ab

nits

ab3cee8

hubertp requested a review from JaroslavTulach January 10, 2024 10:17

hubertp added 3 commits January 10, 2024 11:39

fix native build

9462066

nits

f5855e7

revert, don't want to deal with license update

2e8d7cb

JaroslavTulach approved these changes Jan 11, 2024

View reviewed changes

JaroslavTulach reviewed Jan 11, 2024

View reviewed changes

...untime-language-arrow/src/main/java/org/enso/interpreter/arrow/runtime/ByteBufferDirect.java Show resolved Hide resolved

hubertp added 3 commits January 11, 2024 10:59

PR review

fbb0ee4

nit

431bd5c

get rid of proxy

81de77f

hubertp added 2 commits January 11, 2024 15:32

s/instantiate/execute

52bae38

nits

f5a161d

enso-bot bot mentioned this pull request Jan 12, 2024

Avoid Enso/Python & co. interop problems by using Truffle TCK tests #5238

Closed

hubertp added CI: Ready to merge This PR is eligible for automatic merge CI: Keep up to date Automatically update this PR to the latest develop. labels Jan 12, 2024

Merge branch 'develop' into wip/hubert/7755-arrow

a01b9c0

hubertp removed the CI: Keep up to date Automatically update this PR to the latest develop. label Jan 12, 2024

Akirathan approved these changes Jan 12, 2024

View reviewed changes

hubertp removed the CI: Ready to merge This PR is eligible for automatic merge label Jan 12, 2024

hubertp and others added 3 commits January 12, 2024 15:08

Apply suggestions from code review

2cb5cf4

Co-authored-by: Pavel Marek <[email protected]>

Merge branch 'develop' into wip/hubert/7755-arrow

368d5b0

PR review

7abe3da

hubertp added the CI: Ready to merge This PR is eligible for automatic merge label Jan 12, 2024

mergify bot merged commit 3c29a58 into develop Jan 12, 2024
26 of 27 checks passed

mergify bot deleted the wip/hubert/7755-arrow branch January 12, 2024 18:19

hubertp mentioned this pull request Feb 21, 2024

Arrow language follow up #9118

Closed

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Arrow language #8512

Arrow language #8512

hubertp commented Dec 11, 2023 •

edited

Loading

JaroslavTulach left a comment

JaroslavTulach commented Dec 13, 2023 •

edited by unfurl-links bot

Loading

Akirathan Dec 19, 2023

JaroslavTulach Jan 11, 2024

JaroslavTulach left a comment

JaroslavTulach Jan 10, 2024

JaroslavTulach Jan 11, 2024

JaroslavTulach Jan 11, 2024

hubertp commented Jan 11, 2024

JaroslavTulach commented Jan 11, 2024

hubertp commented Jan 12, 2024

hubertp commented Jan 12, 2024

Akirathan left a comment

Akirathan Jan 12, 2024

Akirathan commented Jan 12, 2024

Arrow language #8512

Arrow language #8512

Conversation

hubertp commented Dec 11, 2023 • edited Loading

Pull Request Description

Checklist

JaroslavTulach left a comment

Choose a reason for hiding this comment

JaroslavTulach commented Dec 13, 2023 • edited by unfurl-links bot Loading

Akirathan Dec 19, 2023

Choose a reason for hiding this comment

JaroslavTulach Jan 11, 2024

Choose a reason for hiding this comment

JaroslavTulach left a comment

Choose a reason for hiding this comment

JaroslavTulach Jan 10, 2024

Choose a reason for hiding this comment

JaroslavTulach Jan 11, 2024

Choose a reason for hiding this comment

JaroslavTulach Jan 11, 2024

Choose a reason for hiding this comment

hubertp commented Jan 11, 2024

JaroslavTulach commented Jan 11, 2024

hubertp commented Jan 12, 2024

hubertp commented Jan 12, 2024

Akirathan left a comment

Choose a reason for hiding this comment

Akirathan Jan 12, 2024

Choose a reason for hiding this comment

Akirathan commented Jan 12, 2024

hubertp commented Dec 11, 2023 •

edited

Loading

JaroslavTulach commented Dec 13, 2023 •

edited by unfurl-links bot

Loading