Regression: Coordinator crashes with OOM in 468 #24572
Comments
|
Executing same load on Memory consumption after 1.5 hours: |
|
OS OOM happens again |
|
@guyco33 can you do a heapdump when gc pressure goes up? |
|
I've run the same exact test queries for about 1 hour on both 
|
|
It looks we need to be looking for something off-heap (java heap seems fine). |
|
@losipiuk not really |
|
As for python: is potentially leaky. We are not closing Instance explicitly - just assuming GC will do the job - and IDK if it does not hold some native memory.Also it looks like we are not always have allocate/deallocate pairs. E.g for argTypeAddress and returnTypeAddress in PythonEngine.doSetup. IDK if that is a problem or not. @electrum PTAL.But this is probably not what @guyco33 is observing as this should not be called if Python functions are not in use. |
I'm using |
|
@losipiuk There is no close for There is a potential leak of That said, that seems unrelated to this issue, as there is no evidence they are being used here. |
|
I ran the same tests after deleting |
|
@guyco33 we don't allocate off-heap memory directly, the only known instances that I'm aware of is the usage of the Apache Arrow (BigQuery, Snowflake connectors) but this is not the case here. |
|
@guyco33 how much memory is assigned to the JVM process? Can you lower Xmx to 70% of the quota? |
|
We are also experiencing the same problem after upgrading to 468. We are using s3, hive metastore and iceberg through the jdbc connector. Due to this issue, the -XX:TrimNativeHeapInterval=5000 option was added among the JVM options, and in terms of metric, native heap memory is removed periodically and fluctuates, but it continues to increase and eventually OOM occurs.
|
|
@teddy-hackle @guyco33 can you do a thread dump? |
@wendigo I'm using 16G for Xmx on |
|
@guyco33 can you provide a thread dump? |
I used to observe behaviour (in different project) which suggested that AWS SDK is allocating stuff off-heap when doing PUTs for S3. |
|
Can one of you run Trino with JVM flag |
|
I'm worried about this Trino itself is not using ByteBuffer's as the Slice is wrapping around heap-allocated Is it possible for you to attach a debugger to a running process and check |
|
I think that these allocations are happening in Jetty: I've reported this issue to Jetty folks |
|
Attached output files of The
|
|
How did you downgrade them? |
|
Build |
|
Did the native memory (other) change with the downgrade? |
No. Same high value |
|
Not sure if it was the same reason, but I believe I faced the same issue on |
|
I have a production cluster running on |
|
@robinsinghstudios base image doesn't affect runtime |
|
Understood. Thanks @wendigo . |
|
I've tested two snapshot versions of a7e72d4 4718011 |
|
First 7 commits doesn't matter which leaves reactor, metrics, aws sdk, airlift |
|
|
so it seems to be the update of
|
|
Airlift 204 is among other things Jetty 12.0.16: https://github.com/airlift/airbase/releases/tag/204 |
|
I've created https://github.com/airlift/airlift/pull/1349/files to bound/reuse memory/heap allocations in Jetty |
|
I've just approved that running |
|
@guyco33 can you test with the airflit fix in place? |
|
@guyco33 airbase is just a set of dependency declarations. Airbase defines versions for airlift and airlift is used by Trino. We need Jetty 12.0.16 as it solves other issue that we were facing so it's not really an option to downgrade. |
|
We've filled jetty/jetty.project#12670 to track Jetty improvement |
Coordinator in
468consumes much more memory than467that leads to OS OOM.What could be the reason for such memory consumption change ?
Listed below the output from free memory stats and gc logs during execution from both coordinator versions.
Trino coordinator is running on EC2 machine type
r6a.xlargewithOpenJDK Runtime Environment Temurin-23.0.1+11 (build 23.0.1+11)jvm.conf is
The text was updated successfully, but these errors were encountered: