Plotting of large data sets fails

[bwbioinfo]

When plotting ~200M or more points, the plotting fails with the following error :

<--- Last few GCs --->

[4015030:0x5b7f43c3f000]     5983 ms: Mark-Compact (reduce) 1398.7 (1402.5) -> 1397.8 (1399.6) MB, pooled: 0 MB, 120.51 / 0.00 ms  (+ 0.1 ms in 0 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 121 ms) (average mu = 0.50[4015030:0x5b7f43c3f000]     6160 ms: Mark-Compact 1400.8 (1401.6) -> 1399.4 (1404.2) MB, pooled: 0 MB, 175.27 / 0.00 ms  (average mu = 0.294, current mu = 0.007) allocation failure; scavenge might not succeed


<--- JS stacktrace --->



#
# Fatal JavaScript out of memory: Reached heap limit
#

I don't see an option to adjust the memory settings but this might be an avenue to explore.

[LukaOber]

I don't want to be that guy, but why are you trying to create a plot with that many points? I would recommend using something like a lttb algorithm to reduce the number of points to plot. Can you provide us with more information about your use case? What renderer are you using? Can you provide an example? What version of charming are you using?

[bwbioinfo]

I'm trying to visualize chromosome-wide genomics events. I could downsample, but if I understand that error it seems to be failing at less than 2 gb, so we're not talking about an obscene amount of resources here.

I'm using the most recent version, pulled from the repo.

This is the function used for plotting :

pub fn plot_chromosome(data: DataFrame, output_path: &str) -> Result<(), Box<dyn Error>> {

    // Extract the columns we need
    let start_position: Vec<Option<i64>> =  data.column("chromStart").unwrap().i64().unwrap().into_iter().collect();
    let percent_modified = data.column("percent_modified").unwrap().f64().unwrap().to_vec();
    // Transpose the data into a format that can be used by the chart
    let datapoints = dz!(start_position, percent_modified);

    // get the chromosome name as a single string
    let chromosome_name = 
        data
        .column("chrom")
        .unwrap()
        .str()
        .unwrap()
        .get(0)
        .unwrap();

    // Create the chart
    let chart = Chart::new()
    // set the chromosome name as the title
    .title(Title::new().text(format!("Chromosome {} Modification Frequency", chromosome_name)))
    .x_axis(Axis::new())
    .y_axis(Axis::new())
    .series(
        Scatter::new()
        .symbol_size(5)
        .data(datapoints)
    );

    // Render the chart
    let mut renderer = 
        ImageRenderer::new(1500, 200)
        .theme(Theme::Default);
    // Render the chart as SVG string.
    renderer.render(&chart).unwrap();
    // Render the chart as PNG bytes.
    renderer.render_format(ImageFormat::Png, &chart).unwrap();
    // Save the chart as SVG file.
    // renderer.save(&chart, output_path).unwrap();
    // Save the chart as PNG file.
    _ = renderer.save_format(ImageFormat::Png, &chart, output_path);


    println!("Saved to: {}", output_path);

    Ok(())
}

[LukaOber]

The v8 engine seems to have a default memory limit per process of ~1.4GB on 64-bit machines. Can you try running your code while setting this env? V8_FLAGS: "--max-old-space-size=4096"

[LukaOber]

@bwbioinfo did it work? Do you need more help?

[l1x]

I have tried to set it:

 export 8_FLAGS="--max-old-space-size=4096"
 export 8_FLAGS="max-old-space-size=4096"

What is the actual way of setting this as env variables ot any other mean?

2025-02-24T11:26:42.586263Z DEBUG handlebars::render: Rendering value: Path(Relative(([Named("chart_option")], "chart_option")))

<--- Last few GCs --->

[96706:0x359250000]    15172 ms: Mark-Compact (reduce) 1399.5 (1403.4) -> 1398.4 (1401.2) MB, pooled: 0 MB, 456.21 / 0.00 ms  (+ 0.4 ms in 0 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 463 ms) (average mu = 0.113, cu[96706:0x359250000]    15668 ms: Mark-Compact (reduce) 1399.4 (1401.2) -> 1398.3 (1401.4) MB, pooled: 0 MB, 451.12 / 0.00 ms  (+ 0.2 ms in 0 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 455 ms) (average mu = 0.101, cu

<--- JS stacktrace --->



#
# Fatal JavaScript out of memory: Ineffective mark-compacts near heap limit
#
==== C stack trace ===============================

    0   etl                                 0x0000000103533118 v8::base::debug::StackTrace::StackTrace() + 24

[LukaOber]

I have tried to set it:

export 8_FLAGS="--max-old-space-size=4096"
export 8_FLAGS="max-old-space-size=4096"

It is V8_FLAGS not 8_FLAGS

What is the actual way of setting this as env variables ot any other mean?

How to set it permanently is os specific. I believe you can also set it with the cargo r command. You can get more info here: Working with Environment Variables