Slow printing of layout if filtered legend contains a layer that has a symbology classified using a virtual attribute

[stafoo]

What is the bug or the crash?

Filtered legends in layouts slow down printing process in QGIS 3.34.5 and 3.36.1.
It happens when one layer has a symbology classified using a virtual attribute (such as a symbol showing only elements that are selected via is_selected() expression, as described here: #57230 (comment))
This behavios was not observed in QGIS 3.28.10 with the same project.

In the video example below, activating the filter option in legend significantly slows down the print process:

• with filtered legend: 30 seconds
• without filtered legend: 2 seconds

Registra_2024_04_23_15_55_37_919_compresso.mp4

This becomes even more critical with multipage layouts and with projects containing several layers.
This problem does not seems to be related to atlas.

Steps to reproduce the issue

1. create a layer with a virtual attribute (in my case I used is_selected() expression; 1 means the element is selected, 0 not selected)
2. classify the layer symbology using the virtual attribute
3. create a layout (with map + legend)
4. activate the "filtered legend" option, then print the layout to pdf (very slow)
5. de-activate the "filtered legend" option, print the layout to pdf again (this time you will notice how it is much faster)

Versions

Versione di QGIS | 3.34.5-Prizren
Revisione codice QGIS | 4b30849
Versione Qt | 5.15.3
Versione Python | 3.9.18
Versione GDAL/OGR | 3.8.4
Versione PROJ | 9.3.1
Versione database del Registro EPSG | v10.098 (2023-11-24)
Versione GEOS | 3.12.1-CAPI-1.18.1
Versione SQLite | 3.41.1
Versione PDAL | 2.6.0
Versione client PostgreSQL | 16.2
Versione SpatiaLite | 5.1.0
Versione QWT | 6.1.6
Versione QScintilla2 | 2.13.4
Versione SO | Windows 10 Version 2009

Plugins Python attivi
AnotherDXF2Shape | 1.3.1
contour | 2.0.13
felt | 2.0.1
LAStools | 2.1.0
mmqgis | 2021.9.10
nominatim_locator_filter | 0.3.2
OnlineRoutingMapper | 0.9
profiletool | 4.2.6
qfieldsync | v4.9.1
quick_map_services | 0.19.34
rivergis | 3.0
slyr_community | 5.0.0
StreetView | 3.2
db_manager | 0.1.20
grassprovider | 2.12.99
MetaSearch | 0.3.6
processing | 2.12.99

Supported QGIS version

• I'm running a supported QGIS version according to the roadmap.

New profile

• I tried with a new QGIS profile

[stafoo]

This problem is still present in QGIS 3.34.6-1

[stafoo]

I investigated the error a bit.
It seems that the slowness happens only if the filtered legend contains a layer that has a symbology classified using a virtual attribute.

Making the attribute Selezionato a "real" attribute (and not a virtual one), the printing process is fast even if the legend is filtered.

[agiudiceandrea]

@stafoo thanks for reporting. It would be useful if you provided a sample project and layet(s) with which the issue occurs.

[stafoo]

TestBug57230_project_gpkg.zip

Sample project attached. Steps to reproduce:

1. open the project, select one element (or more) in layer (the virtual attribute "selected" is now updated with 'true' in the selected elements)
2. open the print layout and export a pdf (slow print)
3. uncheck the filter "Only show items inside linked maps" in the legend option, then print (fast print)
   This test project is made with QGIS 3.34.6

[agiudiceandrea]

I can confirm the issue on Windows using either QGIS 3.36.2 or QGIS 3.34.6. The issue didn't occur using QGIS 3.28.15.

I also see a glitch in the legend's dimensions which incorrectly change when the "Only show items inside linked maps" checkbox is checked /unchecked

[stafoo]

Update on the bug.
The slow printing problem is present even if the Legend in the layout does not contain the "layer" object.
If you remove "layer" from the objects shown in the legend and you print the layout, it will still be slow (with filtered legend).
In addition, Layer must to be visible in the layout map to notice the slow printing.

• removing the virtual attribute filled by the expression is_selected(), the printing process is always fast also with the "Only show items inside linked maps" option checked
• with the virtual attribute filled by the expression is_selected(), the printing process is slow ONLY if the elements of the Layer are shown in the layout map; the "Only show items inside linked maps" option must be checked
• with the virtual attribute filled by the expression is_selected(), the printing process is fast if the elements of the layer are NOT shown in the layout map, even if the "Only show items inside linked maps" option is checked

[elpaso]

I've spent quite some time trying to find a solution but I couldn't.

The issue comes from the hit test task QgsMapHitTestTask which is run in a separate thread by the task manager and gets locked on a semaphore here: https://github.com/qgis/QGIS/blob/master/src/core/expression/qgsexpressionutils.cpp#L179

this happens because the is_selected function like may others is invoked in the layer's thread which happens to be the main thread which is waiting for the hit test to finish.

The 30 seconds come from the if ( mNotFinishedMutex.tryLock( timeout ) ) so this is basically deadlock which eventually times out and by chance continues without errors.

@nyalldawson do you have any recommendation about how to handle this?
I thought about cloning the layers into the hit test thread but I'm not sure it is a viable solution.

[nyalldawson]

@elpaso

Hmm, very tricky

The only safe approach I can think would be to have some QgsPreparedExpressionData" which we generate in the main thread when creating that task. And then a bunch of plumbing to get that object passed to the expression at evaluation time (maybe attaching it to the expression context?). And then if the prepared expression data is available, we use it instead of calling the map layer function direct.
And then we prepopulate the expression data with values we need for functions like this, storing them in a variant map for thread safety.

[stafoo]

@elpaso I have tested again this bug with this build: #57649 (comment)

As noted by @m-kuhn (#57649 (comment)) the expression is_selected still slows down the printing process.

My 2 cents

[stafoo]

Bug confirmed with QGIS 3.34.9