Attached workflow breaks in GUI2

[AdRiley]

Reported by @jdunkerley

The attached workflow does not work correctly in GUI2.

ESG_Employee_Figures.zip

@jdunkerley will be able to provide the data files.

[jdunkerley]

Here is the rough summary:

There is an error in one of the grouped functions (build_energy_source).

When trying to repair this, the GUI won't evaluate (possibly due to the order of nodes being incorrect).
If you try and fix with the code editor (e.g. sorting the imports at the top of the file or manually fixing the function) then the file is not correctly saved.

In the end, once we deleted the metadata from the file we were able to repair it (though was still painful).

Editing any collapsed function also appears to cause a complete recalculation which likewise makes the process very painful.

With the metadata deleted, the main function was laid out correctly but grouped functions were laid out going upwards not downwards.

[farmaazon]

Without data, I cannot follow those steps: I have Unauthorized error.

There is, however, another bug: When opening this project and showing code editor, it seems to be constantly re-evaluated without my action (the errors flicker).

[jdunkerley]

Data files used are:
gridData_summary.csv
Commuting.xlsx
BuildingUsage.csv
buildingEnergyStats.zip

For the postgres access I will message you separately

[farmaazon]

Thanks, @jdunkerley. Yes, the project is very unstable. Actually, single entering node + trying to fix paths actually freezes IDE.

Investigation in progress.

[enso-bot]

Adam Obuchowicz reports a new STANDUP for today (2024-02-27):

Progress: Took another issue with highest priority. investigated and fixed one problem discovered in James's project, then found the function which freezes our ydocs server during some edit. It should be finished by 2024-03-04.

Next Day: Next day I will be working on the same task. Fix the freeze problem and check another issues.

[farmaazon]

I know what is the problem: its combination of several factors:

1. this project's 100 lines of code produced 240KB of idmap
2. After a single edit, the idmap seems completely shuffled. The expected diff is large
3. We try to apply O(ND) algorithm and are choked.

Not sure if there is a workaround. We could use the stupid O(1) algorithm from the old GUI... but in that case we likely send 240KB of edit to the engine on every edit, what is far from ideal.

The better solution would be to reduce size of idmap, (already reported as an issue) - however, we must be very efficient. Currently, a single entry takes 82-86 bytes; even if we reduce it to 12 bytes (two words for span boundary - may not suffice, and 8 bytes for UUID) we quickly get back to ~250KB when the modules will grow - and even a module of 1000 lines should be considered reasonable.

There is also an idea of getting rid of IdMap from file entirely, and make separate API for sharing Ids of AST nodes. In this API we could make a smarter diffing.

[farmaazon]

@jdunkerley you may check package with fixes: https://github.com/enso-org/enso/actions/runs/8079248374?pr=9207 I didn't check yet the problem of nodes ordering after removing metadata.

Also, I saw many duplicated imports - were they added by GUI?

GitHub

Add workaround for huge idmap diffs · enso-org/enso@f45963a

Hybrid visual and textual functional programming. Contribute to enso-org/enso development by creating an account on GitHub.

[jdunkerley]

Definitely seems a lot better, thanks @farmaazon

[jdunkerley]

Also, I saw many duplicated imports - were they added by GUI?

Yes still an issue.

[enso-bot]

Adam Obuchowicz reports a new STANDUP for yesterday (2024-02-28):

Progress: Found the core issue of the freeze and implemeted a workaround. Progressed a bit with drop-down filtering. It should be finished by 2024-03-04.

Next Day: Next day I will be working on the same task. Apply review; I was asked to check more closely what threshold we should pick after which diff take too much time.

[enso-bot]

Adam Obuchowicz reports a new STANDUP for today (2024-02-29):

Progress: Add benchmarks showing diff/fast diff efficiency on our metadata. Checked new tasks in backlog, made triage of some bug reports. Also bookclub and reviews. It should be finished by 2024-03-04.

Next Day: Next day I will be working on the #8942 task. I need to look at transient failures again. If time allows, progress more with widget input filtering.

[farmaazon]

Made follow-up issues: #9255 and #9257.

I cannot reproduce the problem with imports. Let's see if it will happen in future projects.