pcli q dex simulate reports unclaimed arbs

[hdevalence]

Describe the bug
Running pcli q dex simulate gave the following output:

~ $ pcli q dex simulate 1000gm --into gn
Scanning blocks from last sync height 86484 to latest height 86484
[0s] ██████████████████████████████████████████████████       0/0       0/s ETA: 0s
507.771569gm => 506.96109gn via:
    Trace                                                                    Subprice
    183.161906gm =>                                            183.070325gn  1.000501gm/1gn
    200.100051gm =>                                            200gn         1.000501gm/1gn
    2.019324gm =>    6.441643penumbra =>                       2.013013gn    1.003136gm/1gn
    2.02313gm =>     2.013014gn =>        6.441643penumbra =>  2.013013gn    1.005026gm/1gn
    2.02313gm =>     2.013014gn =>        6.441643penumbra =>  2.013013gn    1.005026gm/1gn
    2.02313gm =>     2.013014gn =>        6.441643penumbra =>  2.013013gn    1.005026gm/1gn
    2.02313gm =>     2.013014gn =>        6.441643penumbra =>  2.013013gn    1.005026gm/1gn
    2.02313gm =>     2.013014gn =>        6.441643penumbra =>  2.013013gn    1.005026gm/1gn
    2.02313gm =>     2.013014gn =>        6.441643penumbra =>  2.013013gn    1.005026gm/1gn
    2.02313gm =>     2.013014gn =>        6.441643penumbra =>  2.013013gn    1.005026gm/1gn
    2.02313gm =>     2.013014gn =>        6.441643penumbra =>  2.013013gn    1.005026gm/1gn
    2.02313gm =>     2.013014gn =>        6.441643penumbra =>  2.013013gn    1.005026gm/1gn
    2.02313gm =>     2.013014gn =>        6.441643penumbra =>  2.013013gn    1.005026gm/1gn
    2.02313gm =>     2.013014gn =>        6.441643penumbra =>  2.013013gn    1.005026gm/1gn
    2.02313gm =>     2.013014gn =>        6.441643penumbra =>  2.013013gn    1.005026gm/1gn
    2.02313gm =>     2.013014gn =>        6.441643penumbra =>  2.013013gn    1.005026gm/1gn
    551.244mgm =>    548.487mgn =>        1.755158penumbra =>  548.486mgn    1.005029gm/1gn
    551.244mgm =>    548.487mgn =>        1.755158penumbra =>  548.486mgn    1.005029gm/1gn
    2.02313gm =>     2.013014gn =>        6.441643penumbra =>  2.013013gn    1.005026gm/1gn
    2.02313gm =>     2.013014gn =>        6.441643penumbra =>  2.013013gn    1.005026gm/1gn
    2.02313gm =>     2.013014gn =>        6.441643penumbra =>  2.013013gn    1.005026gm/1gn
    2.02313gm =>     2.013014gn =>        6.441643penumbra =>  2.013013gn    1.005026gm/1gn
    2.02313gm =>     2.013014gn =>        6.441643penumbra =>  2.013013gn    1.005026gm/1gn
    2.02313gm =>     2.013014gn =>        6.441643penumbra =>  2.013013gn    1.005026gm/1gn
    2.02313gm =>     2.013014gn =>        6.441643penumbra =>  2.013013gn    1.005026gm/1gn
    2.02313gm =>     2.013014gn =>        6.441643penumbra =>  2.013013gn    1.005026gm/1gn
    2.02313gm =>     2.013014gn =>        6.441643penumbra =>  2.013013gn    1.005026gm/1gn
    2.02313gm =>     2.013014gn =>        6.441643penumbra =>  2.013013gn    1.005026gm/1gn
    2.02313gm =>     2.013014gn =>        6.441643penumbra =>  2.013013gn    1.005026gm/1gn
    2.02313gm =>     2.013014gn =>        6.441643penumbra =>  2.013013gn    1.005026gm/1gn
    2.02313gm =>     2.013014gn =>        6.441643penumbra =>  2.013013gn    1.005026gm/1gn
    2.02313gm =>     2.013014gn =>        6.441643penumbra =>  2.013013gn    1.005026gm/1gn
    2.02313gm =>     2.013014gn =>        6.441643penumbra =>  2.013013gn    1.005026gm/1gn
    2.02313gm =>     2.013014gn =>        6.441643penumbra =>  2.013013gn    1.005026gm/1gn
    2.02313gm =>     2.013014gn =>        6.441643penumbra =>  2.013013gn    1.005026gm/1gn
    2.02313gm =>     2.013014gn =>        6.441643penumbra =>  2.013013gn    1.005026gm/1gn
    2.02313gm =>     2.013014gn =>        6.441643penumbra =>  2.013013gn    1.005026gm/1gn
    2.02313gm =>     2.013014gn =>        6.441643penumbra =>  2.013013gn    1.005026gm/1gn
    2.02313gm =>     2.013014gn =>        6.441643penumbra =>  2.013013gn    1.005026gm/1gn
    2.02313gm =>     2.013014gn =>        6.441643penumbra =>  2.013013gn    1.005026gm/1gn
    2.02313gm =>     2.013014gn =>        6.441643penumbra =>  2.013013gn    1.005026gm/1gn
    2.02313gm =>     2.013014gn =>        6.441643penumbra =>  2.013013gn    1.005026gm/1gn
    2.02313gm =>     2.013014gn =>        6.441643penumbra =>  2.013013gn    1.005026gm/1gn
    2.02313gm =>     2.013014gn =>        6.441643penumbra =>  2.013013gn    1.005026gm/1gn
    2.02313gm =>     2.013014gn =>        6.441643penumbra =>  2.013013gn    1.005026gm/1gn
    2.02313gm =>     2.013014gn =>        6.441643penumbra =>  2.013013gn    1.005026gm/1gn
    2.02313gm =>     2.013014gn =>        6.441643penumbra =>  2.013013gn    1.005026gm/1gn
    2.02313gm =>     2.013014gn =>        6.441643penumbra =>  2.013013gn    1.005026gm/1gn
    2.02313gm =>     2.013014gn =>        6.441643penumbra =>  2.013013gn    1.005026gm/1gn
    2.02313gm =>     2.013014gn =>        6.441643penumbra =>  2.013013gn    1.005026gm/1gn
    2.02313gm =>     2.013014gn =>        6.441643penumbra =>  2.013013gn    1.005026gm/1gn
    2.02313gm =>     2.013014gn =>        6.441643penumbra =>  2.013013gn    1.005026gm/1gn
    2.02313gm =>     2.013014gn =>        6.441643penumbra =>  2.013013gn    1.005026gm/1gn
    2.02313gm =>     2.013014gn =>        6.441643penumbra =>  2.013013gn    1.005026gm/1gn
    2.02313gm =>     2.013014gn =>        6.441643penumbra =>  2.013013gn    1.005026gm/1gn
    2.02313gm =>     2.013014gn =>        6.441643penumbra =>  2.013013gn    1.005026gm/1gn
    2.02313gm =>     2.013014gn =>        6.441643penumbra =>  2.013013gn    1.005026gm/1gn
    2.02313gm =>     2.013014gn =>        6.441643penumbra =>  2.013013gn    1.005026gm/1gn
    2.02313gm =>     2.013014gn =>        6.441643penumbra =>  2.013013gn    1.005026gm/1gn
    2.02313gm =>     2.013014gn =>        6.441643penumbra =>  2.013013gn    1.005026gm/1gn
    2.02313gm =>     2.013014gn =>        6.441643penumbra =>  2.013013gn    1.005026gm/1gn
    2.02313gm =>     2.013014gn =>        6.441643penumbra =>  2.013013gn    1.005026gm/1gn
    2.02313gm =>     2.013014gn =>        6.441643penumbra =>  2.013013gn    1.005026gm/1gn
    2.02313gm =>     2.013014gn =>        6.441643penumbra =>  2.013013gn    1.005026gm/1gn
    2.02313gm =>     2.013014gn =>        6.441643penumbra =>  2.013013gn    1.005026gm/1gn
~ $

There is one thing wrong here and one thing that's suspicious.

1. This is reproducible (at least at the moment), implying that an arb involving the penumbra token persists across block boundaries, which shouldn't happen.
2. The exact same values appear over and over again, so either there are many of the exact same position (unlikely) or something is wrong with the execution (could this be dex: fix frontier cache incoherence #4283 @erwanor ?)

Can we check the log entries for this block height to see if the arb is erroring out and cancelling @conorsch ?

[erwanor]

We hit the price limit for that block a little bit later:

DEBUG abci:EndBlock{height=86484}:end_block{height=86484}:dex:arbitrage:route_and_fill:path_search{max_hops=6}: penumbra_dex::component::router::path_search: found path price=1 spill_price=1.0126984126984127 src=passet1984fctenw8m2fpl8a9wzguzp7j34d7vravryuhft808nyt9fdggqxmanqm nodes=[]

...

DEBUG abci:EndBlock{height=86484}:end_block{height=86484}:dex:arbitrage:route_and_fill:path_search{max_hops=6}: penumbra_dex::component::router::path_search: path too expensive price=1 price_limit=1

[zbuc]

It seems kind of odd to ever perform this routing, since you lose gn in the process:

2.013014gn => 6.441643penumbra => 2.013013gn

[hdevalence]

It seems like this could have been caused by #4283 ?

[zbuc]

This doesn't seem to be a bug involving unclaimed arb and is a side effect of the design of our routing algorithm. @erwanor is working on implementing some changes to speed up routing and additional dex metrics will provide observability into dex routing performance. I don't think there are any changes that need to happen specific to this issue.