Recursion: Raw transaction endpoint for inscriptions

[alpergundogdu]

We've talked about this endpoint a few months ago with @raphjaph on "Best in Slot <> ord" telegram channel and he mentioned there are other teams looking for a recursive raw transaction endpoint and invited us to open a PR for it.

In our use-case, we want to be able to track the inscriptions' old pkscripts and change the artwork based on some calculations over them. We can get the current txid via /r/inscription/<INSCRIPTION_ID>, we can get the genesis txid via inscription_id so we can apply sat movement rules to current sat location and track the inscription back to the genesis txid.

In order to do that, we need vout (pkscript and value), vin (txid and vout), and also blockhash to get the blocktime of the target transaction (we can use /r/blockinfo/<QUERY> to get the blocktime from blockhash).

I'll attach a PR to this issue in a bit, we updated the code per casey@'s initial suggestion on PR 4100.

Adding just the txhex and parsing client side could be error-prone if a new transaction version is introduced, the parser may stop working and the inscription may lose all history after a single new version tx.

For now, we chose a limited, stable set needed for base ordinals sat tracking, so we can assume these will always be in bitcoin as long as inscriptions work.

Verbose endpoint can keep some optional future fields in case another team needs more information about a tx and this information is also part of bitcoin and/or ordinals rules so it will always be available, new fields can always be added to the response but as casey@ mentioned we can never remove a field after it is introduced so we tried to limit the response fields as much as possible.

[alpergundogdu]

Opened PR 4109 for this.

[raphjaph]

Adding just the txhex and parsing client side could be error-prone if a new transaction version is introduced, the parser may stop working and the inscription may lose all history after a single new version tx.

Transactions have version fields for exactly that reason. Whatever you use client side to parse transactions from hex should handle new versions and be able to parse the subset of fields you're interested in.

I understand the issue but now the onus flips on us to figure out what to return if a future transaction type doesn't have those "stable" fields you suggested. You could build in an upgrade mechanism such that if the sat is reinsribed with a new parsing library that will be used instead.

[samedcildir]

We don't want to use an upgradable script since it defeats one of the main purposes of inscriptions which is to be immutable and to be completely in control of the art that you own.

The same issue also exists in block info endpoint since the fields may also change there but the endpoint has been added instead of a raw block header hex. So I believe if the subset is stable enough (such as pkscript and value of an output and txid and output index of an input will always be needed for both utxo tracking and ordinal theory and blockhash will always be there as long as transactions are kept in a block) we can introduce them to let people experiment with them and make art that was impossible before. If in a very unlikely event one of the fields are removed, we can set them to null or even unknown and document the possibility of this change in the documentation (the same will probably happen to block info endpoint in the unlikely event of a field change)

[casey]

I think there's a little confusion here.

It is not possible for a soft fork to change the transaction format such that old, un-upgraded clients see the transaction as invalid.

Transaction version numbers do not change this.

As such, a JS transaction parsing library would not need to be upgradable, and should keep working indefinitely.

Before adding a recursive endpoint that returns a parsed transaction, I'd like to add a recursive endpoint which returns the transaction as binary or hex.

[samedcildir]

Yes it makes sense, I've missed this difference between soft-fork and hard-fork.

We'll open another PR with recursive endpoint for raw transaction hex 👍

[samedcildir]

Hello @casey, we've opened the pull request (PR 4147) for tx hex recursive endpoint. It mostly uses already existing functions and adds a very little new code for the new functionality.

[samedcildir]

The PR4147 meets our needs for what we want to achieve with our art project, and we appreciate your time.

We’ve also put more thought into the soft-fork changes and immutable tx parser library. Even though raw txhex endpoint is enough to generate the art, it seems like a parsed endpoint may also be needed at one point and an immutable tx parser may not be able to parse the RPC response after a soft-fork.

A soft fork should not change the tx parsing but this mostly applies to P2P communication instead of RPC responses. For example, after SegWit soft fork, old-nodes received the old serialised version of the transaction from P2P network when they request it but the new nodes received the new version. So they actually changed the serialization of transactions in a way that the old nodes cannot parse and sees them as invalid txes but they added a backwards-compatibility layer to P2P network.

The SegWit change added Marker and Flag fields after Version bytes and they are set to 00 and 01 respectively. The old parsing method parsed them as a tx with 0 input and 1 output which is actually an invalid transaction. So even though SegWit was a soft-fork with backwards compatible P2P communication, RPC responses changed and the parsing libraries for those responses changed as well.

Reference: Tx Structure Marker & Flag
How P2P Backwards compatibility achieved

[gmart7t2]

Transaction version numbers do not change this.

As such, a JS transaction parsing library would not need to be upgradable, and should keep working indefinitely.

When the transaction version number is bumped from 2 to 3, the old code will be able to detect that lots of transactions are now using a new version, but won't be able to decode those transactions.

You can say the on-chain tx parser is "working indefinitely", but it is no longer useful if the majority of new transactions use a format it cannot decode.

[emilcondrea]

Depending on how this gets implemented it can be an attack vector if it relies directly on btc node.
A malicious inscription can consume all bandwidth and make node unavailable, it could have a seed of starting txids and navigate the whole chain. Some rate limiting on the recursive endpoint would alleviate this.

[samedcildir]

This new endpoint actually works the same way as "/r/blockinfo/{query}" this endpoint which also directly asks to the BTC Node. A malicious inscription can traverse all blocks in random order for almost the same effect. Also, all recursive endpoints touch a db that is also used for indexing (index.redb or btc node's db) and all of them can act as a DoS attack vector against the indexer. I believe there should be a global rate-limiting for all recursive endpoints but that should be another Issue. I think currently most public ord servers use Cloudflare rules to avoid this situation.

[casey]

The transaction version number bump does not change the format or structure of transactions, just the value of an integer.

The format of transactions only changes for major upgrades, so far the only one of which has been segwit. If there's a future major upgrade which changes the transaction format, we'll just make sure to return transactions in the old format, which is guaranteed to be possible.