Epic: Polyfill Flexibility

[Shaptic]

Epic: Polyfill Flexibility

This is the master issue to track work necessary to tackle the inflexibility of the SDK in regards to dependencies and polyfills. It lives in this repo (stellar/js-stellar-sdk) but the work covers multiple repos.

First, the doc that put us on this path: link (see 2. Packaging).

Package Structure

We need our packages to be more modular:

• Drop polyfills from all packages
  • js-xdr
  • stellar-base
  • stellar-sdk
• Create a version of stellar-base with different crypto backends:
  • stellar-base will use sodium-native
  • stellar-base@unsalted will use tweetnacl
  • stellar-base@node will use Node's crypto (or a polyfill)
• Create a bundled version of the main libraries with everything polyfilled, installable with the @full tag, e.g. yarn add stellar-base@full:
  • stellar-base
  • stellar-sdk
  • These will be "combine-able" with the crypto backends
• Merge soroban-client and stellar-sdk into one library:
  • They will share the entire build pipeline except...
  • Soroban RPC code will live under a Soroban namespace
  • Horizon code will live under a Horizon namespace
  • XDR will continue to live under the xdr namespace
• Move strkey (SEP-23) implementation to its own package: @stellar/strkey
• Move SEP-10 utilities to their own package: @stellar/webauth

Problematic Dependencies

We need to eliminate the use of dependencies that are problematic (meaning: not available in every deployment environment, not cross-platform, not pleasant to use, etc.) or provide thorough instructions on what does or does not need to be polyfilled by the application developer.

The philosophy imbued here follows w3ctag/polyfills#6/comment. The libraries will make the best possible assumption about which library to use if some functionality is needed. For example, if dealing with bytes is commonplace in a library, it will rely on the existence of Buffer. However, as a library, it will not polyfill it on its own. Instead, it should detect the existence of the object at runtime and provide an appropriate error + instructions.

For example, if we started a project:

npm init -y && npm install stellar-sdk
echo 'const sdk = require("stellar-sdk");' > index.js
firefox index.js 

We would get an error in the console:

Error: 'Buffer' polyfill not detected. Configure your bundling pipeline to provide the Buffer class (e.g. https://github.com/feross/buffer).

The following dependencies need work:

• Buffer, which is problematic because it's only available in Node environments and can be replaced with (a) a custom wrapper over the native Uint8Array, (b) an explicit polyfill instruction, or (c) a full port over to Uint8Array.
  • in js-xdr
  • in stellar-base
  • in stellar-sdk
  • in stellar-sdk
• axios in stellar-sdk is problematic because it doesn't support HTTP/2 (see Stellar SDK doesn't fully support HTTP2 #612 and HTTP/2 support axios/axios#1175) and can't be used in a lot of environments (see Replace Axios with Fetch #241)
• eventsource is problematic because not everyone needs streaming support and SSE isn't supported in all environments
• sodium-native is problematic because it requires complicated compilation steps and can't always be elided (see Design Doc: Flexibility in choosing a cryptography provider js-stellar-base#646 for a proposed solution)

In fact, many of the dependencies can be replaced.

• in stellar-base, we have:
  • base32.js can be inlined
  • bignumber.js can use bigints plus Migrate code to use the native BigInt type wherever possible. js-stellar-base#607
  • buffer can be polyfilled or ported to use native Uint8Arrays (noted above)
  • crc can be inlined (see remove the dependency on crc by calculating checksum in strkey js-stellar-base#621)
  • sha.js can probably be inlined

[kalepail]

fwiw I don't personally think this needs to get shipped before Testnet. That seems very ambitious and I don't think it will greatly affect the ability of developers to build and deploy dapps. Definitely important but not essential. Apologies if I gave that impression. It will be a tremendous quality of life improvement but as it's not really Soroban specific and touches more generally all of the Stellar JS experience I think this can get punted downstream a little if necessary.