Add note about E2E tests in README (#160)

* Check submission and decision deposit refunds

In the current governance lifecycle test.

* Check registrar addition event in people chain XCM tests

* Add missing event checks to XCM root calls

In referendum cancelling and killing tests via root XCMs.

This commit also adds `cancel/kill` calls with a signed origin, which
are meant to fail, and records the `system::ExtrinsicFailed` events
in the snapshots.

* Rename people chains' tests and snapshots

* Rename governance tests and snapshots

* Add note to README.md about new E2E tests

* Add note to `README` about current E2E tests

* update snapshots

* update readme

---------

Co-authored-by: Bryan Chen <[email protected]>

README.md

@@ -14,6 +14,9 @@ yarn test
 # Run tests for specific chain
 yarn test <chain>
 
+# Run specific test
+yarn test -t <test-name> #<test-name> is what was passed to `vitest.test()`, usually inside a `describe()` suite
+
 # Run with Vitest UI
 yarn test:ui
 
@@ -50,13 +53,44 @@ LOG_LEVEL=info             # General logging (error/warn/info/debug/trace)
 
 ### Project Structure
 - `packages/shared/src/xcm`: Common XCM test suites
+- `package/shared/src/*.ts`: Common utilities for E2E tests.
 - `packages/kusama/src`: Kusama network tests
 - `packages/polkadot/src`: Polkadot network tests
 
+### About end-to-end tests
+
+This repository contains E2E tests for the Polkadot/Kusama networks.
+
+These include:
+- E2E test suite to the people chains in both networks. This suite contains scenarios such as
+  - Adding, modifying, and removing identities
+  - Requesting judgement requests on registrars, and providing it
+  - Adding registrars to the people chain by sending, from the relay chain, an XCM call with root origin
+  - Adding, modifying, and removing subidentities for an account
+- E2E suite for governance infrastructure - referenda, preimages, and conviction voting. It includes
+  - Creating a referendum for a treasury proposal, voting on it
+  - Cancelling and killing referenda with XCM root-originated calls
+  - Noting and unnoting preimages
+
+The intent behind these end-to-end tests is to cover the basic behavior of relay chains' and system
+parachains' runtimes.
+
+Initial coverage can be limited to critical path scenarios composed of common extrinsics
+from each of a runtime's pallets, and from there test more complex interactions.
+
+Note that since block execution throughput in `chopsticks` on a local development machine is limited
+to roughly `1` and `10` blocks/second, not all scenarios are testable in practice e.g. referenda
+confirmation, or the unbonding of staked funds.
+Consider placing such tests elsewhere, or using different tools (e.g. XCM emulator).
+
 ### Test Guidelines
 - Write network-agnostic tests where possible
 - Handle minor chain state changes gracefully
 - Use `.redact()` for volatile values
+  - Pass `{ number: n }` to `.redact()` to explicitly redact all but the `n` most significant digits
+  - Pass `{ removeKeys: new RegExp(s) }` to remove keys from an object that are unwanted when e.g.
+    using `toMatchObject/toMatchSnapshot`. `s` can contain several fields e.g.
+    `"alarm|index|submitted"`. Check [this page](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/RegExp) for how to use `RegExp`.
 - Leverage snapshots for easier maintenance
 - Follow naming convention: `<chain1>.<chain2>.test.ts` or `<chain1>.test.ts`
 
@@ -66,11 +100,27 @@ LOG_LEVEL=info             # General logging (error/warn/info/debug/trace)
 3. Create notification issue
 4. Update `.github/workflows/notifications.json`
 
+### Adding new E2E tests
+1. Create a file in `packages/shared/src/` with the E2E tests, and their required utilities
+  - This assumes that the E2E test will run on Polkadot/Kusama: its code being shared makes it
+    reusable on both chains
+2. Using the shared utilities created in the previous step, create Polkadot/Kusama tests in
+  `packages/polkadot/src`/`packages/kusama/src`, respectively.
+3. Run the newly created tests so their snapshots can be created in `packages/<network>/src/__snapshots__`
+  - Inspect the snapshots, and make corrections to tests as necessary - or upstream, if the test
+    has revealed an issue with e.g. `polkadot-sdk`
+4. Craete a PR with the new tests.
+
+### Regenerate Snapshots
+
+It is recommended to regenerate snapshots when renaming or removing tests. This can be done by deleting `__snapshots__` folders and running `yarn test -u`.
+
 ### Debugging Tips
 - Use `{ only: true }` to isolate tests
 - Add logging to shared test suites
 - Insert `await chain.pause()` for state inspection
 - Connect via Polkadot.js Apps to paused chains
+  - Check the logs of the terminal running the `.pause`d test for the address and port
 - Carefully review snapshot changes
 
 ### Block Number Management

packages/kusama/src/__snapshots__/kusama.governance.e2e.test.ts.snap → packages/kusama/src/__snapshots__/kusamaGovernance.e2e.test.ts.snap

@@ -24,6 +24,26 @@ exports[`Kusama Governance > preimage submission, query and removal works > unno
 ]
 `;
 
+exports[`Kusama Governance > referendum lifecycle test - submission, decision deposit, various voting should all work > cancelling referendum with signed origin 1`] = `
+[
+  {
+    "data": {
+      "dispatchError": "BadOrigin",
+      "dispatchInfo": {
+        "class": "Normal",
+        "paysFee": "Yes",
+        "weight": {
+          "proofSize": "(rounded 84000)",
+          "refTime": "(rounded 640000000)",
+        },
+      },
+    },
+    "method": "ExtrinsicFailed",
+    "section": "system",
+  },
+]
+`;
+
 exports[`Kusama Governance > referendum lifecycle test - submission, decision deposit, various voting should all work > charlie's class locks after their vote's rescission 1`] = `
 [
   [
@@ -378,6 +398,32 @@ exports[`Kusama Governance > referendum lifecycle test - submission, decision de
 }
 `;
 
+exports[`Kusama Governance > referendum lifecycle test - submission, decision deposit, various voting should all work > refund of decision deposit 1`] = `
+[
+  {
+    "data": {
+      "amount": "(rounded 33000000000)",
+      "who": "HJzQySPFxy81SD4wVMbJvZjJufYV1C8zKEVL7y3h4tbRbyR",
+    },
+    "method": "DecisionDepositRefunded",
+    "section": "referenda",
+  },
+]
+`;
+
+exports[`Kusama Governance > referendum lifecycle test - submission, decision deposit, various voting should all work > refund of submission deposit 1`] = `
+[
+  {
+    "data": {
+      "amount": "(rounded 33000000000)",
+      "who": "FfmSiZNJP72xtSaXiP2iUhBwWeMEvmjPrxY2ViVkWaeChDC",
+    },
+    "method": "SubmissionDepositRefunded",
+    "section": "referenda",
+  },
+]
+`;
+
 exports[`Kusama Governance > referendum lifecycle test - submission, decision deposit, various voting should all work > removal of votes in cancelled referendum 1`] = `
 [
   {
@@ -452,3 +498,23 @@ exports[`Kusama Governance > referendum lifecycle test - submission, decision de
   },
 ]
 `;
+
+exports[`Kusama Governance > referendum lifecycle test 2 - submission, decision deposit, and killing should work > killing referendum with signed origin 1`] = `
+[
+  {
+    "data": {
+      "dispatchError": "BadOrigin",
+      "dispatchInfo": {
+        "class": "Normal",
+        "paysFee": "Yes",
+        "weight": {
+          "proofSize": "(rounded 84000)",
+          "refTime": "(rounded 720000000)",
+        },
+      },
+    },
+    "method": "ExtrinsicFailed",
+    "section": "system",
+  },
+]
+`;

packages/polkadot/src/__snapshots__/polkadot.governance.e2e.test.ts.snap → packages/polkadot/src/__snapshots__/polkadotGovernance.e2e.test.ts.snap

@@ -24,6 +24,26 @@ exports[`Polkadot Governance > preimage submission, query and removal works > un
 ]
 `;
 
+exports[`Polkadot Governance > referendum lifecycle test - submission, decision deposit, various voting should all work > cancelling referendum with signed origin 1`] = `
+[
+  {
+    "data": {
+      "dispatchError": "BadOrigin",
+      "dispatchInfo": {
+        "class": "Normal",
+        "paysFee": "Yes",
+        "weight": {
+          "proofSize": "(rounded 84000)",
+          "refTime": "(rounded 560000000)",
+        },
+      },
+    },
+    "method": "ExtrinsicFailed",
+    "section": "system",
+  },
+]
+`;
+
 exports[`Polkadot Governance > referendum lifecycle test - submission, decision deposit, various voting should all work > charlie's class locks after their vote's rescission 1`] = `
 [
   [
@@ -378,6 +398,32 @@ exports[`Polkadot Governance > referendum lifecycle test - submission, decision
 }
 `;
 
+exports[`Polkadot Governance > referendum lifecycle test - submission, decision deposit, various voting should all work > refund of decision deposit 1`] = `
+[
+  {
+    "data": {
+      "amount": 10000000000,
+      "who": "15jftzMaVPDfhKQ98RbYZ82t1wNxNdw6cS8E6kgSmMhcrxVz",
+    },
+    "method": "DecisionDepositRefunded",
+    "section": "referenda",
+  },
+]
+`;
+
+exports[`Polkadot Governance > referendum lifecycle test - submission, decision deposit, various voting should all work > refund of submission deposit 1`] = `
+[
+  {
+    "data": {
+      "amount": 10000000000,
+      "who": "146SvjUZXoMaemdeiecyxgALeYMm8ZWh1yrGo8RtpoPfe7WL",
+    },
+    "method": "SubmissionDepositRefunded",
+    "section": "referenda",
+  },
+]
+`;
+
 exports[`Polkadot Governance > referendum lifecycle test - submission, decision deposit, various voting should all work > removal of votes in cancelled referendum 1`] = `
 [
   {
@@ -452,3 +498,23 @@ exports[`Polkadot Governance > referendum lifecycle test - submission, decision
   },
 ]
 `;
+
+exports[`Polkadot Governance > referendum lifecycle test 2 - submission, decision deposit, and killing should work > killing referendum with signed origin 1`] = `
+[
+  {
+    "data": {
+      "dispatchError": "BadOrigin",
+      "dispatchInfo": {
+        "class": "Normal",
+        "paysFee": "Yes",
+        "weight": {
+          "proofSize": "(rounded 84000)",
+          "refTime": "(rounded 640000000)",
+        },
+      },
+    },
+    "method": "ExtrinsicFailed",
+    "section": "system",
+  },
+]
+`;

packages/shared/src/governance.ts

@@ -88,11 +88,12 @@ function referendumCmp(
  *   4.2. using a split vote
  *   4.3. using a split-abstain vote
  * 5. cancelling the referendum using the scheduler to insert a `Root`-origin call
- *   5.1 checking that submission/decision deposits are refunded
+ *   5.1 checking that locks on submission/decision deposits are released
  *   5.2 checking that voters' class locks and voting data are not affected
  * 6. removing the votes cast
  *   6.1 asserting that voting locks are preserved
  *   6.2 asserting that voting funds are returned
+ * 7. refunding the submission and decision deposits
  */
 export async function referendumLifecycleTest<
   TCustom extends Record<string, unknown> | undefined,
@@ -124,7 +125,7 @@ export async function referendumLifecycleTest<
    * Submit a new referendum
    */
 
-  const submitReferendumTx = relayClient.api.tx.referenda.submit(
+  const submissionTx = relayClient.api.tx.referenda.submit(
     {
       Origins: 'SmallTipper',
     } as any,
@@ -135,13 +136,13 @@ export async function referendumLifecycleTest<
       After: 1,
     },
   )
-  const submitReferendumEvents = await sendTransaction(submitReferendumTx.signAsync(defaultAccounts.alice))
+  const submissionEvents = await sendTransaction(submissionTx.signAsync(defaultAccounts.alice))
 
   await relayClient.dev.newBlock()
 
   // Fields to be removed, check comment below.
   let unwantedFields = new RegExp('index')
-  await checkEvents(submitReferendumEvents, 'referenda')
+  await checkEvents(submissionEvents, 'referenda')
     .redact({ removeKeys: unwantedFields })
     .toMatchSnapshot('referendum submission events')
 
@@ -509,11 +510,17 @@ export async function referendumLifecycleTest<
   // 2. its decision period, still counting down.
   referendumCmp(ongoingRefSecondVote, ongoingRefThirdVote, ['tally', 'alarm'])
 
-  /**
-   * Cancel the referendum using the scheduler pallet to simulate a root origin
-   */
-
+  // Attempt to cancel the referendum with a signed origin - this should fail.
+
   const cancelRefCall = relayClient.api.tx.referenda.cancel(referendumIndex)
+  const cancelRefEvents = await sendTransaction(cancelRefCall.signAsync(defaultAccounts.alice))
+
+  await relayClient.dev.newBlock()
+
+  await checkEvents(cancelRefEvents, 'referenda', 'system')
+    .toMatchSnapshot('cancelling referendum with signed origin')
+
+  // Cancel the referendum using the scheduler pallet to simulate a root origin
 
   const number = (await relayClient.api.rpc.chain.getHeader()).number.toNumber()
 
@@ -543,6 +550,26 @@ export async function referendumLifecycleTest<
    * Check cancelled ref's data
    */
 
+  // First, the emitted events
+  // Retrieve the events for the latest block
+  const events = await relayClient.api.query.system.events()
+
+  const referendaEvents = events.filter((record) => {
+    const { event } = record;
+    return event.section === 'referenda'
+  });
+
+  assert(referendaEvents.length === 1, "cancelling a referendum should emit 1 event")
+
+  const cancellationEvent = referendaEvents[0]
+  assert(relayClient.api.events.referenda.Cancelled.is(cancellationEvent.event))
+
+  const [index, tally] = cancellationEvent.event.data
+  assert(index.eq(referendumIndex))
+  assert(tally.eq(votes))
+
+  // Now, check the referendum's data, post-cancellation
+
   referendumDataOpt = await relayClient.api.query.referenda.referendumInfoFor(referendumIndex)
   // cancelling a referendum does not remove it from storage
   assert(referendumDataOpt.isSome, "referendum's data cannot be `None`")
@@ -659,6 +686,23 @@ export async function referendumLifecycleTest<
     await check(castVotes).toMatchSnapshot(`${account}'s votes after rescission`)
     assert(castVotes.votes.isEmpty)
   }
+
+  // Check that submission and decision deposits are refunded to the respective voters.
+
+  const submissionRefundTx = relayClient.api.tx.referenda.refundSubmissionDeposit(referendumIndex)
+  const submissionRefundEvents = await sendTransaction(submissionRefundTx.signAsync(defaultAccounts.alice))
+  const decisionRefundTx = relayClient.api.tx.referenda.refundDecisionDeposit(referendumIndex)
+  const decisionRefundEvents = await sendTransaction(decisionRefundTx.signAsync(defaultAccounts.bob))
+
+  await relayClient.dev.newBlock()
+
+  await checkEvents(submissionRefundEvents, 'referenda')
+    .redact({ removeKeys: new RegExp('index') })
+    .toMatchSnapshot('refund of submission deposit')
+
+  await checkEvents(decisionRefundEvents, 'referenda')
+    .redact({ removeKeys: new RegExp('index') })
+    .toMatchSnapshot('refund of decision deposit')
 }
 
 /**
@@ -727,12 +771,20 @@ export async function referendumLifecycleKillTest<
 
   await relayClient.dev.newBlock()
 
+  // Attempt to kill the referendum with a signed origin
+
+  const killRefCall = relayClient.api.tx.referenda.kill(referendumIndex)
+  const killRefEvents = await sendTransaction(killRefCall.signAsync(defaultAccounts.alice))
+
+  await relayClient.dev.newBlock()
+
+  await checkEvents(killRefEvents, 'referenda', 'system').toMatchSnapshot('killing referendum with signed origin')
+
+
   /**
    * Kill the referendum using the scheduler pallet to simulate a root origin for the call.
    */
 
-  const killRefCall = relayClient.api.tx.referenda.kill(referendumIndex)
-
   const number = (await relayClient.api.rpc.chain.getHeader()).number.toNumber()
 
   await relayClient.dev.setStorage({