Generate weights for runtime #2442

nbaztec · 2023-08-18T13:54:15Z

What does it do?

Generates weights for the runtime. Uses the ./scripts/run-benches-for-runtime.sh adapated from polkadot's. Each pallet has its own weight file that contains a generated trait WeightInfo as per the included template. Upon running the run-benches-for-runtime.sh the default template is used which merely implements the WeightInfo from the respective pallet's own weights.rs module, with the values for the current runtime.

The current PR uses weights generated by the benchmarking server, so is production ready.

⚠️ Breaking Changes ⚠️

Reference hardware for collators

Benchmarks introduced in this PR were run over a c6i.4xlarge AWS instance. It is recommended that the hardware used to run a collator matches the following specs in order to ensure the proper block production in time, as well as to prevent other performance issues.

CPU

x86-64 compatible.
Intel Ice Lake, or newer (Xeon or Core series); AMD Zen3, or newer (EPYC or Ryzen).
4 physical cores @ 3.4GHz.
Simultaneous multithreading disabled (Hyper-Threading on Intel, SMT on AMD)
Prefer single-threaded performance over higher cores count. A comparison of single-threaded performance can be found here.

Storage

An NVMe SSD of 1 TB (As it should be reasonably sized to deal with blockchain growth). In general, the latency is more important than the throughput.

Memory

32 GB DDR4 ECC.

System

Linux Kernel 5.16 or newer.

Network

The minimum symmetric networking speed is set to 500 Mbit/s (= 62.5 MB/s).

What important points reviewers should know?

Currently all weights are generated into runtime-common, however that can be easily changed if needed.
moonbeam-xcm-benchmarks diverges on how polkadot uses the generated weights - namely this runtime/weights/mod.rs file implements what we're currently doing in the moonbeam-xcm-benchmarks pallet, this ends up causing a circular dependency on runtime-common so the xcm weights are currently left untouched. All other pallets are migrated.

Is there something left for follow-up PRs?

Figure out how to use XCM weights to be used from runtime-common isntead.

Are there relevant PRs or issues in other repositories (Substrate, Polkadot, Frontier, Cumulus)?

Fix benchmarks in frontier: polkadot-evm/frontier#1162
Currently we're using a patched substrate on our fork moonbeam-foundation/substrate#67 since the ED = 0 constraint caused these benchmarks to fail. We should probably copy these benchmarks in moonbeam and maintain them going forward.
Other repos that were fixed:
Moonsong-Labs/moonkit#4
moonbeam-foundation/crowdloan-rewards#69

github-actions · 2023-08-18T21:53:34Z

Coverage generated "Mon Aug 28 18:35:26 UTC 2023":
https://d3ifz9vhxc2wtb.cloudfront.net/pulls/2442/html/index.html

Master coverage: 87.39%
Pull coverage:

notlesh · 2023-08-28T19:41:17Z

pallets/moonbeam-xcm-benchmarks/src/weights/generic.rs

+		// Minimum execution time: 18_446_744_073_709_551_000 picoseconds.
+		Weight::from_parts(500_000_000_000, 0)


I'm guessing 500_000_000_000 was hard coded, but why the huge Minimum execution time number?

notlesh · 2023-08-28T19:44:52Z

pallets/parachain-staking/src/weights.rs

-		Weight::from_parts(70_805_524, 26879)
-			// Standard Error: 8_036
-			.saturating_add(Weight::from_parts(84_366, 0).saturating_mul(x.into()))
-			.saturating_add(T::DbWeight::get().reads(6_u64))
+		//  Measured:  `1421 + x * (38 ±0)`
+		//  Estimated: `4752 + x * (41 ±0)`
+		// Minimum execution time: 75_412_000 picoseconds.
+		Weight::from_parts(85_543_270, 4752)
+			// Standard Error: 2_633
+			.saturating_add(Weight::from_parts(213_586, 0).saturating_mul(x.into()))
+			.saturating_add(T::DbWeight::get().reads(7_u64))


This is a pretty big increase, is it expected?

Oh, did these staking ones change because they're generated by a proper runtime instead of a mock runtime?

...actually, why is staking the only one where we touched the pallet-level weights? They went up for the most part, which is probably "safe." It's used by other projects which may not generate their own weights, so maybe that's a good thing 🤷

The weights of some pallets were relying on substrate part that were not correctly configured.
I think the goal is to get default values for external team, but still generate weights for our runtimes once ready

So yes we want to provide weight by default on pallets, and then a weight for our runtimes

notlesh

LGTM. I spot-checked a few of the results and they look consistently lower, which is to be expected with improved hardware.

The exception is parachain-staking, I don't have an opinion on whether we want those changes or not, so maybe just ensure it's intentional.

crystalin · 2023-08-29T08:31:39Z

parachain-staking is the one of the reasons we changed those weights, like auto-compound (which also will have a fix separately).
We also changed things relying on the scheduler and other elements which are configured by default to be very high in substrate

crystalin · 2023-08-29T08:36:49Z

Do we have a tool that would display all the weight changes per function ?

nbaztec added 7 commits August 17, 2023 13:05

use generated benchmarks

ed3f6c8

use locally generated weights

86249f1

use generated weights

331527f

use runtime weights

1d9a74c

Merge branch 'master' into nish-regen-all-weights

1fd81cd

use local benchmarked weight

3bca61b

fix xcm benchmarks

feab97b

nbaztec added B7-runtimenoteworthy Changes should be noted in any runtime-upgrade release notes D9-needsaudit👮 PR contains changes to fund-managing logic that should be properly reviewed and externally audited breaking Needs to be mentioned in breaking changes labels Aug 18, 2023

nbaztec added 10 commits August 18, 2023 17:29

update substrate

74e22ec

use benchmark server computed weights

a0cda5b

fmt

6588e14

add BenchmarkHelper to pallet assets mock

2d6e355

fix tests

b9ae338

fix mocks

5ea8d45

fix ts test

3186412

fix rust test

7a8ebaa

fix test

5ebac9a

fix test

efbd614

Agusrodri added 2 commits August 24, 2023 09:37

Merge branch 'master' into nish-regen-all-weights

60b9e7e

fix TARGET_FILL_AMOUNT const

f6ee6ca

librelois mentioned this pull request Aug 28, 2023

runtime-2500 release #2453

Open

28 tasks

Merge branch 'master' into nish-regen-all-weights

c9ee59a

notlesh reviewed Aug 28, 2023

View reviewed changes

notlesh approved these changes Aug 28, 2023

View reviewed changes

nbaztec merged commit d0ae4a5 into master Aug 29, 2023

nbaztec deleted the nish-regen-all-weights branch August 29, 2023 08:44

noandrea changed the title ~~[MOON-2531] Generate weights for runtime~~ Generate weights for runtime Aug 30, 2023

crystalin mentioned this pull request Sep 1, 2023

client v0.33.0 release #2468

Closed

10 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Generate weights for runtime #2442

Generate weights for runtime #2442

nbaztec commented Aug 18, 2023 •

edited by Agusrodri

Loading

github-actions bot commented Aug 18, 2023 •

edited

Loading

notlesh Aug 28, 2023

notlesh Aug 28, 2023

notlesh Aug 28, 2023

notlesh Aug 28, 2023

crystalin Aug 29, 2023

crystalin Aug 29, 2023

notlesh left a comment

crystalin commented Aug 29, 2023

crystalin commented Aug 29, 2023

		// Minimum execution time: 18_446_744_073_709_551_000 picoseconds.
		Weight::from_parts(500_000_000_000, 0)

Generate weights for runtime #2442

Generate weights for runtime #2442

Conversation

nbaztec commented Aug 18, 2023 • edited by Agusrodri Loading

What does it do?

⚠️ Breaking Changes ⚠️

Reference hardware for collators

CPU

Storage

Memory

System

Network

What important points reviewers should know?

Is there something left for follow-up PRs?

Are there relevant PRs or issues in other repositories (Substrate, Polkadot, Frontier, Cumulus)?

github-actions bot commented Aug 18, 2023 • edited Loading

notlesh Aug 28, 2023

Choose a reason for hiding this comment

notlesh Aug 28, 2023

Choose a reason for hiding this comment

notlesh Aug 28, 2023

Choose a reason for hiding this comment

notlesh Aug 28, 2023

Choose a reason for hiding this comment

crystalin Aug 29, 2023

Choose a reason for hiding this comment

crystalin Aug 29, 2023

Choose a reason for hiding this comment

notlesh left a comment

Choose a reason for hiding this comment

crystalin commented Aug 29, 2023

crystalin commented Aug 29, 2023

nbaztec commented Aug 18, 2023 •

edited by Agusrodri

Loading

github-actions bot commented Aug 18, 2023 •

edited

Loading