This project parses and compares Substrate Weight files. It helps tremendously with the review process of large Weight diffs. The results can be displayed conveniently in the CLI or browser.
Manually comparing Substrate Weight files is a task where humans falter and computers excel.
When you want to know what I am talking about; take a look at this diff.
Now tell me which lines are problematic and which are fine ๐? Does not look appealing, does it?
This is where SWC comes to the rescue: It takes the old and the new version and compares all extrinsics.
The human-friendly output looks like this:
This automatically sorts the worst offenders to the top and allows humans to see at a glance what is going on โ without loosing details! Links to the diff are all there, and sharing results is as easy as copying a link to the line. Compatible with most Substrate chains. Enjoy!
You can play around with the following public endpoints that are exposed by swc-web:
- Compare Merge Requests
- Compare Commits
The dev branch is deployed at https://weights.tasty.limo:8443/.
Install both binaries:
```sh cargo install --git https://github.com/ggwpez/substrate-weight-compare swc swc_web
swc --version swc-web --version ```
The rust-toolchain.toml defines the exact Rust version that the code was tested with.
The formatting rules are defined in rustfmt.toml.
```sh git clone https://github.com/ggwpez/substrate-weight-compare
cd substrate-weight-compare/ cargo build --profile production ```
Assuming you have a Substrate compatible repository checked out in the parent directory:
sh
swc-web --root ../ --repos polkadot substrate cumulus
then open your browser and try the following: - http://localhost:8080/
Suppose you have some weight files in:
- OLD=repos/polkadot/ and
- NEW=my_other_repos/polkadot
The base command looks like this:
sh
swc compare files --old $OLD/* --new $NEW/* --method worst
If you want to compare the weights of the Kusama to the Polkadot runtime, the command becomes a bit more longer:
sh
swc compare files --old ../polkadot/runtime/kusama/**/weights/*.rs --new ../polkadot/runtime/polkadot/*/weights/*.rs --method worst --ignore-errors --change changed unchanged --unit time --threshold 10
sh
+-----------------------------------------+-----------------------------+----------+----------+---------------+
| File | Extrinsic | Old | New | Change [%] |
+=============================================================================================================+
| pallet_election_provider_multi_phase.rs | feasibility_check | 1.23ms | 812.80us | -33.90 |
|-----------------------------------------+-----------------------------+----------+----------+---------------|
| frame_benchmarking_baseline.rs | addition | 162.00ns | 112.00ns | -30.86 |
|-----------------------------------------+-----------------------------+----------+----------+---------------|
| runtime_parachains_hrmp.rs | hrmp_cancel_open_request | 27.90us | 39.02us | +39.86 |
|-----------------------------------------+-----------------------------+----------+----------+---------------|
| pallet_tips.rs | slash_tip | 15.86us | 22.61us | +42.56 |
|-----------------------------------------+-----------------------------+----------+----------+---------------|
| runtime_parachains_initializer.rs | force_approve | 3.12us | 4.53us | +45.02 |
|-----------------------------------------+-----------------------------+----------+----------+---------------|
| runtime_parachains_hrmp.rs | clean_open_channel_requests | 366.82us | 590.73us | +61.04 |
|-----------------------------------------+-----------------------------+----------+----------+---------------|
| runtime_parachains_hrmp.rs | hrmp_accept_open_channel | 29.81us | 48.54us | +62.81 |
|-----------------------------------------+-----------------------------+----------+----------+---------------|
| runtime_parachains_hrmp.rs | hrmp_close_channel | 27.58us | 44.92us | +62.89 |
|-----------------------------------------+-----------------------------+----------+----------+---------------|
| runtime_parachains_hrmp.rs | force_process_hrmp_open | 2.17ms | 3.64ms | +67.28 |
|-----------------------------------------+-----------------------------+----------+----------+---------------|
| runtime_parachains_hrmp.rs | hrmp_init_open_channel | 32.67us | 55.70us | +70.49 |
|-----------------------------------------+-----------------------------+----------+----------+---------------|
| runtime_parachains_hrmp.rs | force_process_hrmp_close | 1.21ms | 2.11ms | +74.25 |
|-----------------------------------------+-----------------------------+----------+----------+---------------|
| runtime_parachains_hrmp.rs | force_clean_hrmp | 1.82ms | 3.27ms | +80.04 |
+-----------------------------------------+-----------------------------+----------+----------+---------------+
Cou can use the --print-terms flag to print the terms. This example omits them since the rows get really long.
Compare arbitrary Polkadot commits assuming that you have checked the repo out:
```sh FROM=20467ccea1ae TO=ef922a7110eb THRESHOLD=30
swc compare commits $FROM $TO --threshold $THRESHOLD --repo ../polkadot
palletscheduler.rs::oninitializenamedaborted 4957 -> 3406 ns (-31.29 %) palletelectionprovidermultiphase.rs::finalizesignedphaserejectsolution 33389 -> 19348 ns (-42.05 %) palletelectionprovidermultiphase.rs::submit 77368 -> 42754 ns (-44.74 %) palletelectionprovidermultiphase.rs::oninitializenothing 23878 -> 12324 ns (-48.39 %) palletelectionprovidermultiphase.rs::finalizesignedphaseacceptsolution 50596 -> 25888 ns (-48.83 %) palletscheduler.rs::oninitializeresolved 3886 -> 1701 ns (-56.23 %) palletelectionprovidermultiphase.rs::oninitializeopenunsigned 33568 -> 12320 ns (-63.30 %) palletelectionprovidermultiphase.rs::oninitializeopensigned 34547 -> 12500 ns (-63.82 %) palletelectionprovidermultiphase.rs::createsnapshotinternal 8835233 -> 47360 ns (-99.46 %) palletscheduler.rs::oninitializeperiodicresolved 33 -> 0 ns (-100.00 %) framebenchmarkingbaseline.rs::subtraction 100 -> 131 ns (+31.00 %) framebenchmarkingbaseline.rs::addition 96 -> 126 ns (+31.25 %) palletscheduler.rs::oninitializeperiodic 5139 -> 7913 ns (+53.98 %) runtimecommoncrowdloan.rs::on_initialize 0 -> 4293 ns (+100.00 %) ``` It prints first the ones that decreased (good) and then the ones that increased (bad) sorted by ascending absolute value.
Selects the project to use. SWC has the goal of being compatible with: - [Substrate] - [Polkadot] - [Cumulus]
Other projects which are currently compatible, but not a hard requirement: - [Moonbeam] - [Composable Finance]
Note: Not all repositories are deployed to the SWC web service.
Uses the glob crate to match files in the repository path with the given pattern.
Here are some examples that the web interface uses. These do not catch all files, which is a bug:
- Substrate: frame/*/src/weights.rs
- Polkadot: runtime/*/src/weights/**/*.rs,bridges/modules/*/src/weights.rs
- Cumulus: **/weights/*.rs,**/weights/xcm/*.rs,**/src/weights.rs
weights/**/*.rs is preferred to weights/*.rs to include possible sub-folders like XCM.
The mod.rs file is automatically excluded.
Filter by the pallets to include by using a [Regex].
Examples:
- .* would be any pallet.
- system|assets would be the system and the assets pallet.
Analogous to the Pallet filter this filters by the extrinsics using a [Regex].
Examples:
- .* would be any extrinsic.
- mint|burn would be the mint and the burn extrinsics.
The evaluation method defines how the weight equation is evaluate (=calculated).
This is a deciding factor when making a decision whether or not a weight got worse.
n components is 2^n which is hard-limited to 16 components. "The range of component" in your weight.rs files.NOTE: The storage weights are currently set to RocksDB Substrate default.
Filters the changes results by an absolute percentual threshold.
The percentages values are calculated as increase or decrease.
Eg: from 100 to 150 would be +50% and would be included by any threshold >=50.
Filters the changes results by an absolute threshold.
The weight in Substrate is chromatic (two dimensional). Its dimensions are Reference Time and PoV size. The dimension can therefore be set to either Time or Proof. A good unit will then automatically be selected, for example ยตs for Time or KiB for Proof; depending on the size of the concrete scalars.
- Time: The execution time that the call consumed on reference hardware.
- Proof: The size of the Proof-of-validity (PoV) that the call produced.
The relevant MR is substrate#11637 which requires integration the weight template for your project to emit chromatic weights.
Silently ignore parse errors. This is useful when using inclusive path patterns.
Pull the branch before comparing anything. This ensures that you are on the last commit.
This does not override the Cache. It can therefore take up to 10 minutes for a new change to show up.
The web UI caches success responses for 10 minutes. Currently there is no flag to disable it.
Use commit hashes instead of tags and branches if you need uncached results.
There exist unit and integration tests. Most of them are guarded behind feature flags. The explanation below covers both in one.
Integration tests do currently not exist for all supported repos (see [integration.rs]).
```sh git clone https://github.com/ggwpez/substrate-weight-compare cd substrate-weight-compare
mkdir -p repos git clone https://github.com/paritytech/polkadot/ repos/polkadot git clone https://github.com/paritytech/substrate/ repos/substrate git clone https://github.com/PureStake/moonbeam/ repos/moonbeam git clone https://github.com/ComposableFi/composable/ repos/composable
cargo test --release --all-targets --all-features ```