Same instruction count, 23x the wall clock: working-set effects in a deterministic RISC-V interpreter

I’ve been measuring execution cost in a deterministic RISC-V interpreter, motivated by a narrow problem: assigning a protocol-level execution budget that stays safe across heterogeneous machines.

The budget is denominated in steps, and the question I set out to answer is how tightly a step count bounds wall clock. The answer I got is looser than I expected, so I’m posting the measurements rather than a proposal.

Terminology, because this is the part that will otherwise be misread.

A step is one RV32IM instruction executed by the interpreter loop. It is not a host CPU instruction.

Every number below is interpreter steps per second, so the chain being measured is:

RISC-V instruction → interpreter dispatch → host CPU work

not:

RISC-V instruction → one hardware instruction

This is about native interpreter execution cost, not zk proving cost. The per-opcode work I cite below is about proving; I’m asking whether an analogous distinction shows up when execution itself is the priced resource.

Provenance

Two machines, both --release:

  • Reference: Motorola Edge 40 Neo — MediaTek MT6879V/ZA, 8 cores in two clusters (2 at 2.5 GHz, 6 at 2.0 GHz), 8 GB RAM, Android 15, rustc 1.97.1.

    A relatively constrained reference machine; the budget is calibrated against it. Core placement was not pinned, so a run may land on either cluster.

  • Second: Intel Core i5-9400, 6 cores / 6 threads, 2.90 GHz base, 1.5 MB total L2, 9 MB L3, Windows 11, rustc 1.96.0.

The VM’s own page accounting uses a 4 KiB page, a constant of the design, which is not necessarily the host page size.

Benchmark at commit 94dc7e3.

Sampling differs between the two binaries, and it matters for how you read them.

mezclas runs each mix three times internally and reports the worst — a simple conservative rule, not a claim of a statistical worst case.

conjunto does not: each point in the working-set sweep is a single pass, which is worth knowing before reading the run-to-run spread further down.

I did not instrument CPU frequency, thermal state, or background load.

Instruction mixes at a fixed step count

Seven mixes, each an infinite loop that halts at an exact step count.

The reference load is ML-DSA-44 signature verification, which is the workload this particular budget was designed to price.

Mix M steps/s vs ML-DSA-44
mul 326.6 1.23
random arithmetic (8 opcodes) 325.4 1.22
addi uniform 322.7 1.21
ML-DSA-44 (reference load) 266.2 1.00
divu 197.7 0.74
lw sequential (2 KiB, fits L1) 186.3 0.70
lw pointer-chase (96 pages) 82.1 0.31
lw pointer-chase, page ceiling raised 11.3 0.04

At the same 7,000,000-step ceiling, the pointer-chasing mix is about 23x slower than the ML-DSA-44 mix on this machine: 26 ms against 619 ms.

This is not a correctness difference. Both executions remain deterministic and halt at the same step count; nothing about the slow one is invalid.

Reproducibility note on that last row.

The shipped harness does not report 11.3. Its 96 distinct-page ceiling terminates that case first, so mezclas prints cortada there rather than a rate.

I raised the ceiling only for that one measurement (mezclas.rs:269, m.techo_paginas, default u32::MAX).

Every other number in the table comes from the unmodified harness.

Why opcode identity is not enough

Compare rows 6 and 7.

Sequential lw runs at 186.3, pointer-chasing lw at 82.1, and the opcode stream is identical at the lw level.

A static per-opcode weight cannot distinguish these two executions, because what differs is the memory-access behaviour, not the instruction identity.

A model that wanted to stay safe would need information beyond the opcode itself.

To be explicit about the size of that claim:

This is not evidence that static metering is impossible.

It is evidence that opcode identity alone does not capture all of the execution-cost variance I observed.

The most careful per-opcode work I know of is Measuring Per-Opcode Proving Time, which isolates each operation’s marginal contribution precisely so cheap operations become measurable; that question is posed as holding all else equal, and rows 6 and 7 are the case where “all else” is what moves.

The assumption is stated most plainly in eWASM’s metering design, which sums a per-opcode cost table and says the total “stands as an estimation for computational time.”

A second axis, which surprised me more than the first

Memory is not the only place the two machines disagree.

Compare a single-opcode loop against eight opcodes in unpredictable order — same step count, no memory traffic in either:

Mix Phone (aarch64) i5-9400
addi uniform 322.7 413.0
random arithmetic (8 opcodes) 325.4 156.1
throughput ratio, addi ÷ random 0.99x 2.65x

On the phone the two mixes are effectively tied.

On the desktop addi runs 2.65x the random mix — the same within-machine comparison, moving from roughly 1 to 2.65 between the two machines.

I want to be careful about the mechanism: the two mixes differ in opcode composition as well as in how predictable the order is, so this does not isolate interpreter dispatch prediction as the cause.

What it does show is that the same pair of mixes ranks differently on the two machines, with no memory traffic involved in either — a second place where relative cost is not a property of the program.

Both figures are worst-of-three, from one invocation of mezclas per machine; I would want more machines before leaning on it.

Working set

What is countable at runtime is the number of distinct pages touched.

It is observable while executing and deterministic, and it exposes variance that instruction counting cannot see.

That is the whole of what I am claiming for it — it is not a cost function, and the sweep is also where it stops being one:

Working set 4 KiB pages M steps/s
16 KiB 4 163.6
64 KiB 16 125.5
128 KiB 32 101.1
192 KiB 48 86.2
384 KiB 96 80.8
1 MiB 256 79.3
2 MiB 512 77.6
4 MiB 1024 10.9

On this machine the curve becomes relatively flat between 384 KiB and 2 MiB, followed by a sharp drop at 4 MiB — a 7x fall for a 2x working set.

A budget expressed only in steps cannot express any of this.

But the same shape cuts against the axis too: across the flat region, charging by pages touched would over-charge, and the one place cost really moves is a discontinuity the page count does not locate.

The axis exposes the variance; it does not price it.

The drop is not at a fixed size.

On the i5-9400 the same sweep is still above 50 M steps/s at 4 MiB and only collapses at 16 MiB (15.6).

That machine has 9 MB of L3: 4 MiB fits, 16 MiB does not, and the collapse lands between them.

That is consistent with the host memory hierarchy driving the cliff rather than it being an intrinsic property of the program — consistent with, not proof of.

The part I did not expect

I assumed the weakest hardware would be the worst case, since that is what a budget has to hold on.

Across these two machines that is not what I measured once the working set grows:

Working set Phone (aarch64) i5-9400, run A i5-9400, run B
192 KiB 86.2 122.2 138.3
384 KiB 80.8 78.9 88.3
2 MiB 77.6 40.6 55.5

At a small working set the phone is slower, as expected.

At 2 MiB the desktop is slower in both runs.

I am not treating this as evidence that desktop hardware is generally slower — it is evidence that which machine is the worst case is not obvious from a two-machine sample once the working set is large.

I am also not giving a crossover point, because I cannot pin one: at 384 KiB the two runs of the same desktop land on opposite sides of the phone.

Run B was taken on a machine with an active interactive session; the phone’s spread over the same measurement was 1.6%, the desktop’s was 40 to 88 M steps/s.

The run-to-run variance is worth investigating on its own, because it means a single calibration run is not necessarily stable — but I have not established its cause, and CPU frequency, turbo, thermal state, scheduling and background load are all uncontrolled here.

Run it

No dependencies beyond the Rust toolchain.

git clone https://github.com/cristiandkzk/deterministic-succession
cd deterministic-succession/genesis/predicado/vm

cargo run --release --bin mezclas  # the instruction-mix table
cargo run --release --bin conjunto  # the working-set sweep

If you run either benchmark, please post the output together with the CPU model and architecture you ran it on.

OS and RAM are useful as well.

I’m particularly interested in cases where the relative ordering changes across machines, rather than in absolute throughput — that is what the two sections above turn on, and it is the part a second machine can falsify cheaply.

Two machines are not enough to establish a worst-case hardware baseline.

The goal is to collect measurements across different CPUs and see whether a stable relationship emerges.

If your results do not reproduce the desktop-is-slower result, that is especially useful.

What this does not show

It does not show that opcode metering is useless, that RISC-V is unsuitable as an L1 execution target, that WASM would be better or worse, or that a 23x gap is exploitable on Ethereum today.

It shows something narrower:

In this deterministic interpreter, equal step counts can correspond to materially different execution times, and the difference depends on the instruction mix, on runtime memory behaviour, and on the host.

Deterministic execution buys deterministic semantics and a deterministic step count.

On this evidence it does not by itself buy a tight, hardware-independent bound on physical execution cost.

Questions

  1. Has anyone explored charging for pages actually touched rather than pages allocated or grown, specifically for execution-cost accounting?

    The closest prior art I found is eWASM’s memory metering, which charges for the initial page count plus every grow_memory — the allocated axis.

    My mixes allocate the same memory and differ 2.3x on how they walk it.

    I am especially interested in work that treats this as a protocol resource rather than as a performance-analysis metric.

  2. In the per-opcode thread above, @jochem-brouwer asks how one would find worst-case inputs per client, having done it by hand for ModExp.

    If the result above holds up, part of the answer is uncomfortable: worst-case wall clock is not determined by the program/input pair alone — it also depends on host execution state and microarchitectural behaviour.

    Does that match what the repricing benchmarks have been showing?

  3. The RISC-V vs WASM delivery-ISA thread argues the choice substantially on how cleanly metering can be injected.

    Structured control flow constrains where a program may jump; a load’s address is still computed, and WASM’s bound on linear memory is the declared maximum page count, which is the allocated axis again.

    So: could an adversarial program reproduce a comparable execution-cost gap under either delivery ISA, given the respective memory and execution constraints?

  4. Underneath all three: what is the right protocol-level resource to meter when wall-clock cost depends on both computation and runtime memory locality?

Follow-up: identical work, different wall-clock time — now across cryptographic primitives

A follow-up to my previous post: the same “identical work, different wall clock” problem shows up again, in a place I didn’t expect.

In the original post, I measured this within a single primitive: two lw sequences with identical opcodes and identical step counts differed by 2.3× in wall-clock time, depending on the memory access pattern (sequential vs. pointer-chasing). The interpreter’s step counter cannot observe cache behavior.

I just measured the same class of surprise between cryptographic primitives, motivated by a different question: if a chain migrates its signing scheme, could it cheaply run a composite signature scheme using two independent cryptographic families, so that breaking one does not automatically compromise both?

I benchmarked:

  • ML-DSA-44 — lattice-based, Module-LWE
  • SLH-DSA-128s / SPHINCS+ — hash-based, SHA-2 cascade

For both, I measured decode + verify using the same WASM module across three execution modes:

  • native
  • wasmi (pure interpreter)
  • wasmtime / Cranelift (JIT)

The engine ranking inverts

The relative overhead is surprisingly different between the two primitives:

Primitive Interpreter / native JIT / native
ML-DSA-44 28.3× 3.8×
SLH-DSA-128s 8.0× 0.9×

The last number is particularly interesting: in one run, the JIT version of SLH-DSA-128s was actually slightly faster than its native build.

My current hypothesis is that the JIT can compile a long cascade of relatively regular SHA-2 rounds about as effectively as the native compiler, while an interpreter’s per-instruction dispatch overhead is less problematic when the workload is dominated by predictable computation rather than scattered memory access.

The lattice workload appears to be much more sensitive to the execution engine and its interaction with memory — broadly consistent with the cache-line effects I saw in the previous experiment.

Why this matters for the original question

A naive extrapolation from the native benchmarks suggests that the hash-based primitive costs roughly 10× the lattice-based one.

But under the JIT engine that actually determines the phone step-budget in my earlier experiment, the ratio drops to 9.3×.

More importantly, the cost of the composite verification — verifying both signatures, which is the actual cost a chain would pay — comes out at approximately:

4.2× one ML-DSA-44 verification alone.

So the relevant numbers are not simply “10×” or “30×”. The execution engine changes the relative cost substantially, and the cost of running both primitives together is much smaller than a naive multiplication of the individual interpreter penalties might suggest.

Caveat

These measurements are desktop x86-64 only.

The original experiment already showed that phone and desktop can cross different working-set thresholds, so I would not assume that these ratios remain unchanged on ARM. I don’t have a phone run yet.

Data and code

I’m particularly interested in whether others have seen this kind of JIT/interpreter inversion between SHA-2-heavy and lattice-heavy workloads in constrained WASM or other deterministic VMs.

Is this a reasonably general property of these workload classes, or is it mostly an artifact of the particular implementations and execution engines I measured?