Our of our engineers recently measured bare-bones EVM performance both for JIT and not JIT for transactions that did not include state changes (so this is essentially performance of bytecode interpretation).
It was 20,000 TPS without JIT and 50,000 TPS JIT for simple transaction (Fibonnachi number calculation)
We are currently measuring TPS for transactions that involve state transitions. It looks that state transitions take much longer than math for typical transactions.
What are the number in MIPS (Millions of Instructions Per Seconds), i.e. opcodes interpreted per second? And what is the CPU frequency and generation (Haswell, Skylake, …).
I would expect a big hit of EVM speed compared to naive VM due to the use of uint256 by default, especially for Go as those require heap allocation (or a memory pool).
Here is a wiki where I compiled several resources on state-of-the-art VM optimization. It also includes a naive VM with 7 instructions benchmark that can serve as a baseline to compare EVM against. And can be used to compare language raw speed on the same machine as well.
In practice, these speedups will only be relevant to “number-crunching” contracts because the computation time is otherwise largely dominated by storage access.
Thanks for the Wiki @mratsim, will help me catch up on the literature. Most interesting point I saw so far is that on recent Intel chips simple switch dispatch is much faster than it used to be.
Can I ask which EVM you were measuring, @kladkogex? They vary widely in their performance, as I reported at Devcon3. The numbers you give look about right for the C++ interpreter versus the C++ JIT.
It’s true that the 256-bit registers hurt performance a lot. A compiler could optimize this away in many cases, but that doesn’t change the gas cost. Switching to a gas model like IELE’s might help that – they charge more gas for an operation as the size of the operands increases. eWasm solves the gas problem by not supporting wide registers.
However we solve the gas problem, JITs can’t help us. They are too big of an attack surface, as the eWasm team reported recently on an AllCoreDevs call. Bytecode will need to be translated to machine code at deployment time with DoS-hardened compilers.
And indeed state access currently accounts for more time than contract execution, but that is in part because contract execution is so expensive that people don’t write computationally intensive contracts. Including us–we write new pre-compiles instead.
The security issue here is denial of service. Unless designed to prevented it compilers are subject to quadratic blowup. Fuzz testing of V8 by the eWasm team found that most contracts could be compiled in 100ms, but some “compiler bombs” could take 2 secs. So an attacker could use such bombs to create DoS attacks. (There are other vulnerabilities, such as caches of compiled code that can be defeated.)
One could, but the wasm team doesn’t see that as an answer, since some compilers might choke and others not on the same input. Rather, we’ll need compilers that never take more than about n*log(n) time or space.
BTW - does a (possible) move to EWASM look like a bad thing to you from the point of security? It seems that currently EVM is simple and moving to EWASM introduces lots of things. Just based on a generic complexity lots of things mean lots of potent
ial security holes.
Another question is whats the point for Ethereum to move to EWASM at all? The network for the PoW chain is so slow at 15 transactions per second, that it seems that the best for the PoW chain is to stay with EVM as is. Whats the driver behind the move to EWASM?
The desire for a faster VM is to stop the need for writing precompiled contracts. They only exist because the EVM isn’t fast enough, and/or charges too much gas. Secondarily, to let users do similar expensive crypto stuff that we didn’t put into a precompile for them.
Another reason for eWasm is the desire to reuse other work, so far as languages, compilers, VMs, IDEs etc.
The eWasm subset (not full-on Wasm) needn’t be a security issue, but it needs DoS-hardened compilers, just as a faster EVM would. Even at 15 TPS we’ve had at least two DoS attacks on the EVM: one due to mispricing EXP and another due to geth’s JIT going quadratic.
It’s looking likely that eWasm will continue as part of the shasper work, and I will restart my EIP-615 and EIP-616 work to get the EVM more formally tractable and performant on the main chain, with transpiling from EVM to eWasm providing a bridge.
Much of the performance of VMs rely on efficient branch predictor, especially after improvement Haswell even on switch-based dispatch. Disabling speculative execution might make the EVM much slower.
The Phoronix benchmarks show horror stories of more than 20% performance lost on some benchmarks.
From what I’ve read it’s not at at all clear to me what impact the various patches will have on the EVM, or on the client as a whole The biggest losers on the benchmarks look to be things like process scheduling.
How does this balance with the fact that it will nuke our existing selection of tools, even though they are still young? Existing languages, compilers, etc. would also have to be adopted to the unique challenges of programming consensus-critical, immutable code in a different model than most general purpose languages leverage. How many of these languages have primitives for account-based transactional programming and atomic commitments?
I think it’s a fair assessment from a network liveness standpoint, but I interpret “security” in an immutable programming framework like ours as the ability to write this code well prior to deployment, as to avoid (as much as possible) any dangerous and unanticipated state changes that may alter ownership of funds and assets built into Ethereum. How does this new paradigm affect security from that perspective?
These are the kind of questions that keep me up sometimes, along with the shear complexity of analyzing the security model of a JIT VM to conduct proper code review for a high-value user. Many high end firms have WASM experience, but it may price the lower end of security reviews out of the market due the complexity of the anaylsis required. The simplicity of the EVM is very attractive to me for that reason, but it is a trade with tooling.
The longer we work on EVM, the more painful it will be to port over the ecosystem later.
@fubuloubu 80/20 or 90/10 is the rule of thumb across many domains.
The security of individual contracts is a matter of whether they do what they are intended to do. Verify, review, test… The simplicity or the EVM makes formal verification simpler, and I’m working to make it simpler still. eWasm is simple enough at this level as well, and technically EVM and eWasm can continue to evolve in parallel.
The security of the network is a matter of gas costs aligning well enough with actual client performance to not be DoS vectors. Our simple, battle-hardened EVM interpreters have the advantage there for now.
I agree that network security is a separate concern, and have confidence that it will be handled well to prevent DoS.
Compilers can be updated with hopefully minimal disruption to developers by substituting a new IR compilation process. This is assuming none of the fundamentals of using the EVM change (in a backwards-incompatible way).
Verification tools is a larger effort. Many tools manage their own model of the EVM and present that to the user for analysis, and if they have not created adequete abstraction this process could be painful for them and their users. We should account for it in any release schedule. K framework, Mythril, and Manticore are a few I have in mind, but many more exist.
Perhaps this is already accounted for, but we must have all the stakeholders aligned in advance of releasing eWASM so we all have time to add this ability to our projects and inform our users of how to migrate their code. This needs to happen regardless of whether Ethereum 1.0 ends up a subchain or not.
I have less confidence than you do that network security will be handled well, Bryant. Ethereum’s requirements are not the same as most other networks, and I think many client developers have yet to grapple with the implications.
I’m not sure which compilers it is you think can be updated easily. I think most all our EVMs are interpreters, and wouldn’t trust existing Wasm compilers with any kind of security.
As I understand it the EVM would not wind up a subchain anytime soon, as EVM code must keep running on the mainchain in order for the Shasper beacon chain to work. Further, every accessible EVM contract on the blockchain must keep running somewhere.
And indeed, verification is difficult, and porting verified EVM code to eWasm–and your specs and models and tools–might not be fun. It would easier to trust an evm2wasm compiler.
So I expect it will take a while for the community to sort out where all of this is going, and expect eWasm to expand the ecosystem rather than replace the EVM. Things have by no means all been accounted for–in many ways we are just getting started.
