The general idea in the “Collective Coin Flipping” paper is also known as “low-influence functions”. Another paper on the topic by @iddo and others is this one. A downside of low-influence functions (e.g. see this post by @cleasage) is worsened collusion resistance.
Dfinity’s threshold scheme (which I understand Polkadot also intends to use) is quite cool. The main downside is liveness: if a significant portion of the network goes offline then the randomness beacon—and the whole blockchain!—stalls. (The more minor issues are quantum insecurity, a complex secret sharing setup, and adaptive attacks on infrequently-shuffled committees.) One of the design goals of Ethereum is to survive WW3. Even if 90% of nodes go offline we want the network to continue running.
From a consensus perspective, I think @vbuterin is relatively confident that the Ethereum 2.0 protocol can be made robust enough to run on RANDAO despite it being a “weak” source of entropy. I think the best-case scenario of running on pure RANDAO is that the relevant design parameters—honesty assumptions, committee sizes, committee thresholds, etc.—are worsened to take into account the biasability of RANDAO.
A few remarks on the VDF approach:
- It is the only known approach that is simultaneously un-biasable and un-stoppable (under a minority liveness assumption).
- As pointed by @dlubarov the VDF strictly improves the RANDAO beacon. Even if the VDF is completely broken—i.e. outputs can be computed instantly—we fall back to the underlying RANDAO. As such, the potential for backfiring is limited.
- Part of the value of a VDF-based randomness beacon is to expose strong randomness to dApps via an opcode. Replicating this critical piece of infrastructure at L2 without L1 support (such as VDF rewards and forced RANDAO participation) is hard/impossible.