Verifiable delay functions and attacks

It’s not about the majority of the participants, it’s about who is the fastest participant. If an attacker is the fastest participant, then they can manipulate the difficulty somewhat.