Verifiable delay functions and attacks

Is the difficulty adjustment algorithm in VDF different than the one in PoW? I imagine it should not be since in VDF scheme everyone submits instead of only the fastest one as in PoW.
If we use PoW difficulty adjustment algorithm in VDF(i.e., to limit the fastest participant), participants would start to fail to reveal in time as attackers push up the difficulty and I suppose this is observable hence we could act on this in time to prevent the difficulty from climbing up?