While a builder monopoly/oligopoly is obviously not ideal, it doesn’t break security assumptions other than censorship-resistance. The key point here is that controlling the executable part of a block does not give any control over consensus, as this is determined by attestations.
What this means practically is that even a monopolistic builder cannot do anything to break liveness or safety, nor they can do anything to force reorgs and double spend etc…
What would be really dangerous is for the validator set to centralize, because that opens the door to all of these attacks. It is therefore much much more important for us to preserve the decentralization of the validator set rather than that of the set of builders
Even for censorship-resistance, we wouldn’t have any guarantees if the validator set comes to be dominated by an oligopoly of extractors: even if a non-censoring minority existed, 51% of the stake can completely control the fork-choice and censor it. On the other hand, a decentralized validator set allows us to create mechanisms which give us strong censorship-resistance guarantees even with PBS (see here for the current state of research on this).