I think your assessment is sound. The weak argument I will make is that it is not particularly important that there are many people building blocks regularly, but it is critically important that there are multiple people trying to build blocks regularly, even if some of them never succeed.
What we want to avoid is a single builder who goes offline (for any reason) and then all block production halts. I believe the current plan in all execution clients to deal with this is to fallback to building a block from the public gossip transaction pool if no block builder proposes anything better. However, I do worry about all transactions moving over to private channels with the one builder, and then when it goes offline there are no transactions in the public gossip pool.
Note: The above assumes that we have must-include transaction lists to protect against single-builder censorship.