Sharding phase 1 spec (RETIRED)

@vbuterin, what about including a Bitcoin-NG style approach in the roadmap (e.g. phase 5 or 6)? Blocks are distinguished validating consensus blocks and non-consensus transaction micro-blocks. Additional features include:

  • a poison transaction in each microblock pointing to the header of the first block in the pruned branch as a proof of fraud, accompanied with a maturity window for the reward, within which the poison transaction must be placed on the blockchain; and
  • rewards split by ~40% for the miner/proposer of a leader block followed by micro-blocks and 60% to the leader of the next block).

I’m not sure if this can be used with Casper, however.

But maybe with this proposal it is not necessary or it would be more prone to adaptive adversaries.

Does compute_header_hash() also need height: in128 and proposer_signature: bytes[8192] as arguments to match with the collation header struct fields?

I think floor(block.number / PERIOD_LENGTH) + LOOKAHEAD_LENGTH >= period >= floor(block.number / PERIOD_LENGTH), otherwise we can get only collators for the current period, right?

If this function only gets the collator for the current period, you don’t need to check in add_header for:

Since this is guaranteed by the get_eligible_collator

The VMC used period_start_prevhash as a collation_header field. What’s the alternative? @mhchia @NicLin

Roadmap

The roadmap is an active area of research. The outline below is only intended to provide flavour.

And a lot more: https://ethresear.ch/c/sharding.

Reposting the above roadmap to make it easier to link to it.

Why would a proposer call release_proposer?
The proposer can call deregister_proposer, later withdraw the funds. and leave the registry untouched, never calling release_proposer.

It would be better if the proposer could withdraw with this method, effectively cleaning the registry. For example:
eliminate proposer_withdraw_balance, and modify release_proposer() as release_proposer(uint[] withdraw_shards).

for s in withdraw_shards, withdraw release_proposer.balances[s]
At the end of the call, delete the registry.

So you set the lockup period in deregister_proposer, then only after the lockup period you can withdraw the funds, which AIUI should be done with withdraw_balance, since that withdraws a balance for a shard, and a proposer may have balances in more than one shard.

For your proposed implementation you would need to enter each shard_id as arguments to release_proposer(), do the asserts for each shard, clear the balances in each shard, and still assert proposer_registry[proposer_address].balances == 0 before setting proposer_registry[proposer_address].deregistered = 0. Additionally, you would still need a separate proposer_withdraw_balance() in case proposers just want to withdraw funds in one shard (but not all shards that they have balances, as may be the case), and then not release_proposer(). So it seems better to keep the functions separate, but perhaps there could be a separate function to clear funds from all shards and call release_proposer() at the end of it.

If you want to see how I’ve written the SMC so far, have a look here: https://github.com/Drops-of-Diamond/sharding/blob/develop/smc/Sharding_Manager_Contract.v.py. Speaking of which, I think I need to talk to @mhchia about this, for instance about what to do since there is no period_start_prevhash. Perhaps you would like to review it and we can discuss on Gitter, e.g. a private chat? I think expected_period_number is equivalent to period.

get_eligible_collator(shard_id, period) actually accepts periods from current period to lookahead_period, while we only want period to be the current period in add_header. So we still need that check, if I didn’t understand your meaning wrong.

1 Like

Because the fund is locked up for PROPOSER_LOCKUP_LENGTH. So deregister_proposer simply removes a proposer from the proposer_pool, and locks up its fund. It needs to call release_proposer to actually retrieve the funds after PROPOSER_LOCKUP_LENGTH

It was used for collations to point to a block in the mainchain.
Since we don’t have the field for now, just remove the check?

Yep, I commented it out. Just wondering why we removed it.

I don’t think so. If the proposer calls release_proposer without proposer_withdraw_balance, the funds are lost. proposer_withdraw_balance retrieves the funds, not release_proposer.

Why a rational proposer would call release_proposer?

So as I explained above with the current proposed spec:

  • a proposer can have balances in multiple shards
  • they must withdraw their balances from all shards, one at a time, with proposer_withdraw_balance(shard_id).
  • then once all balances are withdrawn they call release_proposer() to withdraw from the collator pool.

However, a proposer could withdraw all their balances and not call release_proposer(). Then that would fill up empty slots in the proposer_registry. So it seems that there should also be a check to periodically check after the lockup_length to see if proposer_registry[proposer_address].balances == 0, and if so, set proposer_registry[proposer_address].deregistered = 0, otherwise break and do nothing. But this is probably related to storage rent / time to live / pay-to-resurrect schemes, and intuitively I don’t think it poses a security risk.

Actually I just realised that the SMC as I have written it at the moment doesn’t really account that a proposer could have balances in multiple shards, so I’ll need to rewrite it to account for that.

So as I explained above:
The proposer has nothing to gain by calling release_proposer. So rationally this method will never be called, just wastes gas. This whole “periodically check” you’re proposing is a hack to fix a design flaw.

Why assert proposer_registry[proposer_address].balances == 0?
That’s not done in the current implementation, so why it should be done in the other one?
Just transfer whatever is in balances to the proposer.

Also, in the current specification, a proposer can withdraw only one at a time, if I deposited in the 100 shards, I must call deregister_proposer then proposer_withdraw_balance, 100 times. Since the lock up period is not per shard. Assuming proposer_withdraw_balance resets the lockup time (set the proposer as registered again).

P.S.
In your implementation:
self.collator_registry[self.collator_address].deregistered \ = self.collator_lockup_length
This is wrong, you’re saving a constant every time there. See: https://github.com/enriquefynn/go-ethereum/blob/smc_phase1/sharding/contracts/sharding_manager.sol, I’m also doing some stuff there.

OK, fair point. The balances aren’t stored in each shard, so they could be withdrawn with release_collator. But separating functionality also is conventional. In Vyper you could put a @private decorator before proposer_withdraw_balance(), and instead call proposer_withdraw_balance() from within release_proposer(). But a problem with that is that you can’t then pass shard_id directly from proposer_withdraw_balance(), you’d have to pass it to release_proposer(), then internally to proposer_withdraw_balance().

With separate functions, then the assert is there to make sure that balances are empty (from proposer_withdraw_balance() before releasing the collator, to make sure that funds are not lost. But if you call proposer_withdraw_balance() from inside release_proposer(), it still seems like best practice to make that assert to prevent lost funds. Of course I should’ve raised the change with others, but it is there to read.

That’s a fair point. As I mentioned, we can certainly have another function that withdraws all balances from all shards and then releases the proposer. It just adds more complexity because you need to keep track of where proposers are registered in each shard if you want to try to withdraw by shard_id. It certainly is simpler to do it the same way as release_collator and just release all balances, and not bother about withdrawing and adding to individual shards. However, there is an advantage in being able to withdraw and add to one shard, so we could do a similar thing for collators. But withdrawing and adding to individual shards is more of an enhancement that can be done later, as it does add complexity.

It’s good to have others scrutinizing it! I can move it to a separate private function that does this, and call the function. Do you have a better idea? It’s still a WIP, and I will certainly be reviewing it multiple times for security and optimizations, although I am sure that others will too, and I do need to work on the Rust side of things as well, but @ChosunOne has started contributing. Besides, I think all clients should scrutinize the Vyper contract, since the plan is to just interact with it, not have multiple SMCs. I’ll see if I can find time to look at your code.

What’s the advantage of using something different to RLP?

In my understanding, you do release_proposer to get back your deposit?
I mean, it seems balance and deposit are separate concepts.
So you mentioned proposer_withdraw_balance, it only withdraws the balance in a specific shard of that proposer, but the deposit is still locked up in the contract.
So the question might be, why do we need a deposit for each proposer? and why is it needed to be locked?(is there a slashing condition for proposers?)

1 Like

From the sharding spec, the collation pool is defined as “The set of all collators collectively participating in the security of all shards.” Does this mean that a collator must store all the state of all the shards at all times or is it ok for the collator to only keep the state of its currently assigned shard?

Additionally, does a collator need to download the entire state of the assigned shard, or just to the windback depth?

1 Like

Do we need proposer_address as an explicit collation header field? I’m thinking maybe we can save another 32 bytes and just recover the address from the signature, similar to how it’s done with transaction senders in Ethereum right now.

3 Likes