Discussion: P2P message serialization standard

Sharding
14

I’ve been playing with the serialization scheme I proposed on the following:

## This is Nim code, packed to avoid padding.
type
  ValidatorRecord {.packed.} = object
    # The validator's public key
    pubkey:  Uint256
    # What shard the validator's balance will be sent to
    # after withdrawal
    withdrawal_shard: int16
    # And what address
    withdrawal_address: EthAddress
    # The validator's current RANDAO beacon commitment
    randao_commitment: Hash256
    # Current balance
    balance: int64
    # Dynasty where the validator  is inducted
    start_dynasty: int64
    # Dynasty where the validator leaves
    end_dynasty: int64

when isMainModule:
  let x = ValidatorRecord(
    pubkey: 123456789.u256,
    withdrawal_shard: 4455,
    withdrawal_address: hexToPaddedByteArray[20]("0x1234"),
    randao_commitment: Hash256(data: hexToPaddedByteArray[32]("0xAABBCCDDEEFF")),
    balance: 100000,
    start_dynasty: 1,
    end_dynasty: 2
  )

When serialized it has the following binary structure:

Field Value Size (in Bytes)
Magic header [’\x7F’,‘E’,‘T’,‘H’,‘E’,‘R’,‘E’,‘U’,‘M’] 9
Version [1, 0] as [Major, Minor] 2
Start offset of raw data E2 (=226) as big endian int64 8
Blake2_256 hash 0x390056867ac072c4cfcb5d4abc60bdc8ea8cc3941c4ecb9874254af1bd4b281d 32
Schema {“pubkey”:“UInt256”,“withdrawal_shard”:“int16”,…,“start_dynasty”:“int64”,“end_dynasty”:“int64”} 175
Raw data Should be equivalent to SimpleSerialize 110

Implementation in Nim

Unfortunately as you can see including the schema brings lots of overhead for the types we want serialized: the schema is 175 bytes while the raw data is only 110 bytes.

I think the spec should include predefined schemas so that we don’t have to include them in messages.

Edit: Ellipsis in the schema in the table so that we see the sizes at a glance

1 Like