Who will be punished in Casper?

In the following case,Will people be punished?
1)Validator abandons Voting
2) The checkpoint voted by validator are not on the main chain

  1. When the validator is abandons voting, They lose their deposit every times.