I like to think of it as multiple contracts or rules that create different and incompatible kinds of societies, and everyone has to decide what they prefer. E.g. do you want to live in a Nazi society or a pluralistic society? They are fundamentally incompatible, so supporting one means you cannot tolerate the other.

There is no paradox in this, and also no question about who starts the intolerance or whose intolerance is justified. We all just pick which path we prefer, and what we should do follows from there.