That’s a lot to put on the moderators, not to mention it provides a way for the community to help decide what content does not belong. Moderators are far from infallible and can’t read every single comment. Downvotes help clue mods into what they should look over.