FWIW it's quite easy to farm karma to get to the downvote threshold. If you go to any of the energetic political HN threads, you just need to submit an enthusiastic comment that is pro the dominant position and you'll get tens of upvotes quite quickly. Do that for a week and you'll be ready to downvote. Before HN had LLM detection I had an experiment with an LLM doing that (though I approved everything it wrote by hand) and farmed karma very quickly. I stopped because I felt quite gross about it, but realize it's probably trivial to do now.

The problem isn't really the site, it's the people. People like smashing the up and down buttons on things they agree with and unfortunately a comment's score affects where it shows up on the page.