I can’t see how but that both the victims, and everyone else they affect, deciding. That doesn’t mean they’ll all come to the same conclusion, of course.
I’m pretty sure that’s where politics comes from, personally...
Edited to add: I do not mean to imply that if one group decides X, another Y, and a third Z, that it necessarily means that any of them are wrong.
And that doesn’t even bring up the fact that if the farmers have no way to plow the field once they are no longer being harassed, they’re going to starve regardless of what the adventurers do to the elves.