In a society where the will of the majority is ignored in favour of the will of the few I think most people would agree it’s an authoritarian society

Sure. But then the reverse is true, as well. Libertarians call it “mob rule”