As Chassidus teaches, the faults we see in others are the faults that are in us, therefore the "holy group of people" that have decided that others are the Eirev Rav, they are actually the Eirev Rav