In order to defend your village against thieves and cannibals, followers must:
- be without activity (not assigned at a work station, not in an expedition, ...)
- have a weapon in their equipment (or the Pugilist trait)
So make sure some of your villagers are in this condition at all time.
In order to defend the village against zealots, well, then it's up to you!