Surprise: as soon as you form a community, the most dependable members become a governing core. What the fuck do you think a “village elder” is?

Also, what happens when village A decides their neighbours B don’t deserve all of their land? There’s no governing body to mediate, so village A simply attacks B.