Fortunately not, in the town upgrade tree section there's town slot upgrades, so when you get 1 of those upgrades it will let you upgrade the townsfolk.