I've been thinking about this for a while. Madara brought in all the uchiha to his alliance with the first hokage to form Konoha villiage. They were loyal to him. they were all his clansmen. When things went sour between Madara and the first. id imagine that all the uchiha in the villiage would have attacked the first Hokage and anyone loyal to him. When Madara was defeated it would have made alot more sense for every uchiha in the villiage to leave with him. It's just strange to find that they would still be living in Konaha villiage after they're leader was kicked out. Most people would think that they would be plotting some kind of rebellion or something.