^^;; Allen isn't the brightest of people, but he's not so stupid that intelligence would be a factor in such things (he's smarter than Kanda, after all). Age is probably the best reason, and no one could argue against it since everyone else at the meeting is an adult (technically, Link wouldn't be an adult in Japan, but this isn't set there...).
Non-Exorcists would probably be really confused and either not care, avoid him, or act... what's the word... suck-up, I guess? =P Something like that. (And Lavi wouldn't stop teasing him...)