Governments learnt very early on that some of these visitors look identical to us, and that there'd be no way to distinguish human from alien, therefore the potential panic and suspicion would be too great?

I can imagine the reaction to little grey men. But the idea that there are aliens among us? This would truly terrify people.