Academics say claims by politicians and on social media are fueling xenophobia and could compromise their safety