The term for it is “micro-aggression.”

Much of the time, suggests Santa Rosa psychologist Dr. Marco Lopez, casual “micro-aggressions” are more thoughtless than intentionally disparaging.
But the person on the receiving end of such remarks, Lopez continues, is going to recognize their hurtful nature, even if the speaker doesn’t.
One active online forum for discussion of this issue is The Microaggressions Project. You can also see a photo gallery of examples here.