It happens everywhere in Asia

I just got back from a trip to Seoul to visit a friend of mine. 

Everyone expected me to speak Korean, even though I'm convinced they should've been able to tell I wasn't Korean from the comparatively little effort I put into dressing myself.

I again came up against the weird Asian language triangle when I was out with my caucasian friends.  My friend speaks Korean to the Korean person.  The Korean person answers in Korean, but answers me, not my friend.  I make pointless apologies and look at my friend for help.  They speak back to the Korean person. 

Swap Chinese for Korean, and that's exactly the same thing that happens here.  The locals don't think they're being rude, but they automatically speak to the familiar-looking face.  I hope I don't subconsciously do that, but I can't think of a time when I've been in the reverse situation.