Context. Pure context. It's way harder to enter midway into a conversation in Japanese if you're learning it. Native speakers have grown used to it and though it is still difficult are more accurate than learners usually. Mistakes can still happen and this is why puns and the like are very common in Japanese comedy. Spoken Japanese is really difficult either way. I've spent over two years on it and I'd still rather read a sentence I only understand half of than hear one.