I always heard it comes from the fathers mother, and for me, that was a hundred percent fact. My poor grandmother has super thin hair, and has to start wearing pieces fairly young for a women.
I think the reason we notice it more is because the mental torture of modern life is literally just making men go bald faster. It's still genetic, just generally happening earlier.