I always heard it comes from the fathers mother, and for me, that was a hundred percent fact. My poor grandmother has super thin hair, and has to start wearing pieces fairly young for a women.
I think the reason we notice it more is because the mental torture of modern life is literally just making men go bald faster. It's still genetic, just generally happening earlier.
Also, boys that got castrated don't get hair loss. Any hair loss, ever