I'm not assuming anything, I'm concluding what I must based on the evidence. You have some countervening evidence, then we change the conclusion. Otherwise, Occam's razor. Just because it could be more complicated doesn't mean we should believe it is without evidence.

A metastudy above concludes the same thing using unrelated data anyway [1]. Men masturbate much more than women and the effect size is greater than most all other psychological differences.

1. https://www.jstor.org/stable/20182831