According to them and others, three ways:
- sexually antagonistic selection
- kin altruism
This is very much theoretical stuff, but according to the authors, has the potential to "generate testable predictions" regarding the genetics of homosexuality
Genetic models of homosexuality: generating testable predictions
Sergey Gavrilets and William R. Rice
Proceedings of the Royal Society B: Biological Sciences Volume 273, Number 1605 / December 22, 2006 Pages: 3031 - 3038
Abstract: Homosexuality is a common occurrence in humans and other species, yet its genetic and evolutionary basis is poorly understood. Here, we formulate and study a series of simple mathematical models for the purpose of predicting empirical patterns that can be used to determine the form of selection that leads to polymorphism of genes influencing homosexuality. Specifically, we develop theory to make contrasting predictions about the genetic characteristics of genes influencing homosexuality including: (i) chromosomal location, (ii) dominance among segregating alleles and (iii) effect sizes that distinguish between the two major models for their polymorphism: the overdominance and sexual antagonism models. We conclude that the measurement of the genetic characteristics of quantitative trait loci (QTLs) found in genomic screens for genes influencing homosexuality can be highly informative in resolving the form of natural selection maintaining their polymorphism.