From http://esciencenews.com/articles/2009/06/15/stress.puts.double.whammy.reproductive.system.fertility

The new research shows that stress also increases brain levels of a reproductive hormone named gonadotropin-inhibitory hormone, or GnIH ... This small protein hormone puts the brakes on reproduction by directly inhibiting GnRH.

The common thread appears to be the glucocorticoid stress hormones, which not only suppress GnRH but boost the suppressor GnIH - a double whammy for the reproductive system.