The ‘Neighbors’ star said fans had ‘strong takes’ about him not having children