Culture Club singer said he was ‘silenced a lot’ in his childhood home