Life for chart toppers Robson Green and Jerome Flynn hasn't always been easy