Jeff Brazier is 'taking legal action' to prevent his son from seeing his grandmother