A young family say they are frightened to live in their home after mould started appearing "just weeks after moving in."