Sources who know the Prime Minister have spoken out about how the job is affecting him