The construction boss says diverting tradies from building much-needed homes to help governments deliver on their infrastructure agenda has caused problems.