“I just want to be known for doing good work,” Davidson said while confessing he does not want his past relationships with A-listers being his sole identity