The Progressive Conservatives have won a provincial byelection in eastern Ontario, retaining a seat previously held by a popular cabinet minister.