Northern Ireland’s biggest school has failed in a bid to transform to integrated status.