It could annoy some of his most loyal supporters