Recent polls indicate that although former President Trump is getting some traction among Latino and Black voters, one group is moving away from him.