Ramaswamy’s campaign has claimed the former presidential candidate’s comments were a “joke”