John F. Kennedy attempted to push him around, but he stood his ground