‘It’s not like I played a great golfer,’ the actor observed