Anthony Patterson delivered a straight-to-the-point answer when asked about the transfer stories surrounding him