The Australian heartthrob didn't have to force a connection with the LA-born Oscar winner, who's 23 years older than him