‘There were always layers and layers of inappropriateness,’ said ‘Holiday’ actor