The tweet stated that it was depressing to learn that Annamalai is banned from visiting the temple