The castle is now fast approaching a state of dereliction