Wivenhoe residents living opposite an oak which is due to be axed have spoken of their concerns.