Plans have been submitted to build more than 50 new houses on land near to a historic castle.