Former EastEnders star Danny Dyer has said his children were “petrified of getting out the car” when he took them to the council estate where he grew up.