A cafe and gift shop dedicated to the wizarding world of Harry Potter is set to close