Viewers say they're 'not ready' for the show change