Freddie has spoken out following Livi's exit from the show