The LoP also claimed that the Lok Sabha Speaker made an "unsubstantiated remark" about him before adjourning the House.