Time swallows or eats away the remedies that people are seeking, the Supreme Court judge explained