The man asked for directions before stealing her card holder