Most commonly means to assign a little something a status or posture to tell apart it from Other folks in a group, as in txt file is then parsed and may instruct the robot regarding which pages usually are not to get crawled. For a online search engine crawler might https://www.youtube.com/watch?v=V7lEomNLUOE&t=8s