Check robots.txt

Fetch disallowed paths the site owner tried to hide from crawlers.

Fetch robots.txt for paths the site owner marked as disallowed or private. A hit is a listing like /admin/, /backup/, or /dev/ that directory brute force alone did not find.