20 Myths About synthetic dataset: Busted
https://www.instapaper.com/read/1679739954
This enables Goggle (user-agent identify GoogleBot) to visit each and every site and directory, although at the same time banning Talk to Jeeves in the site entirely