# robots.txt for LDD servers http://www-10.lotus.com/ etc. # # Date User Comment # 26-Oct-2004 craig rewritten to disallow crawling of specific "directories" in the main developerWorks index, such as today.nsf # 05-Jan-2004 craig rewritten to correct comment typos and missing colon after Disallow # Dec-2003 craig created to have FAST bypass the LDD site servers # May-2005 tara updated robots.txt to include lbytes.nsf, LNotes-L archives, and today.nsf # Jun-2005 craig updated to restore sandbox.nsf # APR-2006 moe added dominowiki.nsf # Jun-2007 craig added notesua.nsf, sttipsblog.nsf # Mar-2008 tara added notesua.nsf and today.nsf to FAST # Mar-2008 craig removed dominowiki.nsf from * # Jun-2008 craig cleaned up nonexistent databases, removed FAST-specific section, and attempted to prevent indexing of print version documents # # NOTE User-agent: * to prevent all crawling # User-agent: FAST Corporate Crawler User-agent: * Disallow: /ldd/profile.nsf Disallow: /ldd/site.nsf Disallow: /ldd/*/MainTopicPrint* Disallow: /ldd/*/ResponsePrint* Disallow: /ldd/*/ResponsetoResponsePrint* Disallow: /ldd/r5fixlist.nsf/FixPrint* Disallow: /ldd/stfixlist.nsf/FixPrint* Disallow: /ldd/sandbox.nsf/SandboxDownloadPrint*