I run Search Server Express 2010 and would like to add some external content (web-sites like twiki, wordpress...) to be crawled.
By just adding this as a contetn source I have no chance to get it indexed due to the fact that parts of the content is protected by username/password. I did this already and I obviously get the content wich is accessible for anonymous, but the rest?
I started to work with crawl rules where I can use an authentication, but I not familiar with this. However it did noo work out, I searched for some terms I know they are part of the protected area and got no result. I don't even know when these content rules are applied, are they applied while crawling local sharepoint sites or when? Should I keep my content source for the respective website AND additionally the crawl rule with the authentication set?
Is there any step-by-step guide for such a scenario? Just crawl simple protected web-sites, not integrated to Active Directore....
Regards,
Dieter