"I wanted to thank you guys for everything you've done for my company. When I first went to Curt for help about a 1.5 years ago, I was a very tiny company. We're now doing about 1500 jobs a month, and I give a lot of credit to you guys for the exposure. It's been life changing for me. I was working 12 hour days, 7 days a week for 2 years. I am finally able to back off some because I can afford office help as well. Thanks for being so great at what you do. I still don't know what that is exactly, but thanks for doing it so well."
Robots.txt is not an appropriate or effective way of blocking sensitive or confidential material. It only instructs well-behaved crawlers that the pages are not for them, but it does not prevent your server from delivering those pages to a browser that requests them. One reason is that search engines could still reference the URLs you block (showing just the URL, no title or snippet) if there happen to be links to those URLs somewhere on the Internet (like referrer logs). Also, non-compliant or rogue search engines that don't acknowledge the Robots Exclusion Standard could disobey the instructions of your robots.txt. Finally, a curious user could examine the directories or subdirectories in your robots.txt file and guess the URL of the content that you don't want seen.
Optimizing a website may involve editing its content, adding content, and modifying HTML and associated coding to both increase its relevance to specific keywords and remove barriers to the indexing activities of search engines like Google ,Yahoo etc.[citation needed] Promoting a site to increase the number of backlinks, or inbound links, is another SEO tactic. By May 2015, mobile search had surpassed desktop search.[3]