To avoid undesirable content in the search indexes, webmasters can instruct spiders not to crawl certain files or directories through the standard robots.txt file in the root directory of the domain. Additionally, a page can be explicitly excluded from a search engine's database by using a meta tag specific to robots (usually ). When a search engine visits a site, the robots.txt located in the root directory is the first file crawled. The robots.txt file is then parsed and will instruct the robot as to which pages are not to be crawled. As a search engine crawler may keep a cached copy of this file, it may on occasion crawl pages a webmaster does not wish crawled. Pages typically prevented from being crawled include login specific pages such as shopping carts and user-specific content such as search results from internal searches. In March 2007, Google warned webmasters that they should prevent indexing of internal search results because those pages are considered search spam.
Consider your competition. Look at what your competitors are doing and how they are performing in their search marketing before you decide how you can best compete with them. Research what search terms they rank organically for. Consider if you can execute a plan to top their SERP placements. Also, look at what paid terms they are using to drive traffic to their own sites. As you perform this research, look for gaps that you can fill and areas where you will be unable to compete in both paid and organic search.
It might be awkward, but please don't scroll past this. This Monday we humbly ask you to defend Wikipedia's independence. Our 2019 fundraiser will be over very soon. 98% of our readers don't give; they look the other way. If you are an exceptional reader who already donated, we sincerely thank you. If all our readers donated just $2.75 today, Wikipedia could keep thriving for years. Most people donate for a simple reason—because Wikipedia is useful. If Wikipedia gave you $2.75 worth of knowledge this year, take a minute to secure its future with a gift to the Wikimedia Endowment. Show the volunteers who bring you reliable, neutral information that their work matters. Thank you.