And finally, the other really important bucket is authority. Google wants to show sites that are popular. If they can show the most popular t-shirt seller to people looking to buy t-shirts online, that’s the site they want to show. So you have to convince Google - send them signals that your site is the most popular site for the kind of t-shirts that you sell.
Early versions of search algorithms relied on webmaster-provided information such as the keyword meta tag or index files in engines like ALIWEB. Meta tags provide a guide to each page's content. Using metadata to index pages was found to be less than reliable, however, because the webmaster's choice of keywords in the meta tag could potentially be an inaccurate representation of the site's actual content. Inaccurate, incomplete, and inconsistent data in meta tags could and did cause pages to rank for irrelevant searches.[dubious – discuss] Web content providers also manipulated some attributes within the HTML source of a page in an attempt to rank well in search engines. By 1997, search engine designers recognized that webmasters were making efforts to rank well in their search engine, and that some webmasters were even manipulating their rankings in search results by stuffing pages with excessive or irrelevant keywords. Early search engines, such as Altavista and Infoseek, adjusted their algorithms to prevent webmasters from manipulating rankings.
Robots.txt is not an appropriate or effective way of blocking sensitive or confidential material. It only instructs well-behaved crawlers that the pages are not for them, but it does not prevent your server from delivering those pages to a browser that requests them. One reason is that search engines could still reference the URLs you block (showing just the URL, no title or snippet) if there happen to be links to those URLs somewhere on the Internet (like referrer logs). Also, non-compliant or rogue search engines that don't acknowledge the Robots Exclusion Standard could disobey the instructions of your robots.txt. Finally, a curious user could examine the directories or subdirectories in your robots.txt file and guess the URL of the content that you don't want seen.
"I wanted to thank you guys for everything you've done for my company. When I first went to Curt for help about a 1.5 years ago, I was a very tiny company. We're now doing about 1500 jobs a month, and I give a lot of credit to you guys for the exposure. It's been life changing for me. I was working 12 hour days, 7 days a week for 2 years. I am finally able to back off some because I can afford office help as well. Thanks for being so great at what you do. I still don't know what that is exactly, but thanks for doing it so well."
In March 2006, KinderStart filed a lawsuit against Google over search engine rankings. KinderStart's website was removed from Google's index prior to the lawsuit, and the amount of traffic to the site dropped by 70%. On March 16, 2007, the United States District Court for the Northern District of California (San Jose Division) dismissed KinderStart's complaint without leave to amend, and partially granted Google's motion for Rule 11 sanctions against KinderStart's attorney, requiring him to pay part of Google's legal expenses.
Your website is the “hub” of your online brand – so, it’s important to have regular checkups to ensure everything is in order. It’s also important to note that your website is a living digital property, it’s typically not stagnant for long periods of time. In any given year, content is added and/or removed from your site. It is for this reason that audits should occur on a regular basis. We recommend that websites be audited at a minimum of once per year. That allows your teams to fix critical issues as they arise.
Webmasters and content providers began optimizing websites for search engines in the mid-1990s, as the first search engines were cataloging the early Web. Initially, all webmasters only needed to submit the address of a page, or URL, to the various engines which would send a "spider" to "crawl" that page, extract links to other pages from it, and return information found on the page to be indexed. The process involves a search engine spider downloading a page and storing it on the search engine's own server. A second program, known as an indexer, extracts information about the page, such as the words it contains, where they are located, and any weight for specific words, as well as all links the page contains. All of this information is then placed into a scheduler for crawling at a later date.