Provide full functionality on all devices. Mobile users expect the same functionality - such as commenting and check-out - and content on mobile as well as on all other devices that your website supports. In addition to textual content, make sure that all important images and videos are embedded and accessible on mobile devices. For search engines, provide all structured data and other metadata - such as titles, descriptions, link-elements, and other meta-tags - on all versions of the pages.
The third type of crawling tool that we touched upon during testing is backlink tracking. Backlinks are one of the building blocks of good SEO. Analyzing the quality of your website's inbound backlinks and how they're feeding into your domain architecture can give your SEO team insight into everything from your website's strongest and weakest pages to search visibility on particular keywords against competing brands.
We concentrated on the keyword-based aspect of all the SEO tools that included the capabilities, because that's where most business users will primarily focus. Monitoring particular keywords and your existing URL positions in search rankings is important but, once you've set that up, it's largely an automated process. Automated position-monitoring features are a given in most SEO platforms and most will alert you to issues, but they don't actively improve your search position. Though in tools such as AWR Cloud, Moz Pro, and Searchmetrics, position monitoring can become a proactive process that feeds back into your SEO strategy. It can spur further keyword research and targeted site and competitor domain crawling.

Website owners recognized the value of a high ranking and visibility in search engine results,[6] creating an opportunity for both white hat and black hat SEO practitioners. According to industry analyst Danny Sullivan, the phrase "search engine optimization" probably came into use in 1997. Sullivan credits Bruce Clay as one of the first people to popularize the term.[7] On May 2, 2007,[8] Jason Gambert attempted to trademark the term SEO by convincing the Trademark Office in Arizona[9] that SEO is a "process" involving manipulation of keywords and not a "marketing service."
When referring to the homepage, a trailing slash after the hostname is optional since it leads to the same content ("" is the same as ""). For the path and filename, a trailing slash would be seen as a different URL (signaling either a file or a directory), for example, "" is not the same as "".
When Googlebot crawls a page, it should see the page the same way an average user does15. For optimal rendering and indexing, always allow Googlebot access to the JavaScript, CSS, and image files used by your website. If your site's robots.txt file disallows crawling of these assets, it directly harms how well our algorithms render and index your content. This can result in suboptimal rankings.
Keep resources crawlable. Blocking page resources can give Google an incomplete picture of your website. This often happens when your robots.txt file is blocking access to some or all of your page resources. If Googlebot doesn't have access to a page's resources, such as CSS, JavaScript, or images, we may not detect that it's built to display and work well on a mobile browser. In other words, we may not detect that the page is "mobile-friendly," and therefore not properly serve it to mobile searchers.
You should build a website to benefit your users, and any optimization should be geared toward making the user experience better. One of those users is a search engine, which helps other users discover your content. Search Engine Optimization is about helping search engines understand and present content. Your site may be smaller or larger than our example site and offer vastly different content, but the optimization topics we discuss below should apply to sites of all sizes and types. We hope our guide gives you some fresh ideas on how to improve your website, and we'd love to hear your questions, feedback, and success stories in the Google Webmaster Help Forum1.
This helpful tool scans your backlink profile and turns up a list of contact information for the links and domains you'll need to reach out to for removal. Alternatively, the tool also allows you to export the list if you wish to disavow them using Google's tool. (Essentially, this tool tells Google not to take these links into account when crawling your site.)

We expect advertisements to be visible. However, you should not let the advertisements distract users or prevent them from consuming the site content. For example, advertisements, supplement contents, or interstitial pages (pages displayed before or after the content you are expecting) that make it difficult to use the website. Learn more about this topic.38
This can be broken down into three primary categories: ad hoc keyword research, ongoing search position monitoring, and crawling, which is when Google bots search through sites to determine which pages to index. In this roundup, we'll explain what each of those categories means for your business, the types of platforms and tools you can use to cover all of your SEO bases, and what to look for when investing in those tools.

Google is falling into a familiar pattern. First, they offer web publishers increased visibility and SERP display options. Next, they incent participation in specific formats and data structures. Finally, they take that data for themselves, changing the SERPs to favor advertising, their own properties, and/or instant answers that can reduce publisher traffic. For web marketers, it's a prisoner's dilemma. In this presentation, Rand will show data on how Google is being used today, how it's changing, then dive into strategic initiatives and specific examples of how savvy players can build a moat to protect against long-term risk.