You may not want certain pages of your site crawled because they might not be useful to users if found in a search engine's search results. If you do want to prevent search engines from crawling your pages, Google Search Console has a friendly robots.txt generator to help you create this file. Note that if your site uses subdomains and you wish to have certain pages not crawled on a particular subdomain, you'll have to create a separate robots.txt file for that subdomain. For more information on robots.txt, we suggest this Webmaster Help Center guide on using robots.txt files13.

Hi, Brian. Thank you for the great article. I have a question about the part about 4 website addresses. Ours currently is set to https://www., and we would like to change it to just an https:// as the main website. Will this hurt our current link profile, or will everything stay the same? This might be a foolish question, but we are a bit worried. Thank you.
SEM is a broader term than SEO, and is used to encompass different options available to use a search engine’s technology, including paid ads. SEM is often used to describe acts associated with researching, submitting and positioning a website within search engines.  It includes things such as search engine optimization, paid listings and other search-engine related services and functions that will increase exposure and traffic to your Web site. 
Awesome blog post. But I did not understand the logic behind ghost posts. could you please give more clarifications? Is it like just because the new post was published on the 4th page and not on the top of the feed, Google would give higher rankings?. And is the correlation between SEO & promoting content with facebook ads and getting traffic?. Does Getting social traffic from paid social ads improves the organic rankings?.

Hi Claire, you’re welcome. It depends. If the keyword seems like a Featured Snippet would make sense (for example, it’s a term that could use a definition or there’s a list of steps or tips), I’d still try snippetbait. One other thing I’d keep in mind is that Featured Snippets tend to float in and out. For example, the keyword “how to get more subscribers on YouTube”. That featured snippet tends to appear (with us ranking in it) and disappear on a weekly basis. Just Google testing stuff out.

A recent study done by Search Engine Watch has shown that the top position on Google gets over 30% of the visitors for any given search.  I have seen businesses with the worst looking websites bring in client after client simply because they rank higher than their competition.  You might know what I am talking about if you run a small business with a website that is not currently at number 1. Hiring a local SEO pro is a must for your business.


Kelly Wilhelme currently manages all of Weidert Group's marketing efforts. Through her past experience as an inbound marketing consultant on our client service team and, prior to that in financial services communication, she has a deep understanding of complex businesses and a desire to help them grow. Kelly has a passion for communication strategy, layout and design, as well as writing and content creation.

Page and Brin founded Google in 1998.[23] Google attracted a loyal following among the growing number of Internet users, who liked its simple design.[24] Off-page factors (such as PageRank and hyperlink analysis) were considered as well as on-page factors (such as keyword frequency, meta tags, headings, links and site structure) to enable Google to avoid the kind of manipulation seen in search engines that only considered on-page factors for their rankings. Although PageRank was more difficult to game, webmasters had already developed link building tools and schemes to influence the Inktomi search engine, and these methods proved similarly applicable to gaming PageRank. Many sites focused on exchanging, buying, and selling links, often on a massive scale. Some of these schemes, or link farms, involved the creation of thousands of sites for the sole purpose of link spamming.[25]
Google is falling into a familiar pattern. First, they offer web publishers increased visibility and SERP display options. Next, they incent participation in specific formats and data structures. Finally, they take that data for themselves, changing the SERPs to favor advertising, their own properties, and/or instant answers that can reduce publisher traffic. For web marketers, it's a prisoner's dilemma. In this presentation, Rand will show data on how Google is being used today, how it's changing, then dive into strategic initiatives and specific examples of how savvy players can build a moat to protect against long-term risk.
Webmasters and content providers began optimizing websites for search engines in the mid-1990s, as the first search engines were cataloging the early Web. Initially, all webmasters only needed to submit the address of a page, or URL, to the various engines which would send a "spider" to "crawl" that page, extract links to other pages from it, and return information found on the page to be indexed.[5] The process involves a search engine spider downloading a page and storing it on the search engine's own server. A second program, known as an indexer, extracts information about the page, such as the words it contains, where they are located, and any weight for specific words, as well as all links the page contains. All of this information is then placed into a scheduler for crawling at a later date.
×