To avoid undesirable content in the search indexes, webmasters can instruct spiders not to crawl certain files or directories through the standard robots.txt file in the root directory of the domain. Additionally, a page can be explicitly excluded from a search engine's database by using a meta tag specific to robots (usually ). When a search engine visits a site, the robots.txt located in the root directory is the first file crawled. The robots.txt file is then parsed and will instruct the robot as to which pages are not to be crawled. As a search engine crawler may keep a cached copy of this file, it may on occasion crawl pages a webmaster does not wish crawled. Pages typically prevented from being crawled include login specific pages such as shopping carts and user-specific content such as search results from internal searches. In March 2007, Google warned webmasters that they should prevent indexing of internal search results because those pages are considered search spam.[47]
A few years back we decided to move our community forum from a different URL (myforum.com) to our main URL (mywebsite.com/forum), thinking all the community content could only help drive additional traffic to our website. We have 8930 site links currently, which probably 8800 are forum content or blog content. Should we move our forum back to a different URL?

Ranking refers to the process search engines use to determine where a particular piece of content should appear on a SERP. Search visibility refers to how prominently a piece of content is displayed in search engine results. Highly visible content (usually the content that ranks highest) may appear right at the top of organic search results or even in a featured snippet, while less-visible content may not appear until searchers click to page two and beyond
In December 2009, Google announced it would be using the web search history of all its users in order to populate search results.[33] On June 8, 2010 a new web indexing system called Google Caffeine was announced. Designed to allow users to find news results, forum posts and other content much sooner after publishing than before, Google caffeine was a change to the way Google updated its index in order to make things show up quicker on Google than before. According to Carrie Grimes, the software engineer who announced Caffeine for Google, "Caffeine provides 50 percent fresher results for web searches than our last index..."[34] Google Instant, real-time-search, was introduced in late 2010 in an attempt to make search results more timely and relevant. Historically site administrators have spent months or even years optimizing a website to increase search rankings. With the growth in popularity of social media sites and blogs the leading engines made changes to their algorithms to allow fresh content to rank quickly within the search results.[35]
Alt text (alternative text), also known as "alt attributes" describe the appearance and function of an image on a page. Alt text uses: 1. Adding alternative text to photos is first and foremost a principle of web accessibility. Visually impaired users using screen readers will be read an alt attribute to better understand an on-page image. 2. Alt tags will be displayed in place of an image if an image file cannot be loaded. 3. Alt tags provide better image context/descriptions to search engine crawlers, helping them to index an image properly.
You can confer some of your site's reputation to another site when your site links to it. Sometimes users can take advantage of this by adding links to their own site in your comment sections or message boards. Or sometimes you might mention a site in a negative way and don't want to confer any of your reputation upon it. For example, imagine that you're writing a blog post on the topic of comment spamming and you want to call out a site that recently comment spammed your blog. You want to warn others of the site, so you include the link to it in your content; however, you certainly don't want to give the site some of your reputation from your link. This would be a good time to use nofollow.
Both require knowing your audience. To succeed at both strategies, you must have a good understanding of your audience and how they act. By using buyer personas and psychographic segmentation, you can get to know your audience, discover what their needs are, and what they are searching for. Then you can create valuable content that shows up when they go looking for solutions related to your brand.
Creating high quality content takes a significant amount of at least one of the following: time, effort, expertise, and talent/skill. Content should be factually accurate, clearly written, and comprehensive. So, for example, if you describe your page as a recipe, provide a complete recipe that is easy to follow, rather than just a set of ingredients or a basic description of the dish.
Consider the current status of your website. When you create a marketing strategy, look for the “low-hanging fruit”, or the opportunities that will make the biggest impact with the least amount of work Click & Tweet! . So before you launch a search marketing campaign, research your website to see where you may have the potential to grow an organic SEO strategy that is already working before putting money into an SEM campaign.
Most small businesses owners and marketers know a little something about SEO (search engine optimization) and the different tactics to help your website rank well in organic search engine results. Another important tactic for any Internet business to know about is SEM (search engine marketing), which includes things such as search engine optimization, paid listings and other search engine related services.
To get even more insight and data to help you make those decisions, sign up for a free trial of Alexa’s Advanced Plan. You’ll get access to tools that help you research competitor search and link building strategies, find keyword opportunities, review your site’s SEO, and learn about your target audience. These insights, paired with what you know about SEM and SEO, will help you uncover the best search marketing strategy for your unique brand and goals.
Analysis is the key to SEO – Monitoring your ranking on search engines is key to getting better results. Start by tracking the most important website metrics to set a baseline for your performance. Make small content changes and see if you notice a boost in your site traffic or rankings. Avoid making several unrelated changes simultaneously so you can always keep track of what was responsible for improved performance.
Hiring a talented local SEO (short for Search Engine Optimization) pro is the best way to help your business grow online and get in front of the people who are looking for what you have.  Rebel Base SEO is based in Little Elm Texas. If you are not familiar with the area, that is about 20 miles north of Dallas, just a little north of The Colony, to the West of Frisco, Plano, Anna and McKinney, south of Prosper and East of Denton.  We have clients right here in Texas and as far out as California, New York, Pennsylvania and more.
Users will occasionally come to a page that doesn't exist on your site, either by following a broken link or typing in the wrong URL. Having a custom 404 page30 that kindly guides users back to a working page on your site can greatly improve a user's experience. Your 404 page should probably have a link back to your root page and could also provide links to popular or related content on your site. You can use Google Search Console to find the sources of URLs causing "not found" errors31.
A variety of methods can increase the prominence of a webpage within the search results. Cross linking between pages of the same website to provide more links to important pages may improve its visibility.[48] Writing content that includes frequently searched keyword phrase, so as to be relevant to a wide variety of search queries will tend to increase traffic.[48] Updating content so as to keep search engines crawling back frequently can give additional weight to a site. Adding relevant keywords to a web page's metadata, including the title tag and meta description, will tend to improve the relevancy of a site's search listings, thus increasing traffic. URL canonicalization of web pages accessible via multiple URLs, using the canonical link element[49] or via 301 redirects can help make sure links to different versions of the URL all count towards the page's link popularity score.

SEM results show to a select target audience. SEO results do not. While successful SEO and SEM strategies are driven by a plan to connect with a select audience, you can only specify that target audience through SEM. Through SEM, you can (depending on the publisher) select what audiences you want to see the search results by assigning filters based on age, location, income, habits, and more. Through SEO, you cannot specifically choose who will see your search results.

What would be the purpose of/reason for moving back to a different url? If its been a few years, I’d leave it alone unless you watched everything decline since moving to the main url. Moving the forum to a new url now would probably be a bit chaotic, not only for your main url but for the forum itself…. Only reason I could imagine myself moving the forum in this scenario would be if all those links were really awful and unrelated to the url it currently sits on…
Many blogging software packages automatically nofollow user comments, but those that don't can most likely be manually edited to do this. This advice also goes for other areas of your site that may involve user-generated content, such as guest books, forums, shout-boards, referrer listings, etc. If you're willing to vouch for links added by third parties (for example, if a commenter is trusted on your site), then there's no need to use nofollow on links; however, linking to sites that Google considers spammy can affect the reputation of your own site. The Webmaster Help Center has more tips on avoiding comment spam40, for example by using CAPTCHAs and turning on comment moderation.
The audit is for all pages and not only one. What happens in the majority of the cases is that pages / posts have similarities so you can group them together. For example the pages of a website may be ok but the blog post pages may be missing titles. It’s a lot of work especially for a 500 pages website but you can start from the most important pages first and work your way to the rest
There are other parts of SEO which you should pay attention to after your audit to make sure you stay competitive. After all, the technical foundation isn't the end of the road for SEO success. It's important to pay attention to your competition's SEO activity, keep an eye on the newest search engine best practices, and maintain local SEO best practices if your business depends on customers visiting a physical address. All of these are elements of a successful SEO strategy and should be corollary to your audit and ongoing SEO maintenance.

Hiring a talented local SEO (short for Search Engine Optimization) pro is the best way to help your business grow online and get in front of the people who are looking for what you have.  Rebel Base SEO is based in Little Elm Texas. If you are not familiar with the area, that is about 20 miles north of Dallas, just a little north of The Colony, to the West of Frisco, Plano, Anna and McKinney, south of Prosper and East of Denton.  We have clients right here in Texas and as far out as California, New York, Pennsylvania and more.
However, if possible, I would like you to expand a bit on your “zombie pages” tip..we run a site where are definitely enough pages to delete (no sessions, no links, probably not even relevant with the main theme of the site, not even important for the architecture of the site)..Nonetheless, I am not very sure what is the best technical decision for these pages…just deleting them from my CMS, redirecting (if there is a relevant alternative) or something else? Unindex them on Search console? what response code they should have? ..

Hi, Brian. Thank you for the great article. I have a question about the part about 4 website addresses. Ours currently is set to https://www., and we would like to change it to just an https:// as the main website. Will this hurt our current link profile, or will everything stay the same? This might be a foolish question, but we are a bit worried. Thank you.
By relying so much on factors such as keyword density which were exclusively within a webmaster's control, early search engines suffered from abuse and ranking manipulation. To provide better results to their users, search engines had to adapt to ensure their results pages showed the most relevant search results, rather than unrelated pages stuffed with numerous keywords by unscrupulous webmasters. This meant moving away from heavy reliance on term density to a more holistic process for scoring semantic signals.[13] Since the success and popularity of a search engine is determined by its ability to produce the most relevant results to any given search, poor quality or irrelevant search results could lead users to find other search sources. Search engines responded by developing more complex ranking algorithms, taking into account additional factors that were more difficult for webmasters to manipulate. In 2005, an annual conference, AIRWeb (Adversarial Information Retrieval on the Web), was created to bring together practitioners and researchers concerned with search engine optimization and related topics.[14]
×