SEM results show to a select target audience. SEO results do not. While successful SEO and SEM strategies are driven by a plan to connect with a select audience, you can only specify that target audience through SEM. Through SEM, you can (depending on the publisher) select what audiences you want to see the search results by assigning filters based on age, location, income, habits, and more. Through SEO, you cannot specifically choose who will see your search results.
While backlinks are still an important factor used by search engines to determine content quality, linkless mentions are given more weight than ever before. This is partially due to the perception is that linkless mentions are more genuine, unlike black hat SEO techniques like paid links. Social media mentions are also playing an increasingly important role in evaluating website quality.
Robots.txt is not an appropriate or effective way of blocking sensitive or confidential material. It only instructs well-behaved crawlers that the pages are not for them, but it does not prevent your server from delivering those pages to a browser that requests them. One reason is that search engines could still reference the URLs you block (showing just the URL, no title or snippet) if there happen to be links to those URLs somewhere on the Internet (like referrer logs). Also, non-compliant or rogue search engines that don't acknowledge the Robots Exclusion Standard could disobey the instructions of your robots.txt. Finally, a curious user could examine the directories or subdirectories in your robots.txt file and guess the URL of the content that you don't want seen.
Both require knowing your audience. To succeed at both strategies, you must have a good understanding of your audience and how they act. By using buyer personas and psychographic segmentation, you can get to know your audience, discover what their needs are, and what they are searching for. Then you can create valuable content that shows up when they go looking for solutions related to your brand.