Simple Way to Create a robots.txt File

Robots.txt files help you to create links in the search engines more nicely. The robots.txt file is used to a instruct search engine robots about what pages on yours website should be crawled and consequently indexed. Most of the websites have files & the folders that are not relevant for search engines (like images or admin files etc) therefore creating a robots.txt file can be actually improve your website indexation.
A robots.txt is a simple text file that can be created with a  Notepad. If you are using WordPress a sample robots.txt file would be see below:

User-agent: *
Disallow: /wp-
Disallow: /feed/
Disallow: /trackback/

@“User-agent: *” means that all the search bots (from Googles, Yahoo's, MSN and so on) should use those instructions to the crawl fo yours website. Unless yours website is complex you will not need to set different instructions for different spiders.
@“Disallow: /wp-” will make sure that the search engines will not be crawl the WordPress files. This line will exclude all files and foldes starting with “wp-” from the indexation, avoiding duplicated contents and admin files.
If you are not using WordPress just substitute to the Disallow lines with the files or folders on your website that should not be crawled, for the instance:

User-agent: *
Disallow: /images/
Disallow: /cgi-bin/
Disallow: /any other folder to be excluded/

When, After you created the robots.txt file just upload it to your roots directory and you are mostly done!
Enjoy with our articles have a good day.