two: the use of dynamic web site
How to check whether the file
3. was wrong or refused to search engine spiders crawl.
Robots.txt file writing errors what harm?
: a robots.txt file writing error
Robots.txt file writing errors mainly that several aspects:
we know that only the content of your site indexed by search engines, we can go to the optimization ranking, competitive rankings, if your site is very difficult to search engine included, no website ranking naturally or rarely. So what is the reason for not being included in the search engine website? A station network – tell you website is not included in the search engine is actually because some small problems easily overlooked cause, but also can be said to be the cause of you imagined, following a hugh webmaster to help you analyze what point is your matching.
if your robots.txt file writing errors, search engine robots misunderstood your robots.txt files, they may completely ignore you ", resulting in web search engines will not be included.
for this problem, the solution is to carefully check your robots.txt file, and make sure your web page parameters is correct, you can use Google webmaster tools to create a robots.txt file, it will remind you of the website robots.txt file errors.
2.Robots.txt file writing is not standardized.
so far, the search engine (except Google) can not fully read the dynamic web pages, dynamic pages because the variable is too much, what is the address for another address open, dynamic web site also contains some other marks, etc. "
site of the robots.txt file is written we need to pay attention to, if we do not write, then we will not to move it easily, because that is the robots.txt file writing mistakes in web content is not indexed by search engines, so if we can’t write enough not to touch it. The writing rules must first be familiar with the robots.txt file before writing robots.txt files, after a need to control, to avoid mistakes, these are very necessary.
4.Robots.txt writing did not pay attention to the size of writing, it is also very important, many websites are often ignored the problem.
1. robots.txt file is corrupted or error occurred at the time of writing.