Many times we have clients contacting us because they have an unusual bandwidth usage, many times the used bandwidth is cause normals things like, the client advertised the site or they worked in the site SEO and the site has more visits now, but some times the traffic increase is cause a bad robots scanning the site, in most small-medium sites this do not use to much traffic because most probably the site do not have too much information to crawl but when you have a site with a lot of images or videos the bandwidth used to crawl the site can be a lot.
This article is to give you some tools and information so you can evaluate and apply any of the techniques to block bad robots in your account.
The following links explain and give you some examples about the robots.txt file to block the site indexing.
Here you can find an Apache project with .htaccess rules that you can add in your .htaccess file to block many bad robots.
- 0 Users Found This Useful
How can i change the MySQL characters encoding to UTF8?
By default the encoding is latin1 but you could try to workaround your code. To change the...Oscommerce is showing a blank page in the categories pages
If you made no changes in your account or oscommerce script and this happened suddenly then there...I have the register_globals disabled server wide, how can i enable it in my account?
This option is available in all our shared cPanel hosting accounts.Please note if your script is...How could i change the PHP encoding in my scripts?
You could try changing it sending the headers in your php code, i.e.: header('Vary:...How do i parse .html files as .shtml (SSI)?
You can add the following line in your .htaccess file: AddHandler server-parsed .html or...