Content

Introductory

A .htaccess files are very useful props Apache web server, the effective use of which can solve many problems. However, in the case of a more complex server configuration, we need to pay attention to several things, otherwise we can easily generate Apache errors.

In this troubleshooter, we deal with redirects, where in certain situations we can end up in an endless redirect cycle, which eventually results in "AH00124: Request exceeded the limit of 10 internal redirects due to probable configuration error." leads to content errors.

The bug phenomenon

To the Apache error log (error.log) we find error messages similar to the following (for the sake of clarity, I have broken them into several lines):

[Tue Sep 20 22:42:28.956778 2022] [core:error] [pid 18374:tid 140539768854272] [client 116.179.37.84:30821] 
AH00124: Request exceeded the limit of 10 internal redirects due to probable configuration error. 
Use 'LimitInternalRecursion' to increase the limit if necessary. 
Use 'LogLevel debug' to get a backtrace., referer: https://www.linuxportal.info/leirasok/

This means that during an Apache setting, for example, a redirect in one of our .htaccess files caused the execution to go into an infinite loop, so Apache disabled the redirect chain at the 10 redirects specified in the configuration, and describes how to raise this limit, etc. .

Although this is not a big error, as Apache indicates in the error message that it stopped execution after 10 redirects. In such cases, the client finally receives an HTTP response code of 500, which is the error code of the "Internal Server Error" type. The above error messages - in this situation - in the Apache log (access.log) are associated with the following entries with error code 500.

116.179.37.84 - - [20/Sep/2022:16:39:50 +0200] "GET /sites/default/files/theme/linux-logo_kicsi.png HTTP/1.1" 500 5674 
"https://en.linuxportal.info/" "Mozilla/5.0 (compatible; Baiduspider-render/2.0; +http://www.baidu.com/search/spider.html)"

Here we can see that it is about the same thing from an IP address it's a robot (Baidu spider).

The reason of the error

One We blocked unwanted bots with Apache RewriteRule rule, which in certain situations can lead the web server into an endless redirection cycle.

The reason for this can be very diverse, but in this situation it is the following:

Here we used the Apache .htaccess redirect rule sample below:

RewriteCond %{HTTP_USER_AGENT} ^.*(Baiduspider|robot2|robot3|stb).*$ [NC]
RewriteRule .* - [F,L]

In short, this rule does the following:

• RewriteCond line: If the browser ID contains the names listed in brackets, then
• RewriteRule line: It returns a 403 Forbidden response and 

Used flags:

• [NC]: No case-sensitive, i.e. it ignores uppercase and lowercase letters, so you can enter the names of the robots with only lowercase letters. Source.
• [F]: Forbidden. The flag causes the server to give the client a 403 Forbidden response code. Source.
• [L]: Last. Instructs the mod_rewrite Apache module to stop executing additional rule sets in the .htaccess file. Source.

So, in principle, the "L" flag should stop the processing of further rules, but in some cases this does not happen.

Basically what happens is that if we have such a server configuration, where individual error pages are used, the system redirects the client to the corresponding error page for the given HTTP error codes. And if this setting is not set in a .htaccess file, but for example a web page Virtual hosting file, then based on the execution order and priority rules, an additional redirection to the individual error page will be executed there. 

In this case, the only problem with this is that our above RewriteRule rule gives the listed robots a 403 Forbidden header, and then stops the execution of the service. But based on the priority rules, Apache's custom error page setting comes after this, which in this case redirects you to the custom error page as a result of the 403 error code, which happens within the framework of a new request, so our .htaccess file is executed again. Since the robot is banned by the simple rule we set up, it cannot reach this unique error page either, because the RewriteRule bans it again with a 403 header. You will then be redirected to the custom error page again. This closes the loop and the execution enters an infinite loop.

Of course, this is not a problem, since Apache stops the entire process after the set limit, so the robot is eventually banned - in accordance with the original purpose - but in this case kiens - in this case the robot - receives a 500 error code, which is Internal Server Error error code. Furthermore, our error log is littered with the "AH00124: Request exceeded the limit of 10 internal redirects due to probable configuration error. " with error messages. And as we know, robots visit quite often, so quite a lot AH00124 type error message in the error log file.

That's how it is, for example ISPConfigalso in the case of server environments where the web hosts contain a /error subdirectory containing individual html pages (created by ISPConfig developers) for different error codes, such as 403.html. Then, in the page's virtualhost file, the following detail takes care of the management of error pages:

[...]
Alias /error/ "/var/www/linuxportal.info/web/error/"
ErrorDocument 400 /error/400.html
ErrorDocument 401 /error/401.html
ErrorDocument 403 /error/403.html
ErrorDocument 404 /error/404.html
ErrorDocument 405 /error/405.html
ErrorDocument 500 /error/500.html
ErrorDocument 502 /error/502.html
ErrorDocument 503 /error/503.html
[...]

So if the bots are banned with a 500 HTTP error code, it's not the end of the world, it's just a small bug that we're fixing now.

The solution

In the error message, it says that we can also increase the limit of internal redirects, but this is not a good solution, since we are dealing with an infinite loop, so no matter how much you could increase this limit, it would not solve the problem.

The correct solution is much simpler than that, which we have to look for in the cause of the error.

To break this endless cycle, it is enough to add an exception to our redirection rule, which prevents a series of redirections.

RewriteCond %{REQUEST_URI} !^/error/403\.html
RewriteCond %{HTTP_USER_AGENT} ^.*(Baiduspider|robot2|robot3|stb).*$ [NC]
RewriteRule .* - [F,L]

So the first line is the solution. What it does is check in an additional condition that the URL of the request is not the 403 unique error page itself. So it only returns the 403 Forbidden response code if the client is not on the error page itself. So with this we put an exception that does not allow the server to go into an infinite redirect loop.

If we put this condition in our .htaccess file, we will no longer receive "AH00124: Request exceeded the limit of 10 internal redirects due to probable configuration error" error messages, and clients will also receive 403 (Forbidden) error codes instead of 500 (Internal Server Error) instead.

Example:

cat access.log | grep -E " 403 | 500 "

[...]
5.255.253.183 - - [22/Sep/2022:13:58:05 +0200] "GET / HTTP/1.1" 403 6811 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
5.255.253.183 - - [22/Sep/2022:15:15:21 +0200] "GET /robots.txt HTTP/1.1" 403 6811 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
216.244.66.248 - - [22/Sep/2022:15:53:15 +0200] "GET /robots.txt HTTP/1.1" 403 6728 "-" "Mozilla/5.0 (compatible; DotBot/1.2; +https://opensiteexplorer.org/dotbot; help@moz.com)"
[...]

As I mentioned earlier, this error doesn't cause any problems, it just clutters up our error log file, so it's worth making this change.

Conclusion

This little troubleshooter is a How to keep unwanted robots away from our server websites I intend it as a supplement to my previous description, where I did not want to deviate from the main topic of the description with these details. Therefore, this information is specifically related to this specific case, however, based on this analogy, we can also fix other AH00124 type Apache errors.