Content

Introductory

The more complex our server, the more likely it is that something will crash on a causal basis and cause insidious errors that are difficult to unravel. This is how it happened to me the other day, when I could not access the websites and web applications on my server overnight. The browser wrote "ERR_TIMED_OUT" in this case, which is not a very rare error and can be caused by several things, making it harder to identify the cause.

In this troubleshooter, we roll up a Fail2Ban block caused by an Apache error of type "AH01630: client denied by server configuration" caused by Matomo's web analytics system. Finally, we fix the problem by modifying the Fail2Ban filter in apache-auth.

The symptom

As I mentioned in the introduction, the web pages and other web applications on the server are not accessible and the browser displays the error message "ERR_TIMED_OUT". In addition, all other services are available, such as SSH contact, correspondence (IMAP) or that FTP, etc.

The error was on the wired internet in Hungary, but everything worked on mobilnet. Restarting the wired internet modem temporarily fixed the problem, and the pages became unavailable again after a few minutes. I repeated all this several times, and after random times, the problem reappeared. It all seemed pretty weird since the server was accessible via SSH, so I started researching. I also asked the service provider, but there was no maintenance or network problem with the server, so I started looking around the server.

Find the cause of the error

It occurred to me that maybe the Fail2ban may have banned it My IP address, so I checked your Fail2ban log file:

cat /var/log/fail2ban.log | grep "<saját ip-cím>" | tail -40

[...]
2022-05-18 14:43:53,508 fail2ban.filter         [1781]: INFO    [apache-auth] Found 85.238.74.10 - 2022-05-18 14:43:53
2022-05-18 15:10:23,905 fail2ban.filter         [1781]: INFO    [apache-auth] Found 85.238.74.10 - 2022-05-18 15:10:23
2022-05-18 15:10:24,067 fail2ban.actions        [1781]: NOTICE  [apache-auth] Ban 85.238.74.10
2022-05-18 15:10:26,821 fail2ban.actions        [1781]: NOTICE  [apache-auth] Unban 85.238.74.10
2022-05-18 15:10:29,840 fail2ban.filter         [1781]: INFO    [apache-auth] Found 85.238.74.10 - 2022-05-18 15:10:29
2022-05-18 15:11:21,488 fail2ban.filter         [1781]: INFO    [apache-auth] Found 85.238.74.10 - 2022-05-18 15:11:21
2022-05-18 15:11:22,158 fail2ban.actions        [1781]: NOTICE  [apache-auth] Ban 85.238.74.10
2022-05-18 15:11:24,443 fail2ban.actions        [1781]: NOTICE  [apache-auth] Unban 85.238.74.10
2022-05-18 15:20:53,763 fail2ban.filter         [1781]: INFO    [apache-auth] Found 85.238.74.10 - 2022-05-18 15:20:53
2022-05-18 15:20:59,671 fail2ban.filter         [1781]: INFO    [apache-auth] Found 85.238.74.10 - 2022-05-18 15:20:59
2022-05-18 15:21:00,081 fail2ban.actions        [1781]: NOTICE  [apache-auth] Ban 85.238.74.10
2022-05-18 15:21:02,540 fail2ban.actions        [1781]: NOTICE  [apache-auth] Unban 85.238.74.10

My tip came in, so bans and disqualifications came up here. Then, following the "apache-auth" jail recorded in the log, we also look at the filter for the jail:

nano /etc/fail2ban/filter.d/apache-auth.conf

# Fail2Ban apache-auth filter
#

[INCLUDES]

# Read common prefixes. If any customizations available -- read them from
# apache-common.local
before = apache-common.conf

[Definition]

prefregex = ^%(_apache_error_client)s (?:AH\d+: )?<F-CONTENT>.+</F-CONTENT>$

# auth_type = ((?:Digest|Basic): )?
auth_type = ([A-Z]\w+: )?

failregex = ^client (?:denied by server configuration|used wrong authentication scheme)\b
            ^user <F-USER>(?:\S*|.*?)</F-USER> (?:auth(?:oriz|entic)ation failure|not found|denied by provider)\b
            ^Authorization of user <F-USER>(?:\S*|.*?)</F-USER> to access .*? failed\b
            ^%(auth_type)suser <F-USER>(?:\S*|.*?)</F-USER>: password mismatch\b
            ^%(auth_type)suser `<F-USER>(?:[^']*|.*?)</F-USER>' in realm `.+' (not found|denied by provider)\b
            ^%(auth_type)sinvalid nonce .* received - length is not\b
            ^%(auth_type)srealm mismatch - got `(?:[^']*|.*?)' but expected\b
            ^%(auth_type)sunknown algorithm `(?:[^']*|.*?)' received\b
            ^invalid qop `(?:[^']*|.*?)' received\b
            ^%(auth_type)sinvalid nonce .*? received - user attempted time travel\b

ignoreregex = 

# DEV Notes:
#
# This filter matches the authorization failures of Apache. It takes the log messages
# from the modules in aaa that return HTTP_UNAUTHORIZED, HTTP_METHOD_NOT_ALLOWED or
# HTTP_FORBIDDEN and not AUTH_GENERAL_ERROR or HTTP_INTERNAL_SERVER_ERROR.

[...]

This filter is With HTTP authentication is watching for errors, so we need to look for such messages in Apache in our error log files listed under "failregex".

If a simpler LAMP server, the Apache error log entries are set to /var/log/apache2/error.log we need to search the file if we have not set a unique path. In this case, look for the error log file that matches your configuration.

And if ISPConfigIf we use a server environment, it is likely that there are several web pages running on the server, which are collected in separate files for Apache errors that occur at runtime. In the present case, let us now consider this.

When you install ISPConfig, you reconfigure many things, including access to the Apache error logs, which can be accessed in the following structure:

/var/log/ispconfig/httpd/

Then during this, each web host has a subdirectory and that is where Apache is located access.log and that error.log file as well. We need to go through these to find out which web host the error is related to and what is causing it.

If we have multiple websites, the easiest one is one grep using the command, own By filtering to our IP address, we can find the source of the error:

grep -rw --include="error.log" "<saját IP-cím>" /var/log/ispconfig/httpd | tail -12

A -r option provides recursive search, -w and uses full-word search to return only the individual IP address patterns.

/var/log/ispconfig/httpd/linuxportal.info/error.log:[Wed May 18 18:39:23.912143 2022] [authz_core:error] [pid 31857:tid 140418732259072] [client 85.238.74.10:62050] AH01630: client denied by server configuration: /opt/matomo/plugins/CoreHome/javascripts/manifest.json, referer: https://www.linuxportal.info/matomo/index.php?module=CoreHome&action=index&idSite=1&period=day&date=today
/var/log/ispconfig/httpd/linuxportal.info/error.log:[Wed May 18 18:39:48.519643 2022] [authz_core:error] [pid 31857:tid 140418749044480] [client 85.238.74.10:62080] AH01630: client denied by server configuration: /opt/matomo/plugins/CoreHome/javascripts/manifest.json, referer: https://www.linuxportal.info/matomo/index.php?module=CoreHome&action=index&idSite=1&period=day&date=today
/var/log/ispconfig/httpd/linuxportal.info/error.log:[Wed May 18 18:57:54.471065 2022] [authz_core:error] [pid 31858:tid 140418732259072] [client 85.238.74.10:65007] AH01630: client denied by server configuration: /opt/matomo/plugins/CoreHome/javascripts/manifest.json, referer: https://www.linuxportal.info/matomo/index.php?module=CoreHome&action=index&idSite=1&period=day&date=today
/var/log/ispconfig/httpd/linuxportal.info/error.log:[Wed May 18 19:02:51.688512 2022] [authz_core:error] [pid 31858:tid 140418639939328] [client 85.238.74.10:49443] AH01630: client denied by server configuration: /opt/matomo/plugins/CoreHome/javascripts/manifest.json, referer: https://www.linuxportal.info/matomo/index.php?module=CoreHome&action=index&idSite=1&period=day&date=today

And there is the cause of the error. So I have a Matomo web analytics system one of the elements is the source of the tangles, which causes an "AH01630: client denied by server configuration" error and therefore disables the IP address if the specified number of errors accumulates within the specified time frame.

The file that is the subject of the error query is inside the Matomo installation directory: /opt/matomo/plugins/CoreHome/javascripts/manifest.json.

This is not linked to a specific host, but the Matomo system is configured as an Apache config so it can be accessed from any web page on the server. I usually use this under linuxportal.info.

Now that we have the source of the error, all we have to do is find a solution.

The solution

The error I just discovered occurred to me about a day ago, which coincides with the last update of Matomo. From this, I conclude that there may have been a bug in Matomo's software that could be debuged, but many thousands of lines PHP bugging the code is not our job, but hopefully it will be fixed in a later update. I base all of this on the fact that I have been using the Matomo web analytics system for years - which has worked excellently all along - and there were no problems with the Fail2ban security software as part of the ISPConfig server environment. I have had this problem since the Matomo update that arrived a few days ago.

What we can do in this case is to modify conditions outside of Matomo's software so that the error is cleared. There are several solutions for this. For example, we could disable apache-auth jail or relax the filtering conditions, but I don't recommend these because they pose a security risk. In Fail2, its jails and filters do a great job, so it's a good idea to take a closer look at and modify the filter by inserting an exception.

Before you touch the filters, test them, let's see fail2ban-regex how many hits to date:

sudo fail2ban-regex /var/log/<saját hibanapló fájlunk útvonala> /etc/fail2ban/filter.d/apache-auth.conf

So here everyone uses the path to their own error log file.

Here are 91 results for your "AH01630: client denied by server configuration" errors found in today's error log file. Of course, not all of these are errors, because there may be other HTTP errors from other IP addresses, but the java part of this is ours. Then change the filter.

I have already inserted an excerpt from the contents of the apache-auth.conf file above, now let's reopen it:

sudo nano /etc/fail2ban/filter.d/apache-auth.conf

Then we will find such a part here:

failregex = ^client (?:denied by server configuration|used wrong authentication scheme)\b
            ^user <F-USER>(?:\S*|.*?)</F-USER> (?:auth(?:oriz|entic)ation failure|not found|denied by provider)\b
            ^Authorization of user <F-USER>(?:\S*|.*?)</F-USER> to access .*? failed\b
            ^%(auth_type)suser <F-USER>(?:\S*|.*?)</F-USER>: password mismatch\b
            ^%(auth_type)suser `<F-USER>(?:[^']*|.*?)</F-USER>' in realm `.+' (not found|denied by provider)\b
            ^%(auth_type)sinvalid nonce .* received - length is not\b
            ^%(auth_type)srealm mismatch - got `(?:[^']*|.*?)' but expected\b
            ^%(auth_type)sunknown algorithm `(?:[^']*|.*?)' received\b
            ^invalid qop `(?:[^']*|.*?)' received\b
            ^%(auth_type)sinvalid nonce .*? received - user attempted time travel\b

ignoreregex = 

In this ignoreregex =, enter the file name found in the error log, which in my case is as follows:

/opt/matomo/plugins/CoreHome/javascripts/manifest.json

failregex = ^client (?:denied by server configuration|used wrong authentication scheme)\b
            ^user <F-USER>(?:\S*|.*?)</F-USER> (?:auth(?:oriz|entic)ation failure|not found|denied by provider)\b
            ^Authorization of user <F-USER>(?:\S*|.*?)</F-USER> to access .*? failed\b
            ^%(auth_type)suser <F-USER>(?:\S*|.*?)</F-USER>: password mismatch\b
            ^%(auth_type)suser `<F-USER>(?:[^']*|.*?)</F-USER>' in realm `.+' (not found|denied by provider)\b
            ^%(auth_type)sinvalid nonce .* received - length is not\b
            ^%(auth_type)srealm mismatch - got `(?:[^']*|.*?)' but expected\b
            ^%(auth_type)sunknown algorithm `(?:[^']*|.*?)' received\b
            ^invalid qop `(?:[^']*|.*?)' received\b
            ^%(auth_type)sinvalid nonce .*? received - user attempted time travel\b

ignoreregex = /opt/matomo/plugins/CoreHome/javascripts/manifest.json

Save and run another filter test:

sudo fail2ban-regex /var/log/<saját hibanapló fájlunk útvonala> /etc/fail2ban/filter.d/apache-auth.conf

Here's the difference. There are only 19 "AH01630: client denied by server configuration" errors found below, and the Ignoreregex section ignores 72 "/opt/matomo/plugins/CoreHome/javascripts/manifest.json" files. outside.

So we've achieved our goal: the apache-auth filter will no longer take into account this "AH01630: client denied by server configuration" error generated by Matomo. Of course, the bug will still be there, which will hopefully be fixed at Matomo, but it will no longer disable our IP address in Fail2. That's all we can do right now.

When you're done, restart Fail2 in:

sudo systemctl restart fail2ban.service

From now on, we no longer have to worry about our IP address being blocked again due to what we think is an error in Matomo.

Interesting Facts

I will mention a few more things just to see if they are useful information for others.

For me, the Chrome browser is the primary one, of which I use multiple instances of windows on multiple monitors and virtual desktops. Sometimes I run Matomo in multiple instances so I can see it better on multiple desktops. When the framework loads, an error occurs. So when refreshing. In another window, meanwhile, pay attention to the tail command when error messages and disablements come.

There is no problem with the Chrome incognito window. In the plain version, I have already deleted all cookies and history, but the error still occurs. There is also no error in Firefox. So this error may be related in part to an internal Chrome error. Of course, this is hard to tell, but the two together (Chrome + Matomo) generate an HTTP authentication error, which is interesting.

Conclusion

With that, we solved another oddity. The whole thing is quite simple in retrospect, but in the beginning a lot of time goes by getting to know the problem. I made the derivation so detailed because it is not exactly the same combination that causes an error, but something else, but by analogy it is easier to find out and create a unique solution for it. Of course, with this implementation - in this particular case - we did not eliminate the actual error itself, but we achieved our goal of not being banned from our own server.

This little troubleshooter can be a useful example of other cases where you might want to start a scan around Fail2Ban to see if it has banned you from a server service.