Content

2. page content

Continuation

Az on the first page we performed a fresh installation and setup of the Monit system monitor application a Debian 10 (Buster) Perfect Server 1.1on the server version, and on this page One ISPConfig we perform subsequent error correction, adjustment and fine-tuning of the Monit instance already installed and configured by an automated installer.

Configuring Monit after auto-installing ISPConfig

Since here we will be dealing with a pre-installed and configured Monit system, the parameters of which do not cover all our needs, so this part - as a supplement to the previous parts - will be more of a fast-paced troubleshooting and fine-tuning part, where we will repair non-working components and we tailor the settings of Monit to our own needs on our system that is already in use. This chapter can be based on one of the following perfect server installation tutorials based on Debian 11 (Bullseye):

• Perfect server: Debian 11 (Bullseye) v1.0
• How to build and provision our ISPConfig3 server and how to secure our control panel, main services and websites with Let's Encrypt SSL

Setting up an SSL connection

The problem:

The server was successfully installed using the automated installer of ISPConfig, where Monit was also set as an option, so it was also installed. Arming the server (domain name after redirection) asked for Let's Encrypt After installing an SSL certificate, the server's web services are only available HTTPS can be reached via connection. However, Monit's port 2812 does not allow it by default SSL connection, therefore when accessing the Monit interface, it throws an error:

"ERR_SSL_PROTOCOL_ERROR"

The solution:

In the first part of the description, we have already reviewed the SSL setup, so to understand the operation let's review the previous parts, then return here and do the following:

Open the configuration of the web interface for editing:

nano /etc/monit/conf-available/webui

(Unlike in the fresh installation section, the Monit web settings installed by ISPConfig are in this file.)

set httpd port 2812 and
    #SSL ENABLE
    #PEMFILE /usr/local/ispconfig/interface/ssl/ispserver.pem
    allow admin:xxx

Remove the comment from the red spot at the beginning of line 2 and add the extra line highlighted in green so it looks like this:

set httpd port 2812 and
    #SSL ENABLE
    #PEMFILE /usr/local/ispconfig/interface/ssl/ispserver.pem
    PEMFILE /etc/monit/ssl/ispserver.pem
    allow admin:xxx (itt a saját jelszavunk van, amit a telepítő adott a végén)

Save it, then execute the following commands in sequence:

mkdir /etc/monit/ssl
cd /etc/monit/ssl
cp /usr/local/ispconfig/interface/ssl/ispserver.pem .
chmod 700 ispserver.pem
monit reload

If this Monit reload throws the following error:

checksum: file /etc/ssh/ssh_host_dsa_key is not regular file
/etc/monit/conf-enabled/openssh-server:27: Cannot compute a checksum for file /etc/ssh/ssh_host_dsa_key 'sshd'

Then let's jump for this part to fix the error, then continue here.

Then update the Monit web interface:

Then after logging in:

And the valid certificate:

Let's Encrypt ACME your client renews its SSL certificates every 3 months at the latest, so this file should be copied and the permissions set every time. At the end of the description in the appendix we create an automated solution for this to do it yourself. Now we move on to the next task.

Setting a custom SSH port

The problem:

Even with the previous SSL solution, it can be seen that the SSHd process is not monitored by the system. Entering the sshd details, we see the following:

The gist can be read in the lower gray line:

If failed [localhost]:22 type TCP/IP protocol SSH with timeout 5 s then restart

The reason for the problem is that after installing the server we set a unique port number for SSHD, therefore, Monit cannot connect to it, therefore, according to the set rules, it turns off its monitoring after the 5th control cycle. When installing ISPConfig automatically, if is stricter SSH settings, it also stops the SSHd service in this case, so the system throws it out of the terminals as well. If this happens, you need to restart/start the SSHd service on the web interface and temporarily stop the monitoring manually while we perform the following solution here. After that, you can turn on the monitoring of the service again.

The solution:

We have already reviewed it here previously the option to set the individual SSH port.

Accordingly, open the following configuration file:

nano /etc/monit/conf-available/sshd

Here we find a line like this:

[...]
if failed host localhost port 22 with proto ssh then restart
[...]

Here, change the port number 22 to our unique port number, save and then reload the Monit configuration:

monit reload

Then click on the "Enable monitoring" button at the bottom of the SSHD details page on the web interface. Finally, returning to the main page, SSHd monitoring is already working:

Fine tuning of resource limits

The problem:

Monit reaches the limits too often, or vice versa: the system does not even indicate when it already shows extremely high values.

The solution:

By default, the resource limits were set to general values ​​that do not match the resources of our own server. To solve this, we need to fine-tune the limits so that they fit our own server. On doing this, a it was already mentioned in the previous part of the description, so by jumping there we can find out how it is implemented.

Monit cycle setting

Monit has an update cycle, which defaults to 2 minutes for fresh installations, but is overridden to 60 seconds during the automatic installation of the ISPConfig control panel. If you want to set it more densely or less frequently, open Monit's main configuration file:

nano /etc/monit/monitrc

And right at the beginning we find a line like this:

  set daemon 120            # check services at 2-minute intervals

In case of automated installation, we can see a value of 60 here. Set this to the desired update interval.

If you have set the correct value, reload the configuration:

monit reload

Web interface unique port number setting

In a production environment, the need for a higher degree of security may arise. We can change the access port number of Monit's web interface from the default port of 2812 to a unique value, making it much more difficult for outsiders to find our Monit system. To set this up let's jump back to the first part of the description, where I explained this in more detail. So, taking into account what is written there, let's set a port number that is not yet in use in our system. You can find information about the detection of these unused port numbers on the following links:

• How can we check the used and free TCP/UDP ports of our Debian or Ubuntu Linux operating system?
• Basic port scanning - Or how to check the security of our Linux server against unauthorized intruders

Setting up email notifications

The problem:

During automatic installation, the when compiling the installation command we forgot to put the --monit-alert-email switch, so now we don't get notifications when different Monit events occur.

The solution:

In the first part of the description, we already talked about setting up email notifications, like this we can see it there. The difference here is that in the case of the automated ISPConfig installation, the previously mentioned file already exists, as well as other commented templates, with which you don't have to do anything, just rewrite your email address and turn on the setting as usual.

If we did everything right, we will receive emails like this, for example, if the CPU usage has reached the value set in the limit:

Resource limit matched Service vps

    Date:        Mon, 23 Jan 2023 00:05:55
    Action:      alert
    Host:        vps
    Description: cpu system usage of 20.7% matches resource limit [cpu system usage > 20.0%]

Your faithful employee,
Monit

Then, as soon as the load has crept back below the limit, something like this arrives:

Resource limit succeeded Service vps

    Date:        Mon, 23 Jan 2023 00:08:13
    Action:      alert
    Host:        vps
    Description: cpu system usage check succeeded [current cpu system usage = 3.5%]

Your faithful employee,
Monit

And it sends similar letters for the other monitored services as well.

That's about all there is to set here, I'll expand it later if I think of any useful settings.

Appendix

In the following sections, we will add a few useful things to what we learned earlier.

Checking statuses from the command line

Monit provides an opportunity from the command line also view status information. For this a monit command must be run with the appropriate switches.

overall

If you just want to get a quick summary of the monitored services, run the following command:

monit summary

Example output for me:

Monit 5.27.2 uptime: 8d 23h 40m
┌─────────────────────────────────┬────────────────────────────┬───────────────┐
│ Service Name                    │ Status                     │ Type          │
├─────────────────────────────────┼────────────────────────────┼───────────────┤
│ vps                             │ OK                         │ System        │
├─────────────────────────────────┼────────────────────────────┼───────────────┤
│ sshd                            │ OK                         │ Process       │
├─────────────────────────────────┼────────────────────────────┼───────────────┤
│ rspamd                          │ OK                         │ Process       │
├─────────────────────────────────┼────────────────────────────┼───────────────┤
│ redis-server                    │ OK                         │ Process       │
├─────────────────────────────────┼────────────────────────────┼───────────────┤
│ pure-ftpd-mysql                 │ OK                         │ Process       │
├─────────────────────────────────┼────────────────────────────┼───────────────┤
│ postfix                         │ OK                         │ Process       │
├─────────────────────────────────┼────────────────────────────┼───────────────┤
│ php8.1-fpm                      │ OK                         │ Process       │
├─────────────────────────────────┼────────────────────────────┼───────────────┤
│ php8.0-fpm                      │ OK                         │ Process       │
├─────────────────────────────────┼────────────────────────────┼───────────────┤
│ php7.4-fpm                      │ OK                         │ Process       │
├─────────────────────────────────┼────────────────────────────┼───────────────┤
│ php7.3-fpm                      │ OK                         │ Process       │
├─────────────────────────────────┼────────────────────────────┼───────────────┤
│ php7.2-fpm                      │ OK                         │ Process       │
├─────────────────────────────────┼────────────────────────────┼───────────────┤
│ php7.1-fpm                      │ OK                         │ Process       │
├─────────────────────────────────┼────────────────────────────┼───────────────┤
│ php7.0-fpm                      │ OK                         │ Process       │
├─────────────────────────────────┼────────────────────────────┼───────────────┤
│ php5.6-fpm                      │ OK                         │ Process       │
├─────────────────────────────────┼────────────────────────────┼───────────────┤
│ named                           │ OK                         │ Process       │
├─────────────────────────────────┼────────────────────────────┼───────────────┤
│ memcached                       │ OK                         │ Process       │
├─────────────────────────────────┼────────────────────────────┼───────────────┤
│ mariadb                         │ OK                         │ Process       │
├─────────────────────────────────┼────────────────────────────┼───────────────┤
│ fail2ban                        │ OK                         │ Process       │
├─────────────────────────────────┼────────────────────────────┼───────────────┤
│ dovecot                         │ OK                         │ Process       │
├─────────────────────────────────┼────────────────────────────┼───────────────┤
│ crond                           │ OK                         │ Process       │
├─────────────────────────────────┼────────────────────────────┼───────────────┤
│ apache                          │ OK                         │ Process       │
├─────────────────────────────────┼────────────────────────────┼───────────────┤
│ sshd_bin                        │ OK                         │ File          │
├─────────────────────────────────┼────────────────────────────┼───────────────┤
│ sftp_bin                        │ OK                         │ File          │
├─────────────────────────────────┼────────────────────────────┼───────────────┤
│ sshd_rsa_key                    │ OK                         │ File          │
├─────────────────────────────────┼────────────────────────────┼───────────────┤
│ sshd_dsa_key                    │ OK                         │ File          │
├─────────────────────────────────┼────────────────────────────┼───────────────┤
│ sshd_rc                         │ OK                         │ File          │
├─────────────────────────────────┼────────────────────────────┼───────────────┤
│ master_bin                      │ OK                         │ File          │
├─────────────────────────────────┼────────────────────────────┼───────────────┤
│ postdrop_bin                    │ OK                         │ File          │
├─────────────────────────────────┼────────────────────────────┼───────────────┤
│ postqueue_bin                   │ OK                         │ File          │
├─────────────────────────────────┼────────────────────────────┼───────────────┤
│ master_cf                       │ OK                         │ File          │
├─────────────────────────────────┼────────────────────────────┼───────────────┤
│ main_cf                         │ OK                         │ File          │
├─────────────────────────────────┼────────────────────────────┼───────────────┤
│ postfix_rc                      │ OK                         │ File          │
├─────────────────────────────────┼────────────────────────────┼───────────────┤
│ memcache_bin                    │ OK                         │ File          │
├─────────────────────────────────┼────────────────────────────┼───────────────┤
│ memcache_rc                     │ OK                         │ File          │
├─────────────────────────────────┼────────────────────────────┼───────────────┤
│ mysql_bin                       │ OK                         │ File          │
├─────────────────────────────────┼────────────────────────────┼───────────────┤
│ mysql_rc                        │ OK                         │ File          │
├─────────────────────────────────┼────────────────────────────┼───────────────┤
│ fail2ban_log                    │ OK                         │ File          │
├─────────────────────────────────┼────────────────────────────┼───────────────┤
│ cron_bin                        │ OK                         │ File          │
├─────────────────────────────────┼────────────────────────────┼───────────────┤
│ cron_rc                         │ OK                         │ File          │
├─────────────────────────────────┼────────────────────────────┼───────────────┤
│ apache_bin                      │ OK                         │ File          │
├─────────────────────────────────┼────────────────────────────┼───────────────┤
│ apache_rc                       │ OK                         │ File          │
├─────────────────────────────────┼────────────────────────────┼───────────────┤
│ cron_spool                      │ OK                         │ Directory     │
├─────────────────────────────────┼────────────────────────────┼───────────────┤
│ rootfs                          │ OK                         │ Filesystem    │
└─────────────────────────────────┴────────────────────────────┴───────────────┘

Status report

And if we are interested in a detailed status report, then run the following command:

monit status

This is quite a long list, so I won't copy it now, but you can query the specific processes and resources separately, for example:

monit status apache

Monit 5.27.2 uptime: 8d 23h 50m

Process 'apache'
  status                       OK
  monitoring status            Monitored
  monitoring mode              active
  on reboot                    start
  pid                          1453602
  parent pid                   1
  uid                          0
  effective uid                0
  gid                          0
  uptime                       3d 15h 7m
  threads                      1
  children                     10
  cpu                          0.0%
  cpu total                    0.1%
  memory                       0.5% [19.9 MB]
  memory total                 4.7% [185.7 MB]
  security attribute           unconfined
  filedescriptors              19 [0.2% of 8192 limit]
  total filedescriptors        215
  read bytes                   0 B/s [468.9 MB total]
  disk read bytes              0 B/s [23.2 MB total]
  disk read operations         0.0 reads/s [1116138 reads total]
  write bytes                  0 B/s [85.9 MB total]
  disk write bytes             0 B/s [60.9 MB total]
  disk write operations        0.0 writes/s [291832 writes total]
  port response time           0.413 ms to localhost:443 type TCP/IP protocol DEFAULT
  port response time           4.689 ms to localhost:80 type TCP/IP protocol HTTP
  data collected               Tue, 24 Jan 2023 10:41:51

Automatically update SSL on renewal

The ACME client operating in the system - counting from its last renewal - cyclically renews the SSL certificates within 3 months at the latest, therefore the PEM file copied by hand earlier in this description must also be updated and its authorization must be reset. There are several solutions for this, which we will now look at couple, and let's create our own automation with whichever one is more sympathetic.

Also, for the time being, I have only developed one here, which I have tested (on Debian 10 and Debian 11) and it works flawlessly, but I will make up for this part later, and plan one or two other solutions as soon as I have time.

Update using Incron

Taking note of the above, if we want to configure the updating of our SSL file using incron, then continue with what is described here.

In the case of Perfect servers, the backports containers are already configured, so there is nothing to do with this, just install the incron package.

apt-get install incron

Then create a command file next to, say, the ssl file (we don't need to search here):

nano /etc/monit/ssl/monit_ssl_refresh.sh

And let's include the following:

#!/bin/sh
cp /usr/local/ispconfig/interface/ssl/ispserver.pem /etc/monit/ssl/
chmod 700 /etc/monit/ssl/ispserver.pem
monit reload

Here we copy the .pem file, then set its authority to 700, and finally reload the Monit configuration so that it is already loaded with this SSL.

Save the file and make it executable:

chmod +x /etc/monit/ssl/monit_ssl_refresh.sh

Then add the root user to the authorized users of incron, because without it root cannot use it either:

echo "root" >> /etc/incron.allow

Then it will incrontab edit the incrontab command:

incrontab -e

Add the following line:

/usr/local/ispconfig/interface/ssl/ispserver.pem IN_MODIFY /etc/monit/ssl/monit_ssl_refresh.sh

And from here on, our script runs when the original file changes.

Conclusion

So that would be the operation of the Monit service and system resource monitoring application, which we could see in the form of installation and configuration from scratch and also in the form already prepared by the automated installer of ISPConfig, which we only had to fine-tune. I hope that this description will be useful to many people, and that Monit will serve you well in the operation of your server.

Navigation