bookmark_borderMonit: An easy to use monitoring solution for Linux

Besides Nagios, Icinga and check_mk there are some other, more slimmer tools to monitor servers. Especially if you are a private person and you want to monitor your vServer, Raspberry Pi or whatever, you may want to use a smaller and easier monitoring solution than those big three.
This article is about monit. Monit is one of these simple monitoring tool. But just because the configuration is more simple, that doesn’t mean hat you are limited in the ways you monitor your servers with Monit.

How does monit works? And why not just use Nagios?

monit works differently than Nagios or Icinga. While Nagios, Icinga and check_mk needs a monitoring server which connects in given time periods to the machines it checks, monit doesn’t need this kind of server in order to do these checks. So basically that means that wherever you install monit, it checks locally and reports the results via mail.
In the first moment this sounds like a disadvantage whether it be due to reliability or security, but think about it for a second: What does Nagios, Icinga or check_mk basically do? They ping the destination machine and if this ping is successful it opens up an NRPE or SSH connection and executes the given CPU, Ram, service checks and so on, locally. The console output is then used for analysis. If the check exceeds it’s given limit, Nagios / Iciniga / check_mk creates an E-Mail and send it to the user /group of users which is / are responsible for this service. So basically the server executes local commands like monit. The initiator however is not locally, it’s a different (remote) server.
You now may think „but how do I even test if the server is available?“. Well, you just setup another monit instance which is checking if the destination is pingable, just like with Nagios / Icinga / check_mk.
A real benefit for monit is it’s easy configuration and syntax. While you have to dig through a bunch of files in order to create a simple check in Nagios, monit allows you to simply put one (human readable) configuration file in the correct directory and the check is ready to use. In combination with a configuration management like Ansible (I’ve already created an article about Ansible a long time ago) you have an easy central to use monitoring tool with automatically distributed configuration files.

Installation of monit

Monit is available for most Linux distribution. With the following command you install monit on a Ubuntu / Debian machine:

user@machine:~$ sudo apt-get update && sudo apt-get install monit

Or if you’re an openSUSE user, you can install monit like this:

user@machine:~$ sudo zypper ref && sudo zypper in monit

That’s basically it. Compared to Nagios or Icinga, you don’t need an installed Apache web server. Monit comes with it’s own built-in web service. In order to check if monit is running, you can use systemctl to do so:

user@machine:~$ sudo systemctl status monit
● monit.service - LSB: service and resource monitoring daemon
 Loaded: loaded (/etc/init.d/monit; generated; vendor preset: enabled)
 Active: active (running) since Wed 2017-10-11 11:47:39 CEST; 3 months 13 days ago
 Docs: man:systemd-sysv-generator(8)
 Tasks: 2 (limit: 4915)
 CGroup: /system.slice/monit.service
 └─18228 /usr/bin/monit -c /etc/monit/monitrc
Dez 26 06:25:02 machine systemd[1]: Reloading LSB: service and resource monitoring daemon.
Dez 26 06:25:02 machine monit[15742]: Reloading daemon monitor configuration: monit.
Dez 26 06:25:02 machine systemd[1]: Reloaded LSB: service and resource monitoring daemon.
Jan 10 06:25:02 machine systemd[1]: Reloading LSB: service and resource monitoring daemon.
Jan 10 06:25:02 machine monit[15692]: Reloading daemon monitor configuration: monit.
Jan 10 06:25:02 machine systemd[1]: Reloaded LSB: service and resource monitoring daemon.

In order to start, stop or restart the service, use these following commands:

user@machine:~$ sudo systemctl start monit
user@machine:~$ sudo systemctl stop monit
user@machine:~$ sudo systemctl restart monit

Enable the monit built-in web server

If you’re an Debian / Ubuntu user, monit will store it’s main configuration under /etc/monit/monitrc. This files is really well documented. However, I want to give a short brief about how to enable the built-in web server which is one of the most important features.
You can find the following block (at the time of writing) at line 155:

# set httpd port 2812 and
# use address localhost # only accept connection from localhost
# allow localhost # allow localhost to connect to the server and
# allow admin:monit # require user 'admin' with password 'monit

If you uncomment this section the built-in monit web server will be started and will listen on port 2812. The second line sets the network address where the web server instance is listening on. If it only listens on localhost you will not be able to connect from any other host. For e.g. if your server has the IP address 192.168.0.2 you can replace localhost with this IP address. This will get the monit built-in web server to listen on this network interface.
The third line regulates which other hosts are allowed to establish a connection to the web server. You can define single IP addresses or even whole subnets (like 192.168.1.0/24). Just like before, simply replace localhost with the IP address or subnet which you want to allow to connect to the web server.
The fourth line defines a user with a password which will be able to login. monit does not allow an anonymous login which is good from a security point of view obviously. In this example the username would be admin with the password monit.
When you’re done setting these lines, restart the monit service:

user@machine:~$ sudo systemctl restart monit

You can now visit the monit web interface of your server. In this example the address would be something like http://192.168.0.2:2812 or just use the FQDN: http://my.monit.fqdn:2812.
Besides using the web interface, you can also check your services on the command line. To do so you also have to have the web interface enabled, because the command line simply gets an HTTP output and process it. To get a summary of all your monitored processes you can enter the following command:

user@machine:~$ sudo monit status

Enable mail notifications

Nagios or Icinga can send you a mail every time when a host or a service of this host went down. monit also offers this feature. To enable mail notifications, open once more the monit configuration file (/etc/monit/monitrc) and search for the following lines:

## Set the list of mail servers for alert delivery. Multiple servers may be
## specified using a comma separator. If the first mail server fails, Monit
# will use the second mail server in the list and so on. By default Monit uses
# port 25 - it is possible to override this with the PORT option.
#
# set mailserver mail.bar.baz, # primary mailserver
# backup.bar.baz port 10025, # backup mailserver on port 10025
# localhost # fallback rela

As you can see, you can set multiple mail servers which monit should try to connect each time a notification is about to be send. The downside here is that monit does not support authentication against mail servers. This basically means that you can’t just enter public mail servers like GMail. If you want to use those service you have to have a local mail server setup. You can configure the mailserver as a satellite / client to keep it more simple. If you’re mail server is up and running, you can simply set the mail server in the monitrc file like this:

set mailserver localhost

If you have access to a mail server which doesn’t require authentication, you can replace localhost with this mail server of course.
Now that you’ve set the mail server, you also have to enter the recipient for the notification mails. Search for the following line in the monitrc file:

# set alert sysadm@foo.bar # receive all alerts

Uncomment this line and replace sysadm@foo.bar with the recipient of your choice. You can also define which kind of notification are going to be send to this mail address. However, this is more advanced configuration which would burst this articles length.
When everything set, restart monit and you will now get a mail each time a service status changes:

user@machine:~$ sudo systemctl restart monit

Some examples for monitoring services

For monitoring new programs or services, you have to create a new file under the subdirectory /etc/monit/monitrc.d/. Simply name this file like the service which is going to be monitored. The following examples are showing how to monitor services, programs and system resources with monit:

Monitoring a running program / service or service without a PID file
check process TeamSpeak
  matching "ts3"

In this example we simple check if a process called „ts3“ is running on the machine. You can use ps ax on the command line by yourself on the target machine to see if the process is running. That’s basically what monit is doing. The name of this check is TeamSpeak as seen in the first line. You can change this to whatever you want.

Monitoring a program / service with a PID file
check process mysqld with pidfile /var/run/mysqld/mysqld.pid
  start program = "systemctl start mysql"
  stop program = "systemctl stop mysql"
  if failed host localhost port 3306 protocol mysql with timeout 15 seconds for 3 times within 4 cycles then restart
  if failed unixsocket /var/run/mysqld/mysqld.sock protocol mysql for 3 times within 4 cycles then restart
  if 5 restarts with 5 cycles then alert

In this specific example we monitor a MySQL instance installed on the system. If the service crashes, monit tries three times within four cycles to restart the service. For this restart monit uses the two defined commands which are defined in start program and stop program. In this case monit checks the unixsocket as well as a network connection to get in touch with MySQL before it trys to restart the service.
If after four cycles (which includes restarting the service after each cycle in this case) the service is still unreachable for monit, monit runs into the fifth cycle. This last cycle is defined with „alert“. The administrator, which has been declared in the monitrc file, will get a mail. If there is another competent person defined for this specific service, this person will be mailed as well. The name of the service in this example is mysqld.

Check for disk usage
check device root with path /
  if SPACE usage > 90% then alert

With this check, we check the root file system. If the used space in percentage is over 90, monit will send out an alert via mail. It would also be possible to check against the used inodes of a partition as well. If you have more than just one partition, you have to create a check for each partition of want to monitor.

Check for ram, swap and CPU usage
check system my.machine.fqdn
  if memory usage > 90% then alert
  if cpu usage > 90% for 4 cycles then alert
  if swap usage > 95% for 4 cycles then alert

In this example we check the memory, CPU and swap usage by the system. This example is similar to the disk usage. If the memory usage is higher than 90% of the overall memory of this system, monit will send out an alert. If the CPU usage is over 90% for four cycles, monit will send an alert as well. The four cycles ensure that monit is not starting to report every little peak a CPU has. Same goes to SWAP. If 95% of the overall SWAP is in use, monit will report this when this has happened four times / cycles in a row. In the first line you would have to change my.machine.fqdn to the real FQDN your system has.

Check an external host (ping)
check host other.host.fqdn with address other.host.fqdn
  if failed icmp type echo
    count 3 with timeout 5 seconds
    2 times within 3 cycles
then alert

With this check, monit tries to ping the specified other.host.fqdn. With every ping command monit starts, it does three pings with a timeout of five seconds for each. If the ping fails two times within a three times cycle, monit sends out an alert that the host is unreachable / offline. In order to get this command up and running, you have to change other.host.fqdn to the desired host which you want to check.
For every check you change, add or remove, you have to restart the monit service:

user@machine:~$ sudo systemctl restart monit

You can now see the status of the services with the help of the web interface or with the command line:

user@machine:~$ sudo monit status

If you want to get the status of one single check, you can append the name of the check at the end of the status command. You have to use the name you defined within the configuration file for this check in your command. For e.g. to get the status of the first check TeamSpeak we’ve configured in these examples:

user@machine:~$ sudo monit status TeamSpeak

mmonit: If you need a central web instance

So while monit offers a web interface which you can use in order to get an overview about the actual checked services, I understand that literally nobody wants to connect to each single monit web instance if you have 30 hosts or more.
Now, what you could do is to write a wrapper script (with bash, Python, etc.) which helps you to get the actual status of all services on all hosts. However, a more comfortable and easier solution is to use mmonit. mmonit is written by the official monit developers and gathers all information from each single monit instance and is showing the collected data on one single page.
Sadly, mmonit isn’t free. You can grab a 30 day trial if you want to, but at the end you have to pay a small one time fee in order to use the software. The prices are starting as low as 65 € for 5 hosts and is dropping sharply as more hosts you’re using. With 50 hosts you also get support for just 349 €. Again, this is a one-time pay license. There are no more additional costs coming up. If you compare this to other solutions like check_mk, mmonit is really cheap.

The price table as of 24th January 2018

Sadly, I’m unable to tell you more about mmonit for now. I wasn’t in the need to have to use it for now. My own wrapper script is doing more than just fine. Right now I’m monitoring +30 hosts with monit. But maybe the future will push me to get a mmonit license 😉

Final words

Is monit a full replacement for Nagios or Icinga? I’m unable to say if monit really fits into server farms with more than 100 servers. But if you want to have a easy monitoring solution which works with scripts (and Nagios plugins if you configure your checks correctly), you will be most likely happy with monit.
I’ve seen some talks recently on StackOverflow about monit and while a lot of people are saying that they wouldn’t use monit in bigger clusters, they use it on certain spots. For e.g. somebody talked about a customer with round about 20 servers. He rolled out monit there because of the fast and easy configuration. In order to check all 20 hosts at once, he created a bash script which connects via SSH and Public Key authentication to each host and issues the command monit status. In order to keep track if the host is still accessible from outside he just pings from another host with a ping check configured on another monit instance.
As you can see, even if monit looks like it’s limited or „not as powerful as Nagios“, there are no borders. A lot of people use and love monit due to it’s easy configuration and flexibility. If you’re looking for a monitoring solution or for an Nagios / Icinga alternative, you should give monit a try.
Last hint: If you want to monitor a single server (for e.g. home server or a Raspberry Pi) monit is would you should look up before Nagios or Icinga IMHO. In a setup like this, monit is definitely about to shine.

Further links

bookmark_borderHow to convert/split p12 certificates into single files

So, you got a brand new personal certificate via a authorized issuer and all you got is a single file which has a ending of .p12? You want to use this certificate in various software solutions, but these solutions want single files for the user certificate and the private key? Then you have to split your .p12 file.

What is a .p12 file?

A .p12 file is a bundle which contains your private key as well as your private certificate. For a lot of certificate issuers, distributing these two things in a bundle is obviously easier.
Even if there is a lot of software which supports working with those bundles, there are others which don’t. The most prominent example I know is Network Manager under Linux. If you want to use a .p12 file with the Network Manger OpenVPN extension, you have to split up the .p12 file in it’s single parts. To split p12 certificates into single files will end up in having two files: Your user certificate and key.

Which software is needed?

Under Linux you need to have OpenSSL ready. OpenSSL is installed by default on every Linux based machine nowadays. But just to be sure, we will install OpenSSL again for this tutorial. For Debian, Linux Mint and Ubuntu simply enter the following command:

user@systen:~$ sudo apt-get update && sudo apt-get install openssl

Windows user have to downloaded the OpenSSL tools on their official homepage which can be found here: OpenSSL Windows Binaries

How to split a .p12 file?

Firstly, you have to navigate into the directory were your SSL file is actually stored. You can do this with the command cd. In this example we assume, that the p12 certificate file is stored in the directory ssl:
user@system:~$ cd ssl
Now that you are in the correct directory, you can extract the user key with the following command:

user@system:~/ssl$ openssl pkcs12 -nocerts -in your_file.p12 -out user_key.pem

The user certificate can be exported like this:

user@system:~/ssl$ openssl pkcs12 -nokeys -clcerts -in your_file.p12 -out user_cert.pem

During these two steps you might get asked for a password of the actual .p12 file and for a password for the new exported files. It’s up to you if you want to protect the new exported single files with a password. However, it is recommended of course. You can also do the two commands above within one statement like this (if you want):

user@system:~/ssl$ openssl pkcs12 -nocerts -in your_file.p12 -out user_key.pem && openssl pkcs12 -nokeys -clcerts -in your_file.p12 -out user_cert.pem

Further links

bookmark_borderPrevent / Block package updates under Ubuntu / Debian

Did you know that you can block package updates under Ubuntu and Debian? Let’s say you have a lot of packages installed on your Ubuntu / Debian system and (for whatever reason) you want that specific packages aren’t getting updated whenever you do a system upgrade. This short article is going to show you how to prevent this packages from being updated.

APT or Aptitude: Both can block package updates

Debian / Ubuntu basically has two ways to manage packages. To be more specific there are two package managers which can be used on the console for updating, installing and removing packages / software under your Ubuntu / Debian systems. These two solutions are APT and Aptitude. This article describes how to prevent packages from being updated with both solutions. If you don’t know which of those two you should go with: Simply go with the APT tools (apt-get, apt-mark, apt-cache, …).

How to prevent packages from being updated.

You can always prevent packages from being updated with the help of APT. APT comes with every Ubuntu / Debian installation, so the following command should definitely work on any Debian / Ubuntu based system:

user@system:~$ sudo apt-mark hold <name of the package>

You have to change <name of the package> with the package you want to hold of course. So for e.g. if you want to prevent vlc from getting updated, the command would look like this:

user@system:~$ sudo apt-mark hold vlc

If you’re and Aptitude user instead, the command (with the exact same result) looks like this:

user@system:~$ sudo aptitude hold vlc

If you now update your system with the classical apt-get upgrade command for e.g., the package vlc isn’t going to be upgraded. APT, as well as Aptitude, will echo a notice which is saying that the package has been prevented from being updated.

How to unhold the package?

So, to hold a package is rather easy. But what to do when you want to unhold this package in order to get it updated again? If we use our vlc package from the example above again, the command to unhold and make a package available for an update with APT looks like this:

user@system:~$ sudo apt-mark unhold vlc

Again, the same command with the exact same result in Aptitude does look like this:

user@system:~$ sudo aptitude unhold vlc

But why to hold a package anyway?

You may ask yourself why you should hold a package anyway. Well, there are several reasons to do this. For e.g. sometimes you update a package and after this update the software doesn’t work as expected. So if you encounter a problem after an update on a test system, you could hold / block the specific package which causes you trouble on a production system before updating that system. Another example would be that you might have to check the configuration files first before updating a specific software. However, you want to install the latest security updates for the other installed packages. With holding the package you can update the other packages without touching the once you block.
Of course there are many other reasons why holding a package is a useful and a needed feature. You can also do this with a graphical solution like Synaptics. However, the console way of block package updates is much more easier and faster (IMHO) 😉

Further links

bookmark_borderownCloud/nextCloud collaboration with EtherPad and EtherCalc

Besides Collabora Office there is an easy but reliable way to enable collaboration editing of text and spreadsheet files in ownCloud or nextCloud (just like in proprietary solutions like Microsoft OneDrive or Google Drive). This article describes how to install and enable EtherPad along with EtherCalc in your ownCloud or nextCloud instance.

Requirements and preparation

First you need a running ownCloud or nextCloud instance of course. (Note: If you’re using ownCloud, ensure that you don’t use versions higher than 9.1 right now. As of January 2018, the used plugin for the EtherPad / EtherCalc integration doesn’t work with ownCloud higher than version 9.1.) It doesn’t matter if this instance is running on your local home server, an Raspberry Pi or if you rented some space at a webhosting provider of your choice. If you haven’t already a working nextCloud / ownCloud instance and you’re okay with hosting your cloud files at a provider, I recommend you the German webhoster All-Inkl.com. They offer cheap webhosting packages with an integrated installer. The installer automatically installs an ownCloud or nextCloud instance for you. You can directly start working with it afterwards. Just for the records: I’m using All-Inkl.com as well.
Besides the running instance you also need the ownPad plugin. It extends your ownCloud / nextCloud instance in order to communicate with an etherPad / etherCalc instance. You can download the latest release of ownPad for nextCloud and ownCloud here.

Installation

After you’ve downloaded ownPad you have to extract the plugin. If you’re using Windows, you can use WinRAR or 7-Zip to do this. For Linux you can double click the archive in order to open it or you simply extract the archive with the following command:

user@machine:~$ tar xfvz ownpad.tar.gz

You now have to move the so extracted ownpad directory to your webspace / server and place it under the apps folder of your ownCloud or nextCloud installation. Check that the directory has the chmod rights of 755 after you’ve uploaded it, otherwise ownCloud / nextCloud may be unable to find the plugin later on.

Configuration

As next we have to tell ownCloud / nextCloud how to handle files that are using the filename ending .pad or .calc (the standard filename endings for EtherPad and EtherCalc files). To do so, you have to copy the file mimetypemapping.dist.json from the subdirectory resources/config/, which is located within the ownCloud / nextCloud root directory, to the subdirectory config and rename it afterwards to mimetypemapping.json.
Open the so copied file config/mimetypemapping.json with your favorite text editor and add the following two lines to the end of the file:

 "pad": ["application/x-ownpad"],
 "calc": ["application/x-ownpad"]

So for example the end of the final mimetypemapping.json file could look like this:

 ...
 "yaml": ["application/yaml", "text/plain"],
 "yml": ["application/yaml", "text/plain"],
 "zip": ["application/zip"],
 "pad": ["application/x-ownpad"],
 "calc": ["application/x-ownpad"]
}

Now that every requirement has been met, you can go on and activate ownPad within your ownCloud / nextCloud instance. For nextCloud, click on the upper right (gear symbol) and click on Apps. Search for ownPad and click on Enable:

Deactivated ownPad App in the nextCloud administration overview (click to enlarge)

For ownCloud however, you have to click the category selection on the upper left and select Apps (plus symbol). To your left, click on Disabled in order to get a list of all the apps and plugins that are disabled in your ownCloud instance. Search for ownPad in this list and click on Enable.
Now that you’ve enabled the ownPad plugin you should ensure that EtherPad and EtherCalc are also activated. For nextCloud: navigate to Settings (gear symbol upper right), click on Additional settings and search for the Collaborative documents section. Ensure that both check boxes are set. For ownCloud you find these settings under the upper right (click on your username), followed by a click on Administration. To your left you will see a bunch of options. One of them is Collaborative documents. Click on it in:
Enabled EtherPad / EtherCalc

By default, EtherPad and EtherCalc are using the instances which are provided by the developers. If you want to use another public instance you can always changes these Host lines to your needs. A full list of public instances can be found here.
It is also possible to setup and host your own EtherPad / EtherCalc instance. Take a look at the official setup guide provided by the developers if you want to: How to setup your own EhterPad / EtherCalc instance.

Testing

To test the functionality, create a new EtherPad or EtherCalc file by clicking on the plus icon in your ownCloud / nextCloud file browser:

New file

As you can see, you have two new options which are called Pad and Calc, while Pad is EtherPad (text documents) and Calc is EtherCalc (spreadsheet documents). To create a new spreadsheet document for e.g., click on Calc and give the file a name you desire. After you’ve done this, you will see a new file with a .calc ending in your file browser:
Newly created test EtherCalc file (click to enlarge)

In order to open this file and start editing, just click on it. The EtherCalc editor is going to be loaded and you’re ready to edit the file. If there is no editor popping up (for e.g. your browser wants to download the file instead), recheck that you’ve executed the installation and configuration steps correctly or try a different browser and delete your browser cache as well.
Test EtherCalc file in action. EtherCalc also supports some formulas. (click to enlarge)

About collaboration …

Now that you have a working ownCloud or nextCloud instance with ownPad running, you can start sharing an ownCloud / nextCloud link, which points to your EtherPad or EtherCalc file, with your co-workers. You and your co-workers are able to work simultaneously on that file that way. EtherPad as well as EtherCalc supports color highlighting while multiple people are working on one document. Besides this there is a chat functionality built-in for a better and easier communication. However, you can also use EtherPad and EtherCalc for non collaborative documents.

Final words

At the beginning of this article I mentioned that EtherPad / EtherCalc allows you to do collaboration text / spreadsheet editing like in Microsoft OneDrive or Google Drive. To be honest, to get the same functionality like in these two proprietary solutions you may should better go with Collabora Office which basically provides LibreOffice in your browser. However, for a simple but also effective solution you really should take a look at EtherPad / EtherCalc. It’s easy to setup and use. Besides this it comes with a nice chat functionality which may be helpful as well. And hey, it’s free right? I love open source 😉
 

bookmark_borderHappy new year 2018

Header Image Source: pixabay.com
 
The year 2017 is coming to an end. We have seen a lot of development in the Linux world. 2018 will be as successful as 2017. That’s what I’m hoping and wishing at least 🙂
In 2018 I plan to do at least one post every two weeks on this blog. I really like to write and the increased feedback these last months showed me that I’m doing something right. So I will go on in 2018 in creating new and high quality content.
I wish everyone a good start in 2018. Stay healthy and have fun whatever you’re doing. We will read us 😉

Veröffentlicht in Talk