bookmark_borderMonit: An easy to use monitoring solution for Linux

Besides Nagios, Icinga and check_mk there are some other, more slimmer tools to monitor servers. Especially if you are a private person and you want to monitor your vServer, Raspberry Pi or whatever, you may want to use a smaller and easier monitoring solution than those big three.
This article is about monit. Monit is one of these simple monitoring tool. But just because the configuration is more simple, that doesn’t mean hat you are limited in the ways you monitor your servers with Monit.

How does monit works? And why not just use Nagios?

monit works differently than Nagios or Icinga. While Nagios, Icinga and check_mk needs a monitoring server which connects in given time periods to the machines it checks, monit doesn’t need this kind of server in order to do these checks. So basically that means that wherever you install monit, it checks locally and reports the results via mail.
In the first moment this sounds like a disadvantage whether it be due to reliability or security, but think about it for a second: What does Nagios, Icinga or check_mk basically do? They ping the destination machine and if this ping is successful it opens up an NRPE or SSH connection and executes the given CPU, Ram, service checks and so on, locally. The console output is then used for analysis. If the check exceeds it’s given limit, Nagios / Iciniga / check_mk creates an E-Mail and send it to the user /group of users which is / are responsible for this service. So basically the server executes local commands like monit. The initiator however is not locally, it’s a different (remote) server.
You now may think „but how do I even test if the server is available?“. Well, you just setup another monit instance which is checking if the destination is pingable, just like with Nagios / Icinga / check_mk.
A real benefit for monit is it’s easy configuration and syntax. While you have to dig through a bunch of files in order to create a simple check in Nagios, monit allows you to simply put one (human readable) configuration file in the correct directory and the check is ready to use. In combination with a configuration management like Ansible (I’ve already created an article about Ansible a long time ago) you have an easy central to use monitoring tool with automatically distributed configuration files.

Installation of monit

Monit is available for most Linux distribution. With the following command you install monit on a Ubuntu / Debian machine:

user@machine:~$ sudo apt-get update && sudo apt-get install monit

Or if you’re an openSUSE user, you can install monit like this:

user@machine:~$ sudo zypper ref && sudo zypper in monit

That’s basically it. Compared to Nagios or Icinga, you don’t need an installed Apache web server. Monit comes with it’s own built-in web service. In order to check if monit is running, you can use systemctl to do so:

user@machine:~$ sudo systemctl status monit
● monit.service - LSB: service and resource monitoring daemon
 Loaded: loaded (/etc/init.d/monit; generated; vendor preset: enabled)
 Active: active (running) since Wed 2017-10-11 11:47:39 CEST; 3 months 13 days ago
 Docs: man:systemd-sysv-generator(8)
 Tasks: 2 (limit: 4915)
 CGroup: /system.slice/monit.service
 └─18228 /usr/bin/monit -c /etc/monit/monitrc
Dez 26 06:25:02 machine systemd[1]: Reloading LSB: service and resource monitoring daemon.
Dez 26 06:25:02 machine monit[15742]: Reloading daemon monitor configuration: monit.
Dez 26 06:25:02 machine systemd[1]: Reloaded LSB: service and resource monitoring daemon.
Jan 10 06:25:02 machine systemd[1]: Reloading LSB: service and resource monitoring daemon.
Jan 10 06:25:02 machine monit[15692]: Reloading daemon monitor configuration: monit.
Jan 10 06:25:02 machine systemd[1]: Reloaded LSB: service and resource monitoring daemon.

In order to start, stop or restart the service, use these following commands:

user@machine:~$ sudo systemctl start monit
user@machine:~$ sudo systemctl stop monit
user@machine:~$ sudo systemctl restart monit

Enable the monit built-in web server

If you’re an Debian / Ubuntu user, monit will store it’s main configuration under /etc/monit/monitrc. This files is really well documented. However, I want to give a short brief about how to enable the built-in web server which is one of the most important features.
You can find the following block (at the time of writing) at line 155:

# set httpd port 2812 and
# use address localhost # only accept connection from localhost
# allow localhost # allow localhost to connect to the server and
# allow admin:monit # require user 'admin' with password 'monit

If you uncomment this section the built-in monit web server will be started and will listen on port 2812. The second line sets the network address where the web server instance is listening on. If it only listens on localhost you will not be able to connect from any other host. For e.g. if your server has the IP address 192.168.0.2 you can replace localhost with this IP address. This will get the monit built-in web server to listen on this network interface.
The third line regulates which other hosts are allowed to establish a connection to the web server. You can define single IP addresses or even whole subnets (like 192.168.1.0/24). Just like before, simply replace localhost with the IP address or subnet which you want to allow to connect to the web server.
The fourth line defines a user with a password which will be able to login. monit does not allow an anonymous login which is good from a security point of view obviously. In this example the username would be admin with the password monit.
When you’re done setting these lines, restart the monit service:

user@machine:~$ sudo systemctl restart monit

You can now visit the monit web interface of your server. In this example the address would be something like http://192.168.0.2:2812 or just use the FQDN: http://my.monit.fqdn:2812.
Besides using the web interface, you can also check your services on the command line. To do so you also have to have the web interface enabled, because the command line simply gets an HTTP output and process it. To get a summary of all your monitored processes you can enter the following command:

user@machine:~$ sudo monit status

Enable mail notifications

Nagios or Icinga can send you a mail every time when a host or a service of this host went down. monit also offers this feature. To enable mail notifications, open once more the monit configuration file (/etc/monit/monitrc) and search for the following lines:

## Set the list of mail servers for alert delivery. Multiple servers may be
## specified using a comma separator. If the first mail server fails, Monit
# will use the second mail server in the list and so on. By default Monit uses
# port 25 - it is possible to override this with the PORT option.
#
# set mailserver mail.bar.baz, # primary mailserver
# backup.bar.baz port 10025, # backup mailserver on port 10025
# localhost # fallback rela

As you can see, you can set multiple mail servers which monit should try to connect each time a notification is about to be send. The downside here is that monit does not support authentication against mail servers. This basically means that you can’t just enter public mail servers like GMail. If you want to use those service you have to have a local mail server setup. You can configure the mailserver as a satellite / client to keep it more simple. If you’re mail server is up and running, you can simply set the mail server in the monitrc file like this:

set mailserver localhost

If you have access to a mail server which doesn’t require authentication, you can replace localhost with this mail server of course.
Now that you’ve set the mail server, you also have to enter the recipient for the notification mails. Search for the following line in the monitrc file:

# set alert sysadm@foo.bar # receive all alerts

Uncomment this line and replace sysadm@foo.bar with the recipient of your choice. You can also define which kind of notification are going to be send to this mail address. However, this is more advanced configuration which would burst this articles length.
When everything set, restart monit and you will now get a mail each time a service status changes:

user@machine:~$ sudo systemctl restart monit

Some examples for monitoring services

For monitoring new programs or services, you have to create a new file under the subdirectory /etc/monit/monitrc.d/. Simply name this file like the service which is going to be monitored. The following examples are showing how to monitor services, programs and system resources with monit:

Monitoring a running program / service or service without a PID file
check process TeamSpeak
  matching "ts3"

In this example we simple check if a process called „ts3“ is running on the machine. You can use ps ax on the command line by yourself on the target machine to see if the process is running. That’s basically what monit is doing. The name of this check is TeamSpeak as seen in the first line. You can change this to whatever you want.

Monitoring a program / service with a PID file
check process mysqld with pidfile /var/run/mysqld/mysqld.pid
  start program = "systemctl start mysql"
  stop program = "systemctl stop mysql"
  if failed host localhost port 3306 protocol mysql with timeout 15 seconds for 3 times within 4 cycles then restart
  if failed unixsocket /var/run/mysqld/mysqld.sock protocol mysql for 3 times within 4 cycles then restart
  if 5 restarts with 5 cycles then alert

In this specific example we monitor a MySQL instance installed on the system. If the service crashes, monit tries three times within four cycles to restart the service. For this restart monit uses the two defined commands which are defined in start program and stop program. In this case monit checks the unixsocket as well as a network connection to get in touch with MySQL before it trys to restart the service.
If after four cycles (which includes restarting the service after each cycle in this case) the service is still unreachable for monit, monit runs into the fifth cycle. This last cycle is defined with „alert“. The administrator, which has been declared in the monitrc file, will get a mail. If there is another competent person defined for this specific service, this person will be mailed as well. The name of the service in this example is mysqld.

Check for disk usage
check device root with path /
  if SPACE usage > 90% then alert

With this check, we check the root file system. If the used space in percentage is over 90, monit will send out an alert via mail. It would also be possible to check against the used inodes of a partition as well. If you have more than just one partition, you have to create a check for each partition of want to monitor.

Check for ram, swap and CPU usage
check system my.machine.fqdn
  if memory usage > 90% then alert
  if cpu usage > 90% for 4 cycles then alert
  if swap usage > 95% for 4 cycles then alert

In this example we check the memory, CPU and swap usage by the system. This example is similar to the disk usage. If the memory usage is higher than 90% of the overall memory of this system, monit will send out an alert. If the CPU usage is over 90% for four cycles, monit will send an alert as well. The four cycles ensure that monit is not starting to report every little peak a CPU has. Same goes to SWAP. If 95% of the overall SWAP is in use, monit will report this when this has happened four times / cycles in a row. In the first line you would have to change my.machine.fqdn to the real FQDN your system has.

Check an external host (ping)
check host other.host.fqdn with address other.host.fqdn
  if failed icmp type echo
    count 3 with timeout 5 seconds
    2 times within 3 cycles
then alert

With this check, monit tries to ping the specified other.host.fqdn. With every ping command monit starts, it does three pings with a timeout of five seconds for each. If the ping fails two times within a three times cycle, monit sends out an alert that the host is unreachable / offline. In order to get this command up and running, you have to change other.host.fqdn to the desired host which you want to check.
For every check you change, add or remove, you have to restart the monit service:

user@machine:~$ sudo systemctl restart monit

You can now see the status of the services with the help of the web interface or with the command line:

user@machine:~$ sudo monit status

If you want to get the status of one single check, you can append the name of the check at the end of the status command. You have to use the name you defined within the configuration file for this check in your command. For e.g. to get the status of the first check TeamSpeak we’ve configured in these examples:

user@machine:~$ sudo monit status TeamSpeak

mmonit: If you need a central web instance

So while monit offers a web interface which you can use in order to get an overview about the actual checked services, I understand that literally nobody wants to connect to each single monit web instance if you have 30 hosts or more.
Now, what you could do is to write a wrapper script (with bash, Python, etc.) which helps you to get the actual status of all services on all hosts. However, a more comfortable and easier solution is to use mmonit. mmonit is written by the official monit developers and gathers all information from each single monit instance and is showing the collected data on one single page.
Sadly, mmonit isn’t free. You can grab a 30 day trial if you want to, but at the end you have to pay a small one time fee in order to use the software. The prices are starting as low as 65 € for 5 hosts and is dropping sharply as more hosts you’re using. With 50 hosts you also get support for just 349 €. Again, this is a one-time pay license. There are no more additional costs coming up. If you compare this to other solutions like check_mk, mmonit is really cheap.

The price table as of 24th January 2018

Sadly, I’m unable to tell you more about mmonit for now. I wasn’t in the need to have to use it for now. My own wrapper script is doing more than just fine. Right now I’m monitoring +30 hosts with monit. But maybe the future will push me to get a mmonit license 😉

Final words

Is monit a full replacement for Nagios or Icinga? I’m unable to say if monit really fits into server farms with more than 100 servers. But if you want to have a easy monitoring solution which works with scripts (and Nagios plugins if you configure your checks correctly), you will be most likely happy with monit.
I’ve seen some talks recently on StackOverflow about monit and while a lot of people are saying that they wouldn’t use monit in bigger clusters, they use it on certain spots. For e.g. somebody talked about a customer with round about 20 servers. He rolled out monit there because of the fast and easy configuration. In order to check all 20 hosts at once, he created a bash script which connects via SSH and Public Key authentication to each host and issues the command monit status. In order to keep track if the host is still accessible from outside he just pings from another host with a ping check configured on another monit instance.
As you can see, even if monit looks like it’s limited or „not as powerful as Nagios“, there are no borders. A lot of people use and love monit due to it’s easy configuration and flexibility. If you’re looking for a monitoring solution or for an Nagios / Icinga alternative, you should give monit a try.
Last hint: If you want to monitor a single server (for e.g. home server or a Raspberry Pi) monit is would you should look up before Nagios or Icinga IMHO. In a setup like this, monit is definitely about to shine.

Further links

bookmark_borderHowTo setup a Counter Strike: Global Offensive Server on Linux

A lot of people who are running a rented Linux (v)Server are interested in creating a Counter-Strike: Global Offensive server. Creating a CSGO server under Linux is rather easy, Valve really did their homework here. The following short tutorial will give you the needed instructions to create a Counter-Strike: Global Offensive Server under Debian, Ubuntu or openSUSE:

1. Create a new user

First of all, you should really create a new user in your Linux system. The reason for that is simple: security! If your main user, or even your root user, is running the CS:GO server and is hacked later on, the hacker maybe is in the position to get access to the system behind the Server. With that being said, he may be able to get full access to the shell and he may be able to manipulate the system. To create new account on your Debian, Ubuntu or openSUSE system, you have to enter the following commands:

root@server:~# useradd csgosrv
root@server:~# passwd csgosrv
root@server:~# mkdir /home/csgosrv
root@server:~# chown csgosrv:users /home/csgosrv

So, the first commands creates a new user, called csgosrv. The seconds commands sets a new password for this user. The password you enter here will not be prompted.
The third command creates a new directory called csgosrv under the directory /home. This will be the standard home directory for the user csgosrv we’ve created before.
The fourth and last command sets the owner to the created user csgosrv and the group owner of the created csgosrv home directory to users.

2. Install needed dependencies

In this step we need to download the needed libraries in order to get the Steam command line tool working. If these libraries / tools are not installed, the Steam command line client (provided by Valve), will fail to start.
If you’re on a 64-Bit Debian or Ubuntu system you have to enable the i386 architecture in the first place. This is needed because the server software is written for the 32-Bit architecture. If you’re on a 32-Bit Ubuntu or Debian, you can skip this command:

root@server:~# dpkg --add-architecture i386

The following two commands are needed for 32-Bit and 64-Bit systems. They will update the repository information and install the needed libraries:

root@server:~# apt-get update
root@server:~# apt-get install gcc-multilib libstdc++6:i386 libgcc1:i386 zlib1g:i386 libncurses5:i386 libc6:i386 wget screen

If you’re on a openSUSE system, the commands are doing the same, but the syntax is different. The following two commands are updating the package repository and installs the needed libraries on a openSUSE system:

root@server:~# zypper ref
root@server:~# zypper in wget libgcc_s1-32bit libgcc_s1-gcc6-32bit ca-certificates screen

Now, that you have installed all the needed libraries, we can go on and start downloading and installing the server software.

3. Download Steam

Downloading the Steam command line tool is very easy. You can always get it directly from Valve. But before we start downloading the client, we should change to the beforehand created user csgosrv. We can do this with the following command:

root@server:~# su - csgosrv

As an alternative you could close your SSH session and reconnect with the user csgosrv.
Now that we’re working with the right user, it’s time to download and extract Steam:

csgosrv@server:~$ mkdir steam
csgosrv@server:~/steam> cd steam
csgosrv@server:~/steam> wget http://media.steampowered.com/client/steamcmd_linux.tar.gz
csgosrv@server:~/steam> tar xfvz steamcmd_linux.tar.gz

So, the first two commands are creating a new directory, called steam, and directly changed into it with the help of the cd command. The third command is downloading the Steam CMD tool with the help of the tool wget. The fourth and last command extracts the so downloaded .tar.gz archive.
After you’ve done this steps successfully you can go on and download the CS:GO server software.

4. Setup Steam and install CS:GO server files

This command will update the steam command line tool and installs the application „740“ which is the Counter-Strike: Global Offensive server. For now you don’t need to have your steam credentials ready. You can update steam and install the server software as an anonymous user:

csgosrv@server:~/steam> ./steamcmd.sh +login anonymous +force_install_dir ./csgo_server/ +app_update 740 validate +quit

This command will take some time and the so downloaded server software needs round about 15 GB hard disk space. The Server software will be installed in the directory /home/csgosrv/steam/csgo_server.

5. First start of the CS:GO server

Now that you have successfully downloaded and updated the Counter-Strike: Global Offensive server, we can do a test run. With the following command we can start the server with the mode Classic Casual and the Mapgroup mg_bomb which includes the old bomb spot maps like de_atztec or de_dust / de_dust2:

csgosrv@server:~/steam> cd csgo_server
csgosrv@server:~/steam/csgo_server> ./srcds_run -game csgo -console -usercon +game_type 0 +game_mode 0 +mapgroup mg_bomb +map de_dust

You will now get a lot of output. This is normal and shows you, that the software is working. However, at the end of all the lines you will find a message like this one:

****************************************************
*                                                  *
*  No Steam account token was specified.           *
*  Logging into anonymous game server account.     *
*  Connections will be restricted to LAN only.     *
*                                                  *
*  To create a game server account go to           *
*  http://steamcommunity.com/dev/managegameservers *
*                                                  *
****************************************************

Basically this message means that your server is not registered at Vale. As long as you don’t register your server at Vale, you will only be able to connect to the server via LAN, not via the internet. It will never be an official server played on. To make your server ready for the internet, you have to go on with the next step (you can terminate the actual running session with an CTRL+C combination).

6. Make your server an official registered one

To register your CS:GO server at Valve and make it ready for internet play, you have to register the server directly at Valve. To do so, visit the following link: Steam Server management. Login with your steam credentials (if you haven’t already) and enter the number 730 as the App ID in the first text box. The second text box can be filled with whatever you want. It’s just a comment field. For e.g. if you have more than one server you could write down the hostname here so that you can always directly see which token belongs to which server. Click on Create to get your Token:
After you clicked on Create you will see your token for your server. The token can only be used on one server for one active session. You can’t use the token for multiple servers at the same time. You have to create a new token for each of your servers.
Now that you have created your token, you can use it to start the server as an official CS:GO server with the following command:

csgosrv@server:~/steam/csgo_server> ./srcds_run -game csgo -console -usercon +game_type 0 +game_mode 0 +mapgroup mg_bomb +map de_dust +sv_setsteamaccount YOURTOKEN -net_port_try 1

Of course you have to replace the YOURTOKEN with the token you’ve created before. If there is no firewall blocking the traffic, you should now be able to find your server via the CS:GO server browser or connect directly via the hostname / IP to it.

7. Start the CS:GO server in the background

You may already noticed, that the CS:GO server stops running when you close the SSH connection. This is, because the CS:GO server software needs an active terminal / SSH session to go on running. However, there is a tool which is called screen. If you’ve followed this tutorial, you already have installed screen at the first steps. The following commands starts the CS:GO server in the background with the help of screen.

csgosrv@server:~/steam/csgo_server> screen -A -m -d -S csgo_server ./srcds -game csgo -console -usercon +game_type 0 +game_mode 0 +mapgroup mg_bomb +map de_dust +sv_setsteamaccount YOURTOKEN -net_port_try 1

If you want to jump into the screen session you can easily do this with this command:

csgosrv@server:~/steam/csgo_server> screen -r csgo_server

As long as you are connected (attached) to the screen session, you can do anything here like in a normal terminal session. If you want to leave the screen session again, just press CTRL+a, followed by the key d (CTRL+a sends a signal to the screen program that the next key stroke is something screen has to handle. The d key says to screen than: detach!).

Optional information

If you have problems connecting to your server, you should check if there is no firewall blocking the traffic. If there is any firewall doing so, you have to unlock the port 27015 (UDP) for your server.
For a more further configuration of the server (settings like warm up, max player count, map rotation and so on) you should visit the following page. It contains a almost complete server.cfg with a lot of comments: server.cfg on csgodev
For further fine tuning at the game modes you should read the official tutorial wikipedia page from Valve about this: CS:GO Server gamemodes
I also want to provide my own server.cfg file here. It’s rather simple and just sets some basic features, like friendly fire and warm up time. You can see my server.cfg file here: server.cfg
T
he server.cfg has to be stored in the directory cfg which is a subdirectory of the csgo_server directory, the directory which we had chosen while downloading the CS:GO server software.
 
Enough of the words. I wish you a good time in CS:GO and best wishes for you and your server 🙂
 
 

bookmark_borderRocket League under Linux: "There has been an error connecting to the Rocket League servers please try again later"

One of my favorite online competitive games right now is Rocket League. It’s competitive, supports splitscreen (yes, even on PC), makes a lot of fun and now it’s also available for Linux. I played Rocket League before as it was a part of the PS Plus monthly free games months ago. Now that Rocket League has official arrived for Linux, I decided to buy this game on Steam as well.

Unable to connect to servers

Well, the bad thing was, after I started the game, the following message appeared:

There has been an error connecting to the Rocket League servers please try again later

I started to check my network connection, was googling if the Rocket League server were down and so on. Then I started to feel that this has something to do with my distribution (for the records, I’m actually using openSUSE Tumbleweed, a rolling release distribution). After a short time of searching the web I found the solution for the problem. For openSUSE you simply have to issue the following command as root or with the sudo command:

user@opensuse:~$ sudo ln -s /etc/ssl/ca-bundle.pem /etc/ssl/certs/ca-certificates.crt

This command creates a symbolic link which can later be found in /etc/ssl/certs/ca-certificates.crt. The link itself points to /etc/ssl/ca-bunble.pem/. Rocket League needs this certificate to connect to their servers. It’s looking in the directory /etc/ssl/certs/ for the certificate which can’t be found. To solve this, we need the symbolic link of the certificate file where it is originally stored in openSUSE Tumbleweed (which is /etc/ssl/).
The solution for this problem was originally discussed at the Steam community site: Link.
Restart Rocket League and you should be able ready to go. Please keep in mind that the path of the certificate can differ if you use another distribution. Anyway the target path (which is /etc/ssl/certs/) is always the same.
Good luck and have fun with a working Rocket League 🙂

Further links

bookmark_borderOpenSource projects and their donations

Today I checked out a video, which was recorded at the LFNW 2011 and presented by Bryan Lunduke.
Most of you people will know about Bryan from „The Linux Action Show“, but he also is presenting the „Why Linux sucks“ at the LFNW every year.
So, I don’t want to repeat the whole video now, but I want to go further with some (in my point of view) very important things, which the video has brought to me to think about:
1. Programmers have to eat: Well, this is just true. Bryan comes here to the point, that the OpenSource developers also have families and just want to have a roof over their head. I’m totally agree to that!
2. We as Linux users have to understand that makeing software costs money: Also just true. If we look only at the time which the development is devours the costs getting really high for some projects (e. g. GNOME). At least, there are more points which are comeing with that, for e. g. the planning and design, test phases and so on.
3. We should buy propietary software for Linux or donate money to OpenSource projects (or both) which comes nearly to the licence costs of a proprietary software for another platform: Well, this is a little bit tricky. Bryan says in the presentation that he doesn’t mean with that, that the people now should go out and donate 5000$ to LibreOffice, but he just want that the people honour the work, even if it’s free and OpenSource. Additionally, buying software on this way or makeing donations, shows other developers and companies, that there is a market for Linux and at least, there is a reason to develop for it!
After the video, here the link: http://www.youtube.com/watch?v=Ukd-Am2bbDo , I started to think a short time about his words. Also, I made a little conclusion for me, which „donations“ I made for openSource projects in the last time:
– I bought the openSUSE boxes from OpenSource Press
– I bought the Ubuntu boxes from OpenSource Press
– I made a „one time“ donation to WINE
– I have a CrossOver subscription, which I’m also renew every time, after the subscription isn’t valid anymore
Also, I bought a software which is called „moneyplex“ which allows me to handle my HBCI online banking on a easier way on Linux. This software is propietary, but it works just nice, fast and it’s stable!
Well, overall, that isn’t that much. Because of this situation, I started to check, how much money I can donate for now and which projects I want to donate.
My first donations will go to the GNOME project and to LibreOffice. This two peacies of software are just awesome, and I want, that they will never stop there work! So, I have to donate 🙂
Also, I want to donate a few bucks to the VLC Player. Also, just a awesome piece of software which never should be missed at a new fresh OS installation 😉
My recommendation for you is, watch the video. Bryan is right in many cases. Think about the words and if it’s possible (or if you want) donate to your favourite projects at the OpenSource scene. Even if you just have 5 bucks available for a donation … donate it! 🙂