Extending the functionality of Zabbix agents with your own checks

About a month ago, I started playing around with Zabbix to improve my monitoring needs. While setting up Zabbix on Debian/Ubuntu itself is pretty straightforward when using apache2, and also doable when preferring nginx, the documentation regarding plugins and custom checks felt missing and a bit cumbersome to me. This is especially a sad thing when you stumble upon a great collection of Zabbix community projects or ztc but you have a hard time figuring out how to use them properly. Using Zabbix would quickly reach its limits without your own collection of useful (custom) checks. Of course you can grab templates from other users from Zabbix Share but maybe you prefer to write your own, learn something, or do it just a bit different for your custom checks. It seems that there are multiple ways to extend Zabbix, but it looks like most solutions are using user parameters (UserParameter) for this.

For example, let's say you run a LAMP/LEMP stack, and since it's 2016 you are using php{5,7}-fpm for PHP. You would like better monitoring for PHP-FPM and it's OPcache, so you could either write your own plugin or you end up using one of these existing ones. The first one mentioned has quite some documentation in the README, the second one has just enough and the third one has no documentation at all. This might feel a bit overwhelming to users new to Zabbix.

The main thing that you need to understand at that point, is that most custom checks (which feel like plugins to me) consist of the following 3 parts (filenames for an example php-fpm plugin):

1. php-fpm.conf: Configuration file for on the host that runs zabbix-agent. Contains at least one line in the following format: UserParameter=<key>,<script> (<arguments>). Could be inlined in /etc/zabbix/zabbix_agentd.conf, but it's cleaner to put all custom checks (plugins) in a separate folder, and in /etc/zabbix/zabbix_agentd.conf define to include that folder. By default, on Debian/Ubuntu, there is an include line like this: Include=/etc/zabbix/zabbix_agentd.conf.d/, so you could put your *.conf files in /etc/zabbix/zabbix_agentd.conf.d/ and they will get picked up on the agent. An additional plus side is that you could put this in git or a package to distribute it.
2. php-fpm.sh or php-fpm.php: The executable script for on the host that runs zabbix-agent. This script actually gathers the data for the UserParameter defined in the .conf file. The script might need input or modifications to match your environment, especially when you are using custom checks (plugins) from third parties. Hopefully they documented this for you. Or you can write your own if that suits you better.
3. php-fpm.xml: The Zabbix Template definition for on the Zabbix Server. Exported and imported via Configuration -> Templates in the Zabbix Server Web Interface. This could define what items it should monitor, and which specific keys to look for. Could also define triggers and graphs. See the default templates or linked repositories above for examples.

With this in mind, the Zabbix documentation on this matter is still cumbersome, but makes a lot more sense as a whole to me.

UserParameters can be either be tested in 2 ways. The first one is on the host that runs zabbix-agent, via # zabbix_agentd -t mysql.questions if you added the Zabbix example UserParameter=mysql.questions,mysqladmin -uroot status | cut -f4 -d":" | cut -f1 -d"S" as root, however, note that the agent may have different permissions when launched as a daemon. The second one is on the Zabbix Server, from there you can do test this parameter by using the zabbix_get utility.

Considering that Munin lists Zabbix as a Competitor, I think it could reasonably be expected from Zabbix to have a proper documentation page on how to write custom checks (plugins) for Zabbix, as the Munin documentation on this matter is not far from excellent.

At the end of the day, please remember this brilliant quote from the ztc Zabbix Template Collection: "if it’s not monitored, it doesn’t exist".