Monitoring, Log File Analysis & Visualization – The Open Source Way

What Is Zabbix + ELK In A Box – Those of you who work on designing, building and supporting applications at some point might have felt the need to see what’s happening across your environment, understand the nature of errors your applications are throwing or obtain a view of the performance of your infrastructure. Being able to afford commercial monitoring tools, machine data analysis, log file analysis and visualization tools does make your life easier. But for the majority of hackers (individuals who love playing with technology) out there, we struggle with basic monitoring, machine data analysis and log file analysis resorting to using arcane scripts and manual monitoring of the systems, log files, applications and databases. Non commercial offerings from various vendors e.g. Splunk light, Sumologic basic, etc. could also be one possible answer here if you are keen to stay away from Open Source software.

Zabbix + ELK In A Box (Free Software, GPL v3) on the other hand is designed to make it easier for users to monitor their applications, aggregate log files and machine data from across their environment, collect application performance metrics, infrastructure performance metrics and easily export them for purposes of visualization, modelling and forecasting. Zabbix + ELK In A Box is design to perform the following tasks –

  • Monitor infrastructure performance
  • Monitor application performance
  • Collect machine data and logs
  • Collect application logs
  • Provide visualization capability
  • Allow for exporting of the relevant performance metrics for use in VisualizeIT

The Zabbix + ELK In  A Box stack consists of –

  • Ubuntu Linux Virtual Machine (Ubuntu 14.04)
  • Zabbix 3.0 for purposes of infrastructure and application monitoring
  • ELK (Elasticsearch, Logstash, Kibana) for purposes of collecting machine data and log files

Zabbix + ELK In A Box in essential is a collection of software that will allow you to monitor your applications, infrastructure and collect performance metrics that can be exported for further visualization, modelling and forecasting. You can access Zabbix + ELK In A Box at Sourceforget.net. Please also note that you will need “some basic” linux/unix skills to be able to make use of Zabbix + ELK In A Box.

If you are shell shy i.e. not very keen on working with the Unix/Linux console….please go no further. This project is not for you. Save yourself sometime and energy and pick up the basic trial/free versions commercial vendors have to offer.

What are the components that make up Zabbix + ELK In A Box – Zabbix + ELK In A Box are essentially made up of the three components we’ve listed above.

  • Ubuntu Linux Virtual Machine (Ubuntu 14.04)
  • Zabbix 3.0 for purposes of infrastructure and application monitoring
  • ELK (Elasticsearch, Logstash, Kibana) for purposes of collecting machine data and log files
    • Filebeat – For collecting and parsing of machine data and log files locally
    • Topbeat – For collecting and parsing of infrastructure performance metrics locally

Ubuntu is a Debian-based Linux operating system and distribution for personal computers, smartphones and network servers. Ubuntu Linux as you would know is one of the most popular Linux distributions around with a large community following (very important for any opensource project you consider implementing). We love Ubuntu Linux for many reasons, one because its Opensource, second because it’s Linux, third because it’subuntu-logo112 based on Debian and uses Debian’s awesome package management system i.e. apt-get. Talk to any Debian user and you’ll appreciate the beauty of apt-get and why the Opensource community is just so much in love with it.

Ubuntu is based on free software and named after the Southern African philosophy of ubuntu (literally, “human-ness”), which often is translated as “humanity towards others” or “the belief in a universal bond of sharing that connects all humanity”.  Development of Ubuntu is led by UK-based Canonical Ltd., a company owned by South African entrepreneur Mark Shuttleworth. Canonical generates revenue through the sale of technical support and other services related to Ubuntu. The Ubuntu project is publicly committed to the principles of open-source software development; people are encouraged to use free software, study how it works, improve upon it, and distribute it.

You can read more about Ubuntu at www.ubuntu.com.

Elasticsearch, Logstash, Kibana or ELK on the other hand is intended for purposes of collecting machine data, log aggregation, visualization and analytics solution based on a completely Open Source stack. ELK is  ELK consists of consists of three different Open Source components –elk-logos

  • Elasticsearch
  • Logstash
  • Kibana.

The three components that make up the ELK stack offer the following functionality –

  • Elasticsearch: A powerful open source search and analytics engine that makes data easy to explore. It is a search server based on Apache Lucene.
  • Logstash: A log management tool used for centralised logging, log enrichment and parsing.
  • Kibana: A browser-based HTML5 dashboard used to visualize Elasticsearch data.

The open-source ELK stack provides the ability to perform operational and data analytics including comprehensive text based search functionality on almost any type of structured or unstructured data source.

Although they’ve all been built to work exceptionally well together, each one is a separate project that is driven by the open-source vendor Elastic—which itself began as an enterprise search platform vendor. Elasticsearch has now become a full-service analytics software company, mainly because of the success of the ELK stack and its acceptance globally as an Open Source analytics and visualization solution. Wide adoption of Elasticsearch for analytics has been the main driver of its popularity. ELK is very similar to Splunk or Sumologic in terms of basic functionality but is run as an Open Source platform.elk-flow

To learn more about ELK (Elasticsearch, Logstash, Kibana) please visit – Elastic.

Zabbix is enterprise open source monitoring software for networks and applications, created by Alexei Vladishev. It is designed to monitor and track the status of various network services, servers, and other network hardware. Zabbix uses MySQL, PostgreSQL, SQLite, Oracle or IBM DB2 to store data. Its backend is written in C and the web frontend is written in PHP.

Simple checks can verify the availability and responsiveness of standard services such as SMTP or HTTP without installing any software on the monitored host. A Zabbix agent can also be installed on UNIX and Windows hosts to monitor statistics such as CPU load, network utilization, disk space, etc. As an alternative to installing an agent on hosts, Zabbix includes support for monitoring via SNMP, TCP and ICMP checks, as well as over IPMI, JMX, SSH, Telnet and using custom parameters. Zabbix supports a variety of real-time notification mechanisms, including XMPP.

Zabbix offers several enterprise class features you would expect from a monitoring solution:

  • High performance, high capacity (able to monitor hundreds of thousands of devices)zabbix
  • Auto-discovery of servers and network devices. Low-level discovery
  • Distributed monitoring with centralized web administration
  • Support for both polling and trapping mechanisms
  • Native high performance agents (client software for Linux, Solaris, HP-UX, AIX, FreeBSD, OpenBSD, OS X, Tru64/OSF1, Windows 2000, Windows Server 2003, Windows XP, Windows Vista, Windows Server 2008, Windows 7)
  • Agent-less monitoring
  • JMX monitoring
  • Web monitoring
  • Secure user authentication
  • Flexible user permissions
  • Web-based interface
  • SLA, and ITIL KPI metrics on reporting
  • Flexible e-mail notification on predefined events
  • High-level (business) view of monitored resources through user-defined visual console screens and dashboards
  • Audit log

Released under the terms of GNU General Public License version 2, Zabbix is free software.

You can learn more about Zabbix at www.zabbix.com.

What do you need to set up Zabbix + ELK In A Box – You need the following in your environment to be able to use Zabbix + ELK In A Box.

  • Virtual Machine Manager e.g. Virtualbox (https://www.virtualbox.org/)
  • Static IP Address
  • 30 GB storage space to host the Virtualmachine
  • 2VCPU’s to be allocated to Zabbix + ELK In A Box
  • 4 GB of Memory to be allocated to Zabbix + ELK In A Box

The minimum recommended configuration would be 2 vCPU’s, 2GB Memory while the recommended configuration for Zabbix + ELK In A Box is 2 vCPU’s, 4 GB Memory. The Zabbix 3.0 frontend service designed in PHP and hosted on Apache 2.x combined with the Zabbix Server doesn’t really consume a lot of resources. It’s the ELK (Elasticsearch, Logstash, Kibana) stack that runs on Java 7 which will consume bulk of the resources on the virtual machine. We would highly recommend that you consider the following –

  • Use Zabbix + ELK In A Box for small environments i.e. 15-20 machines
  • Provision Zabbix + ELK In A Box on dedicated high performance storage e.g. FLASH Storage Disks
  • If you have larger monitoring requirements consider disabling the ELK (Elasticsearch, Kibana, Logstash) service and only running Zabbix on the virtual machine
  • If you have larger requirements for visualization, machine data collection and log file analysis we would highly recommend turning of Zabbix on the virtual machine and only running the ELK stack

We haven’t performed any major capacity planning exercises or performance validation exercises to validate the limits of what Zabbix + ELK In A Box can do at the recommended configuration i.e. 2 vCPU’s, 4 GB RAM. However, we would highly recommend that you use limit use of Zabbix + ELK In A Box to a small setup i.e. ~15-20 machines. For larger environments we would highly recommend designing and building your own scalable monitoring and log file analysis solution from scratch to meet your requirements.

To learn more about Zabbix + ELK In A Box head straight to – Sourceforge.net.


Trevor Warren (Linked In) loves hacking open source, designing innovative solutions and building trevor_warrencommunities. Trevor is inquisitive by nature, loves asking questions and some times does get into trouble for doing so. He’s passionate about certain things in life and building solutions that have the ability to impact people’s lives in a positive manner is one of them. He believes that he can change the world and is doing the little he can to change it in his own little ways. When not hacking open source, building new products, writing content for Practical Performance Analyst, dreaming up new concepts or building castles in the air, you can catch-him bird spotting (watching planes fly above his house).

Practical Performance Analyst as an Open Body Of Knowledge on Systems Performance Engineering (SPE) built + maintained by Trevor with the support of his army of volunteer elves (PPA Volunteers). You can reach trevor at –  trevor at practical performance analyst dot com. The views expressed on this web site are his own.

Related Posts

  • Using Performance Analytics, Forecasting & Prediction on Agile, DevOps ProjectsUsing Performance Analytics, Forecasting & Prediction on Agile, DevOps Projects Introduction - The move to digital over the last decade has brought about significant changes in the way most industries and business interact with their customers. The pace of change in most industries has been relatively high with the evolution of technology driving a lot of that […]
  • Detecting Anomalies that Matter!  Like needles in a haystackDetecting Anomalies that Matter! Like needles in a haystack As Netuitive's Chief Data Scientist, I am fortunate to work closely with some of the worlds' largest banks, telcos, and eCommerce companies. Increasingly the executives that I speak with at these companies are no longer focused on just detecting application performance anomalies - they […]
  • Lightweight Systems for Realtime Monitoring – Oreilly Lightweight Systems for Realtime Monitoring – Oreilly Lightweight Systems for Realtime Monitoring by author Sam Newman and published by Oreilly is a short book that provides a snapshot of the various Open Source light weight real time system monitoring tools available out there. This free ebook from Oreilly makes for quick reading and is in […]
  • Application Monitoring : Comparing OptionsApplication Monitoring : Comparing Options What happens when mission critical Java applications slow down or keep crashing in production? The vast majority of IT Operations (Ops) today bury their heads in log files. Why? because thats what they’ve been doing since IBM invented the mainframe. Diving into the weeds feels good, […]
  • Zubair Syed

    Open source is on its journey, very nice solution and really nicely written.

    • Thanks. Glad you liked the piece Zubair. Let me know if you are keen to share some of your experience through PPA so that other readers can benefit as well.