OpenStack :: Logstash Elasticsearch Kibana (on apache)

January 16th, 2014

OpenStack has lots of moving parts. One of the challenges in administering a cluster is sifting through all the logs on the multiple nodes that make up a cluster. I help to administer TryStack and am always looking for tools to help managing this cluster go smoother.

I was introduced yesterday to Logstash + elasticsearch + Kibana. I’m not really sure which of these names to call what I was introduced to, I guess all of them. Idea is that all the logs from all the nodes are sent to a central location so that you can filter through them. I think there’s much more advanced usage of this trio. I’m still figuring out what to do with it beyond basic log searching.

My understanding is that Logstash helps to gather the logs, elasticsearch indexes them and kibana is the webui that queries elasticsearch. Here’s a screenshot of what I ended up with.

Logstash + Elasticsearch + Kibana

 

My co-worker Kambiz (kam-beez) pointed me to a couple links that he used to setup an instance of this and he more-or-less followed this post:

http://blog.basefarm.com/blog/how-to-install-logstash-with-kibana-interface-on-rhel/

The main modifications to what he ended up with was to pull the logstash rpm from the logstash Jenkins instance:  http://build.logstash.net/view/logstash/
Then to collect logs using this link’s method: http://cookbook.logstash.net/recipes/rsyslog-agent/
And finally there were a couple config changes to what the original post provided to get logstash running.

I already had apache running on the TryStack utility hosts and didn’t think it was nessesary to add nginx to the mix, which is what the post uses, so I figured it may be helpful to document the process I went through to get this running on apache. This install that has been very useful thus far and I’m glad I have it collecting logs.

First, get the rpms from http://www.elasticsearch.org/download and the logstash jenkins instance. I used these two links like this:

[root@host1 ~]# yum install https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-0.90.10.noarch.rpm
[root@host1 ~]# yum install http://build.logstash.net/view/logstash/job/logstash-daily/79/artifact/pkg/logstash-1.3.2-1_centos.201401070105.5cd5b2e.noarch.rpm

also get a copy of the latest kibana stuff, this is just html and javascript so I don’t think there is an rpm afaict. I unpacked it and moved it to /var/www.

[root@host1 ~]# wget https://download.elasticsearch.org/kibana/kibana/kibana-latest.tar.gz
[root@host1 ~]# tar xzf kibana-latest.tar.gz
[root@host1 ~]# mv kibana-latest /var/www
[root@host1 ~]# restorecon -R /var/www/kibana-latest

make sure that apache is installed too, I already had it installed from my Foreman and Nagios instances running on this server. Now lets start to configure this stuff. Start with Kibana. Edit /var/www/kibana-latest/config.js and update the elasticsearch: line:

- elasticsearch: "http://"+window.location.hostname+":9200",
+ elasticsearch: "http://yourhostname.com/elasticsearch",

Note that the 9200 in the config.js file is dropped and replaced with /elasticsearch. When you fire up kibana in apache it will try to connect directly to elasticsearch on 9200. To avoid having to punch extra holes in the firewall we’ll setup a proxypass in apache to pass the traffic from yourhostname.com:80/elasticsearch to localhost:9200. We’ll configure apache once we finish kibana.

Before we get to apache back up the kibana-latest/app/dashboards/default.json file (if you want to) and replace it with my copy:

[root@host1 ~]# cd /var/www/kibana-latest/app/dashboards
[root@host1 dashboards]# cp default.json default.json.backup
[root@host1 dashboards]# wget http://www.jaddog.org/wp-content/uploads/2014/default.json

edit that file to have a title to your liking, I used “TryStack :: OpenStack LogStash Search”

 - "title": "TryStack :: OpenStack LogStash Search",
 + "title": "Your ingenious title here",

The default.json file is a definition of what panels to put on your default view in kibana. I modified the one referenced on the other blog post, that’s why I gave you a new link instead of using the one on that post. There were a couple redundant panels that I consolidated. Also the timespan it referenced by default was old and static so I changed it to show the last hour by default.

So lets add that apache config now. Add /etc/httpd/conf.d/elasticsearch.conf. You could call this whatever.conf if you wanted to. I put both my elasticsearch proxy pass and my kibana alias in this file like this:

ProxyPass /elasticsearch http://localhost:9200
ProxyPassReverse /elasticsearch http://localhost:9200
alias /kibana /var/www/kibana-latest

<Location /elasticsearch>
    Order allow,deny
    Allow from all
</Location>

For this to work you’ll need mod_proxy and mod_proxy_http and if you’re using selinux you’ll need the httpd_can_network_connect bool set on. Google those if you’re not sure how to set them up. There’s lots of docs out there about them. Finally lets configure logstash. First edit /etc/sysconfig/logstash and set the START to true instead of false. The service won’t start if you don’t.

Next create /etc/logstash/conf.d/logstash.conf with this content:

input {
  syslog {
    type => syslog
    port => 5544
  }
}

filter {
  mutate {
    add_field => [ "hostip", "%{host}" ]
  }
}

output {
  elasticsearch {
    host => "localhost"
  }
}

Last open up your firewall to allow your hosts to send rsyslog messages to port 5544. I added an iptables rule to /etc/sysconfig/iptables and restarted the firewall: (note: the minus in this is intended to be part of the line and does not indicate you should remove it)

-A INPUT -i em1 -m state --state NEW -m tcp -p tcp --dport 5544 -j ACCEPT

This rule listens on the em1 interface, my internal network. Easiest way to do this for your host is edit that file, copy the rule for port 22 and update the duplicated line to accept port 5544 instead of 22. Then restart iptables.

*** IMPORTANT *** there are security implications to opening this port. Please do not open this port to the world and allow anyone to pollute your logs. I’ve opened mine up only to my internal network for my cluster. You should also restrict traffic somehow to that only the hosts you expect to get logs from can connect to this port.

Finally fire it all up:

[root@host1 ~]# service elasticsearch start
[root@host1 ~]# service logstash start
[root@host1 ~]# service httpd start

This should give you a pretty uninteresting kibana interface. There won’t be any logs in it yet. Key here is to watch the top of the page and make sure that there isn’t a message that kibaba can’t connect to elasticsearch. If it can’t visit yourhostname.com/elasticsearch and make sure that you get a 200 back.

To populate with logs you could use the logstash client, but the logstash cookbook post referenced above suggests it’s a bit heavy weight. I’ve had really good results thus far just having rsyslog send over the logs. To do that, on each of the hosts that you want to aggregate logs you’ll need to create the file /etc/rsyslog.d/logstash.conf with this content:

*.* @@your_logstash_host:5544

This will send ALL LOGS from that host to logstash for indexing. Google rsyslog if you would like to find out how not to send all logs to logstash.

Once you start to see logs flow into the web interface you can change the time span in the upper right hand corner. You can query your logs. Try putting a hostname or ip into the query box at the top. You can use wildcards like *error* to find errors. You can layer filters. Try adding a filter for a host in the filter box just under the query box. Then add another one for *error* and you’ll get the errors for just that host within the timespan you’ve chosen.

Hope this helps you track down issues. I immediately found I had a rouge iscsi session on one of my compute nodes and was able to put it out of its misery. 🙂

*** Update Jan 20 ***

I noticed that not all the OpenStack logs were showing up in my logstash searches. Turns out you can toggle openstack using syslog. My puppet configs turn it off by default so I had to turn it on on all my hosts. This boils down to setting ‘use_syslog = true’ in all your component’s conf files. Here’s a link that talks more about it:

http://docs.openstack.org/trunk/openstack-ops/content/logging_monitoring.html

Leave a Reply

You must be logged in to post a comment.