i testing elk stack on 1 ubuntu 14.04 box. has 6 gb of ram , 1tb storage. pretty modest, amount of data getting, should plenty right? followed guide elk stack guide. in summary, have kibana4, logstash 1.5, , elasticsearch 1.4.4 running on 1 box, nginx server acting reverse proxy can access kibana outside. main difference guide instead of syslogs, taking json input logstash-forwarder, sending 300 events/minute.
once started, fine -- logs show on kibana , there no errors. after 3 hours, elasticsearch crashes.
discover: cannot read property 'indexof' of undefined error on site. logs can seen on pastebin. seems shards become inactive , elasticsearch updates index_buffer size.
if refresh kibana ui, starts working again json logs. however, if test different log source (using tcp input instead of lumberjack), similar errors above, except stop processing logs -- anywhere 10 min hour, not process more logs , cannot stop logstash unless perform kill -kill.
killing logstash (pid 13333) sigterm waiting logstash (pid 13333) die... waiting logstash (pid 13333) die... waiting logstash (pid 13333) die... waiting logstash (pid 13333) die... waiting logstash (pid 13333) die... logstash stop failed; still running. logstash error log shows . logstash .log file empty...
for tcp input, 1500 events every 15 minutes, in bulk insert process logstash.
any ideas here?
edit: observed when starting elasticsearch process, shards set lower mb...
[2015-05-08 19:19:44,302][debug][index.engine.internal ] [eon] [logstash- 2015.05.05][0] updating index_buffer_size [64mb] [4mb]
@jeffrey, have same problem dns filter.
i did 2 things. installed dnsmasq dns caching resolver. if have high latency or high load dns server.
and second increased number of worker threads of logstash. use -w option.
trick threads working without dnsmasq. trick dnsmask without threads not.
Comments
Post a Comment