We’ve been running HBase at Biz360 for about six months, but it worked so smoothly at first that I never did much tuning. I’ve recently increased the volume of data we’re storing by about 300%, and have started running into some problems like blocks going missing. I found a good way to monitor all my servers logs to get an idea what’s going on today, and it’s so simple I wanted to share.

for x in 01 02 03 04 05 06 07 08 09 10 ; do
    server=prod-hbase$x
    echo === $server ===
    ssh $server tail -30f '/app/hbase/logs/*region*.log'
done

It does a tail -f on each server’s log in turn. To move to the next server, just hit ctrl-c.

@squarecog suggests

for server in `cat hbase_servers.txt`; do ...