For some reason my instance of Juggernaut 2, keeps crashing.  I have not figured out why, it just crashes and from what I can tell it doesn’t log why.  Though I haven’t found where it logs to yet either. That is another issue at the moment.  This post describes how to set up Monit to monitor Juggernaut 2, and restart it if it crashes. My machine is Debian Lenny, but this can be tweaked to work with other linux distros as well.

  1. Install Monit
  2. wget http://mmonit.com/monit/dist/monit-5.2.5.tar.gz
  3. tar -zxvf monit-5.2.5.tar.gz
  4. cd monit-5.2.5
  5. ./configure
  6. make
  7. sudo make install
  8. monit -V  # should return version you installed
  9. mkdir /etc/monit
  10. cd /etc/monit
  11. nano monitrc
  12. Paste this code:
###############################################################################
## Monit control file
###############################################################################
##
## Comments begin with a '#' and extend through the end of the line. Keywords
## are case insensitive. All path's MUST BE FULLY QUALIFIED, starting with '/'.
##
## Below you will find examples of some frequently used statements. For
## information about the control file, a complete list of statements and
## options please have a look in the monit manual.
##
##
###############################################################################
## Global section
###############################################################################
##
## Start monit in the background (run as a daemon) and check services at
## 2-minute intervals.
#
# set daemon  120
#
#
## Set syslog logging with the 'daemon' facility. If the FACILITY option is
## omitted, monit will use 'user' facility by default. If you want to log to
## a stand alone log file instead, specify the path to a log file
#
# set logfile syslog facility log_daemon
#
#
## Set the list of mail servers for alert delivery. Multiple servers may be
## specified using comma separator. By default monit uses port 25 - this
## is possible to override with the PORT option.
#
# set mailserver mail.bar.baz,               # primary mailserver
#                backup.bar.baz port 10025,  # backup mailserver on port 10025
#                localhost                   # fallback relay
#
#
## By default monit will drop alert events if no mail servers are available.
## If you want to keep the alerts for a later delivery retry, you can use the
## EVENTQUEUE statement. The base directory where undelivered alerts will be
## stored is specified by the BASEDIR option. You can limit the maximal queue
## size using the SLOTS option (if omitted, the queue is limited by space
## available in the back end filesystem).
#
# set eventqueue
#     basedir /var/monit  # set the base directory where events will be stored
#     slots 100           # optionaly limit the queue size
#
#
## Monit by default uses the following alert mail format:
##
## --8<--
## From: monit@$HOST                         # sender
## Subject: monit alert --  $EVENT $SERVICE  # subject
##
## $EVENT Service $SERVICE                   #
##                                           #
##  Date:        $DATE                   #
##  Action:      $ACTION                 #
##  Host:        $HOST                   # body
##  Description: $DESCRIPTION            #
##                                           #
## Your faithful employee,                   #
## monit                                     #
## --8<--
##
## You can override this message format or parts of it, such as subject
## or sender using the MAIL-FORMAT statement. Macros such as $DATE, etc.
## are expanded at runtime. For example, to override the sender:
#
# set mail-format { from: monit@foo.bar }
#
#
## You can set alert recipients here whom will receive alerts if/when a
## service defined in this file has errors. Alerts may be restricted on
## events by using a filter as in the second example below.
#
# set alert sysadm@foo.bar                       # receive all alerts
# set alert manager@foo.bar only on { timeout }  # receive just service-
#                                                # timeout alert
#
#
## Monit has an embedded web server which can be used to view status of
## services monitored, the current configuration, actual services parameters
## and manage services from a web interface.
#
# set httpd port 2812 and
#     use address localhost  # only accept connection from localhost
#     allow localhost        # allow localhost to connect to the server and
#     allow admin:monit      # require user 'admin' with password 'monit'
#
#
###############################################################################
## Services
###############################################################################
##
## Check general system resources such as load average, cpu and memory
## usage. Each test specifies a resource, conditions and the action to be
## performed should a test fail.
#
#  check system myhost.mydomain.tld
#    if loadavg (1min) > 4 then alert
#    if loadavg (5min) > 2 then alert
#    if memory usage > 75% then alert
#    if cpu usage (user) > 70% then alert
#    if cpu usage (system) > 30% then alert
#    if cpu usage (wait) > 20% then alert
#
#
## Check a file for existence, checksum, permissions, uid and gid. In addition
## to alert recipients in the global section, customized alert will be sent to
## additional recipients by specifying a local alert handler. The service may
## be grouped using the GROUP option.
#
#  check file apache_bin with path /usr/local/apache/bin/httpd
#    if failed checksum and
#       expect the sum 8f7f419955cefa0b33a2ba316cba3659 then unmonitor
#    if failed permission 755 then unmonitor
#    if failed uid root then unmonitor
#    if failed gid root then unmonitor
#    alert security@foo.bar on {
#           checksum, permission, uid, gid, unmonitor
#        } with the mail-format { subject: Alarm! }
#    group server
#
#
## Check that a process is running, in this case Apache, and that it respond
## to HTTP and HTTPS requests. Check its resource usage such as cpu and memory,
## and number of children. If the process is not running, monit will restart
## it by default. In case the service was restarted very often and the
## problem remains, it is possible to disable monitoring using the TIMEOUT
## statement. This service depends on another service (apache_bin) which
## is defined above.
#
#  check process apache with pidfile /usr/local/apache/logs/httpd.pid
#    start program = "/etc/init.d/httpd start"
#    stop program  = "/etc/init.d/httpd stop"
#    if cpu > 60% for 2 cycles then alert
#    if cpu > 80% for 5 cycles then restart
#    if totalmem > 200.0 MB for 5 cycles then restart
#    if children > 250 then restart
#    if loadavg(5min) greater than 10 for 8 cycles then stop
#    if failed host www.tildeslash.com port 80 protocol http
#       and request "/monit/doc/next.php"
#       then restart
#    if failed port 443 type tcpssl protocol http
#       with timeout 15 seconds
#       then restart
#    if 3 restarts within 5 cycles then timeout
#    depends on apache_bin
#    group server
#
#
## Check device permissions, uid, gid, space and inode usage. Other services,
## such as databases, may depend on this resource and an automatically graceful
## stop may be cascaded to them before the filesystem will become full and data
## lost.
#
#  check device datafs with path /dev/sdb1
#    start program  = "/bin/mount /data"
#    stop program  = "/bin/umount /data"
#    if failed permission 660 then unmonitor
#    if failed uid root then unmonitor
#    if failed gid disk then unmonitor
#    if space usage > 80% for 5 times within 15 cycles then alert
#    if space usage > 99% then stop
#    if inode usage > 30000 then alert
#    if inode usage > 99% then stop
#    group server
#
#
## Check a file's timestamp. In this example, we test if a file is older
## than 15 minutes and assume something is wrong if its not updated. Also,
## if the file size exceed a given limit, execute a script
#
#  check file database with path /data/mydatabase.db
#    if failed permission 700 then alert
#    if failed uid data then alert
#    if failed gid data then alert
#    if timestamp > 15 minutes then alert
#    if size > 100 MB then exec "/my/cleanup/script"
#
#
## Check directory permission, uid and gid.  An event is triggered if the
## directory does not belong to the user with uid 0 and gid 0.  In addition,
## the permissions have to match the octal description of 755 (see chmod(1)).
#
#  check directory bin with path /bin
#    if failed permission 755 then unmonitor
#    if failed uid 0 then unmonitor
#    if failed gid 0 then unmonitor
#
#
## Check a remote host network services availability using a ping test and
## check response content from a web server. Up to three pings are sent and
## connection to a port and a application level network check is performed.
#
#  check host myserver with address 192.168.1.1
#    if failed icmp type echo count 3 with timeout 3 seconds then alert
#    if failed port 3306 protocol mysql with timeout 15 seconds then alert
#    if failed url
#       http://user:password@www.foo.bar:8080/?querystring
#       and content == 'action="j_security_check"'
#       then alert
#
#
###############################################################################
## Includes
###############################################################################
##
## It is possible to include additional configuration parts from other files or
## directories.
#
  include /etc/monit/config/*
#
#
  1. Next create a config file for juggernaut
  2. sudo nano /etc/monit/config/juggernaut
  3. paste in the below code
set daemon 15
set logfile /var/log/monit.log

set mailserver smtp.gmail.com port 587
        username "you@gmail.com" password "password"
        using tlsv1
        with timeout 30 seconds
set alert you@gmail.com

set httpd port 2613 and
        use address localhost
        allow admin:admin

check host juggernaut with address 127.0.0.1
    start program = "/usr/local/bin/node /usr/local/lib/node_modules/juggernaut/server.js >> /var/log/juggernaut.log"
    stop program  = "/usr/bin/pkill -f 'node /usr/local/lib/node_modules/juggernaut/servers.js'"
    if failed port 8080 protocol HTTP
        request /application.js
        with timeout 10 seconds
 then restart

Setting up Monit to start automatically

  1. sudo nano /etc/init.d/monit
  2. paste in below start up script
#!/bin/sh
### BEGIN INIT INFO
# Provides:          monit
# Required-Start:    $remote_fs
# Required-Stop:     $remote_fs
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6

PATH=/sbin:/bin:/usr/sbin:/usr/bin
DAEMON=/usr/local/bin/monit
startup=1
CONFIG="/etc/monit/monitrc"
DELAY="/etc/monit/monit_delay"
CHECK_INTERVALS=15
# We default to 180s (3min) check intervals
NAME=monit
DESC="daemon monitor"

set -e

# Check if DAEMON binary exist
test -f $DAEMON || exit 0

if [ -f "/etc/default/monit" ]; then
     . /etc/default/monit
fi

ARGS="-d $CHECK_INTERVALS -c $CONFIG -s /var/lib/monit/monit.state"

monit_not_configured () {
    printf "%b\n" "monit won't be started/stopped\n\tunless it it's configured"
    if [ "$1" != "stop" ]
        then
        printf "%b\n" "\tplease configure monit and then edit /etc/default/monit"
        printf "%b\n" "\tand set the \"startup\" variable to 1 in order to allow "
        printf "%b\n" "\tmonit to start"
    fi
    exit 0
}

monit_check_config () {
    # Check for emtpy config, probably default configfile.
    if [ "`grep -s -v \"^#\" $CONFIG`" = "" ]; then
        echo "empty config, please edit $CONFIG."
        exit 0
    fi
}

monit_check_perms () {
    # Check the permission on configfile.
    # The permission must not have more than -rwx------ (0700) permissions.

    # Skip checking, fix perms instead.
    /bin/chmod go-rwx $CONFIG

}

monit_delayed_monitoring () {
    if [ -x $DELAY ]; then
      $DELAY &
    elif [ -f $DELAY ]; then
      echo
      echo "[WARNING] A delayed start file exists ($DELAY) but it is not executable."
    fi
}

monit_check_syntax () {
  $DAEMON -t;
#  if [ $? ] ; then
#      echo "syntax good"
#  else
#      echo "syntax bad"
#  fi
}

monit_checks () {
    # Check if startup variable is set to 1, if not we exit.
    if [ "$startup" != "1" ]; then
        monit_not_configured $1
    fi
    # Check for emtpy configfile
    monit_check_config
    # Check permissions of configfile
    monit_check_perms
}

case "$1" in
  start)
    echo -n "Starting $DESC: "
    monit_checks $1
    echo -n "$NAME"
    start-stop-daemon --start --quiet --pidfile /var/run/$NAME.pid \
        --exec $DAEMON > /dev/null 2>&1 -- $ARGS
   monit_delayed_monitoring
    echo "."
    ;;
  stop)
    echo -n "Stopping $DESC: "
    #monit_checks $1
    echo -n "$NAME"
    start-stop-daemon --retry 5 --oknodo --stop --quiet --pidfile /var/run/$NAME.pid \
        --exec $DAEMON  > /dev/null 2>&1
    echo "."
    ;;
  restart|force-reload)
    $0 stop
    $0 start
    ;;
  syntax)
   monit_check_syntax
   ;;
  *)
    N=/etc/init.d/$NAME
    echo "Usage: $N {start|stop|restart|force-reload|syntax}" >&2
    exit 1
    ;;
esac

exit 0
  1. sudo chmod +x /etc/init.d/monit
  2. sudo update-rc.d -f monit defaults
  3. sudo /etc//init.d/monit start
  4. That’s it!

Now, with this setup Juggernaut will not start until monit has started and sees that it is not running and then it will start it. You will get a few alerts indicating this. If you want to start Juggernaut on server start, you can create a init.d script to do so.

Now on to why node js gets killed on my server at random.