Monitor Warp 10

Sensision service is available for collecting Warp 10 and his host metrics.

The complete list of the metrics exposed by Warp 10 is available here. Below are a few examples of what you can monitor:

Host metrics: CPU, memory, disks, network, ...
Warp 10 metrics: number of datapoints fetched/deleted, number of Geo Time Series, ...

The easiest way to monitor, gather and analyze Warp 10 metrics is to use another Warp 10 instance. But Warp 10 can expose its metrics to Sensision, and Sensision can push them locally too, using two instances is not mandatory.

monitoring

Setup the monitoring Warp 10 instance

First, install a Warp 10 instance where you will do the monitoring, gathering the metrics from the main instance and analyzing them via WarpScript.

If needed, look at the getting started page for the detailed instructions to install Warp 10

Generate monitoring tokens

The monitoring metrics from the main Warp 10 will be pushed to the monitoring Warp 10 as any other GTS. As such, before pushing them, you need to generate both a READ and a WRITE token for the monitoring application.

To generate these tokens, refer to Token Management. If you want really simple tokens for tests, you can use manually defined tokens.

Installing Sensision

In order to gather the monitoring metrics from the main Warp 10, you only need to install and configure Sensision service in the main Warp 10 server.

In this documentation page, we are going to suppose that you're using Warp 10 Standalone. If you are using the distributed mode, you will need to install Sensision in all the Warp 10 nodes but ideally also on the Kafka and HBase servers.

Download Sensision archive on GitHub releases, untar the Sensision service file. For 1.0.24 release:

wget "https://github.com/senx/sensision/releases/download/1.0.24/sensision-service-1.0.24.tar.gz"
tar xvf sensision-service-1.0.24.tar.gz
sudo mkdir /opt/sensision
sudo mv sensision-1.0.24/* /opt/sensision
echo "java home is : $JAVA_HOME"

If you already have Warp 10 running, your JAVA_HOME environment variable should be already set. If undefined, you must set it in the Sensision init script sensision.init.

Set the SENSISION_HOME in sensision.init, then execute start sensision to deploy default configuration.

sudo sed -i "s|#SENSISION_HOME=/opt/sensision-\${VERSION}|SENSISION_HOME=/opt/sensision|g" /opt/sensision/bin/sensision.init
(cd /opt/sensision && sudo /opt/sensision/bin/sensision.init bootstrap)

Bootstrap will create the folder /var/run/sensision and all the stuff surrounding it. Then the init script tries to start the Sensision service with a configuration by default, and stops with 'WARP10_CONF' is not defined error.

Sensision has been only partially initialized, as it lacks the information on the Warp 10 instance it must push the metrics to. You need to edit /opt/sensision/etc/sensision.conf configuration file to set it.

Look at the Default queue part of the configuration file, set sensision.qf.token.default with you write token, and set sensision.qf.url.default to the Update endpoint of your monitoring Warp 10 instance.

#
# Default queue
#

sensision.qf.url.default=http://127.0.0.1:8080/api/v0/update
sensision.qf.token.default=SensisionW
sensision.qf.topn.default=250
sensision.qf.period.default=1000
sensision.qf.batchsize.default=100000
#sensision.qf.proxy.host.default=
#sensision.qf.proxy.port.default=
#sensision.dedup.maxsize.default=xxx
#sensision.dedup.maxage.default=xxx

Sensision install a /etc/init.d/sensision rc script. You can now launch it with:

sudo service sensision start

Note systemd also wrap rc scripts, so sudo systemctl start sensision.service and sudo journalctl -f -u sensision also works out of the box.

If everything is ok, you can see that a Sensision target file for your main Warp 10 instance is created on /var/run/sensision/targets/:

root@XXX:ls -al /var/run/sensision/targets/
total 8
drwx-wx-wt  2 XXX XXX 4096 sept. 27 09:52 .
drwxr-xr-x 10 XXX XXX 4096 juil. 21 16:29 ..
-rw-rw-r--  1 XXX XXX    0 sept. 27 09:52 7ffffea13dad1efc.60000.af4ec5da-f721-493d-aaa0-25b3b8e70d05.38261.target
-rw-rw-r--  1 XXX XXX    0 sept. 27 09:52 7ffffea13dad4e0c.60000.f519167d-c4f7-435f-ae8e-2b87f67b102f.42893.warp.target

In this example, the target file for your Warp 10 is 7ffffea13dad4e0c.60000.f519167d-c4f7-435f-ae8e-2b87f67b102f.42893.warp.target. The number before .warp.target is the port opened by Warp 10 to expose its metrics, you can manually poll it to see if it is working:

root@XXX:curl 127.0.0.1:42893/metrics
1506498948335000// warp.revision{component=standalone} '1.2.10'
1506498948335000// warp.script.mobius.sessions.scheduled{} 0
1506498948335000// warp.script.repository.macros{} 0
1506498948335000// warp.script.repository.jars{} 0
1506498948335000// warp.script.bootstrap.loads{} 6
1506498948335000// warp.script.function.count{function=DEF} 6
1506498948336000// warp.directory.gts{} 458810
1506498948336000// warp.script.function.time.us{function=DEF} 324

Check the Sensision logs to see if everything is OK.

tail -f /opt/sensision/logs/sensision.log

Using the metrics

The complete list of the metrics exposed by Warp 10 is on GitHub.

You can manipulate those metrics in any way you want. Some suggestions:

Use WarpView custom elements and some simple WarpScript FETCH to get and plot the last values of the metrics you want to monitor.
Use Discovery to build nice custom dashboards from within WarpScript.
Use the Grafana Warp 10 plugin and some WarpScript code to build a fully featured monitoring dashboard.
Make your favorite monitoring tool do some WarpScript requests (they are simple HTTP requests, almost any tool is able to do it) to get the metrics and add them to your favorite dashboard or monitoring tool.

Setup the monitoring Warp 10 instance
Installing Sensision
Using the metrics