Getting started

This page describes how to install and interact with a standalone version of the Warp 10 platform. It also describes WarpScript IDE tools.

If you are not interested in installing your own platform but only want to test it, instead try out the free sandbox where you can get your hands on in no time.

The standalone version of the Warp 10 platform uses LevelDB as its storage layer, this version is suitable for managing a few tens of millions of time series and few hundreds of billions of datapoints. If you have larger needs, a distributed version of the platform is also available which uses Apache HBase as a scalable storage layer.

Warp 10 is a server. It uses http requests on different endpoints, one for each task. In this introduction, we will use only two endpoints:

• Ingress https://host:8080/api/v0/update: for sending data to the Warp 10 platform
• Egress https://host:8080/api/v0/exec: for executing WarpScript code

Once you get a Warp 10 instance running with valid tokens, you do not need physical or ssh access to the server machine anymore to work. Everything could be done with these two endpoints.

In this very first tutorial, you will learn to:

• start Warp 10
• push data
• read these data in a WarpScript
• use WarpStudio as a WarpScript IDE
• use Visual Studio Code as a WarpScript IDE

Step 1. Download Warp 10

Go to download page, download the latest tar.gz from GitHub.

If you run windows, you can install Warp 10 with docker. Follow docker instruction, then jump to Step 6.

All commands below are for an Ubuntu test machine. It could be a virtual machine.

Step 2. Extract Warp 10

Create /opt/warp10, extract the downloaded archive, and go into bin directory.

sudo mkdir -p /opt/warp10
tar xf warp10-X.Y.Z.tar.gz
sudo mv warp10-X.Y.Z/* /opt/warp10/
cd /opt/warp10/bin

Step 3. Install Java 8

You must have a java 8 JRE installed.

sudo apt update
sudo apt install openjdk-8-jre-headless

On some very recent Debian distribution, you may not have java 8 package available. You can use Eclipse Temurin JRE:

curl --silent -LO https://github.com/adoptium/temurin8-binaries/releases/download/jdk8u312-b07/OpenJDK8U-jre_x64_linux_hotspot_8u312b07.tar.gz
curl --silent -LO https://github.com/adoptium/temurin8-binaries/releases/download/jdk8u312-b07/OpenJDK8U-jre_x64_linux_hotspot_8u312b07.tar.gz.sha256.txt
sha256sum -c OpenJDK8U-jre_x64_linux_hotspot_8u312b07.tar.gz.sha256.txt
tar xf OpenJDK8U-jre_x64_linux_hotspot_8u312b07.tar.gz -C /opt
rm OpenJDK8U-jre_x64_linux_hotspot_8u312b07.tar.gz OpenJDK8U-jre_x64_linux_hotspot_8u312b07.tar.gz.sha256.txt
export PATH="$PATH:/opt/jdk8u312-b07-jre/bin"
export JAVA_HOME="/opt/jdk8u312-b07-jre"

The Warp 10 installer will try to detect automatically your Java path. If you have a custom path or several versions of java installed, you have to define JAVA_HOME explicitly.

Check your default java version:

java -version

If the answer is java 1.8.x_xxx, you can jump to Step 4

If you have several versions of java installed on your system, make sure Warp 10 will use Java 8. Possible methods are:

• You can update alternatives on your system with sudo update-alternatives --config java
• edit /opt/warp10/bin/warp10-standalone.sh with your favorite text editor, and replace the commented line by your JVM path
• use command line and adapt the path to your configuration:

sed -i 's/#JAVA_HOME.*/JAVA_HOME=\/usr\/lib\/jvm\/java-8-openjdk-amd64/' warp10-standalone.sh

Step 4. Bootstrap Warp 10

This will prepare for you the environment of the Warp 10 platform:

• Create the warp10 user if needed
• Generate the standalone configuration file from template
• Prepare the directory hierarchy
• Set file permissions

Once bootstrapped, you cannot move Warp 10 files easily. Choose wisely before you do bootstrap. Path /opt/warp10 is a not mandatory, just a best practice.

Start bootstrap as root:

sudo ./warp10-standalone.init bootstrap

Bootstrap should end on message Warp 10 config has been generated here: followed by the path of Warp 10 config files.

Then change to warp10 user:

sudo su warp10

While discovering Warp 10, you will hit some restrictions (number of data in a FETCH, maximum number of operations). All the Warp 10 configurations are in etc/conf.d/.

Step 5. Run Warp 10

As warp10 user:

./warp10-standalone.sh start

You may also launch Warp 10 as root with sudo ./warp10-standalone.init start

At the first start, this script will create the LevelDB database and all the stuff surrounding it. An initial set of tokens will be provided and some useful command examples to start playing with your Warp 10 instance.

Useful resources:

• Warp 10 standalone config .conf files has been generated in the etc/conf.d/ directory
• Initials tokens are available in the initial.tokens file in the etc directory
• Logs are available in the logs directory
• Data are stored via LevelDB in the leveldb directory

Snapshots of LevelDB data can be performed via the init script. More information here.

Step 6. Generate Warp 10 tokens

The Warp 10 platform is built with a robust security model that allows you to have a tight control on who has the right to write and/or read data. The model is structured around the concepts of data producer, data owner, application, within WRITE and READ tokens.

By default, at the first start, Warp 10 init script will generate both a READ and a WRITE token for the test application io.warp10.bootstrap for a test user that is both the producer and the owner of the data. You will find these in etc/initial.tokens. Use these tokens for getting started, you have one year from now before they expire.

Later, you can generate your own WRITE and READ tokens by executing the TOKENGEN command, or extend the life of existing tokens. Refer to the token management documentation.

Step 7. Prepare data in GTS data format

Data is sent into the platform via HTTP POST requests to the Warp 10 API.

API Endpoint

The HTTP endpoint used to send data is http(s)://host:port/api/vX/update, where vX is the version of the API you want to use (currently v0). In order to be accepted by the platform, requests to this endpoint need to be authenticated, by using a X-Warp10-Token HTTP header with your write token.

Data format

If you have your own data, you can now convert them into GTS format. If you do not have, you can use our example datasets.

• One GTS is one time series.
• A GTS is defined by its CLASS and LABELS value.

Knowing these facts, up to your use cases, you must choose wisely what you will put in names and labels.

Data is sent in the body of the POST request, one data point per line. Each line follows the GTS input format:

TS/LAT:LON/ELEV CLASS{LABELS} VALUE

Where:

TS	Optional Timestamp of the reading, in microseconds since the Unix Epoch. If omitted, the timestamp associated to the data point will be the one of the platform when the point is pushed.
LAT:LON	Optional geographic coordinates of the reading, using WGS84
ELEV	Optional elevation of the reading, in millimeters
CLASS	Class name of the reading as a URL encoded UTF-8 character string. The encoding of character { (Unicode LEFT CURLY BRACKET, 0x007B) is MANDATORY.
LABELS	Comma separated list of labels, using the syntax key=value where both key and value are URL encoded UTF-8 character strings. If a key or value contains , (Unicode COMMA, 0x002C), } (Unicode RIGHT CURLY BRACKET, 0x007D) or = (Unicode EQUALS SIGN, 0x003D), those characters MUST be encoded.
VALUE	The value of the reading. It can be of one of four types: LONG, DOUBLE (you must have a dot decimal separator), BOOLEAN (character T or F), STRING (anything url encoded between single quotes). Since Warp 10 2.1, binary or multi value is also possible.

Example of valid input lines:

#latitude, longitude, elevation.
959860945800000/37.399386:-3.071806/214748 engine.torque{vehicletype=x21,serial=VF451112448F45B} 206

#no timestamp, no elevation
/48.0:-4.5/ bar{label0=val0} 3.14

#LONG type
// aa{} -4521
#DOUBLE type
// bb{} -4521.0
#BOOLEAN type
// cc{} T
#STRING type
// dd{} 'it%27s ok'

#Since Warp 10 2.1, you can also ingest:
#binary type, base64
// ee{} b64:aGVsbG8gV2FycCAxMAo=
#multi value type
// ff{} [ 4 4.5 6 7 -2.2 ]

Performance tip: Gather your data together. If you do one ingress call per line, the http overhead will be huge. Ingress size has no limit. You can curl a 200 GB file, the ingestion rate will be very high. Gathering your data per channel will be even faster, because you don't need to repeat the class name and labels on each line. Just start the line with '='.

#no latitude, no longitude, no elevation, same channel for the following lines.
#starting a line with "=" means "keep previous names and labels from previous line".
1521444669000000// tool_speed{plant=paris,machine=47} 104.392492
=1521444669000020// 105.240093
=1521444669000040// 105.336131
=1521444669000060// 106.168065

Data model example

This is a car maker example: Each vehicle records time, position, vehicle speed (km/h), engine rotation speed (rpm), and engine torque (Nm). You plan to look for the time a customer spends on the second gear ratio, and try several models of gear wear, up to torque applied. You can choose to have three different class names:

• VehicleSpeed
• EngineRpm
• EngineTorque

And labels that will help you to identify the GTS:

• VehicleSerialNumber
• GearboxType

Because you know the formula will depend on the gearbox, and the gearbox type will never change in the vehicle life. Adding gearbox type as a label will allow you later to select only this gearbox type to perform a computation.

Your resulting GTS text file will look like:

959861036900000/37.399192:-3.071124/214748 VehicleSpeed{VehicleSerialNumber=VF0000000001,GearboxType=PKZ} 104.59757588
959861037000000/37.399192:-3.071124/214748 EngineRpm{VehicleSerialNumber=VF0000000001,GearboxType=PKZ} 1005.25
959861037000000/37.399192:-3.071124/214748 EngineTorque{VehicleSerialNumber=VF0000000001,GearboxType=PKZ} 225
959861046900000/37.399192:-3.071124/214748 VehicleSpeed{VehicleSerialNumber=VF0000000001,GearboxType=PKZ} 103.145
959861047000000/37.399192:-3.071124/214748 EngineRpm{VehicleSerialNumber=VF0000000001,GearboxType=PKZ} 999.0
959861047000000/37.399192:-3.071124/214748 EngineTorque{VehicleSerialNumber=VF0000000001,GearboxType=PKZ} 200

Step 8. Pushing data

In the default configuration, Warp 10 listens on 127.0.0.1 only.

If you plan to access it from another machine on your network, you must change standalone.host to listen on any interface (0.0.0.0 on Linux). Edit config file etc/conf.d/00-warp.conf with your favorite text editor, and restart Warp 10.

Pushing data with curl

If you're on a *nix system, you can push the data using cURL. The Warp 10 platform listens to the port 8080 of your container. In the setup we have mapped this port to the port 8080 of the host machine, so you can make your request to 127.0.0.1:8080.

POST /api/v0/update HTTP/1.1
Host: host
X-Warp10-Token: wsedrfg.53ef32.detgertyyhefwe5t677456rrfhgbfyhgherdtyreeryer
Content-Type: text/plain

1380475081000000// foo{label0=val0,label1=val1} 123
/48.0:-4.5/ bar{label0=val0} 3.14
1380475081123456/45.0:-0.01/10000000 foobar{label1=val1} T

To test the running container, push a single GTS containing one data in the platform using your WRITE token. You will find initial tokens in etc/initial.tokens.

As any user, or maybe on another machine.

curl -v -H 'X-Warp10-Token: 0RJlPOj4g7SbicLhRq8AAR3egfhMeoJFy7dXm_RtDz5LUGSx3HGlBYvSwa7i_0KIxEqjp9JUPHv4bgBIXJ0TfUyfmvsvQsnD3pNQJ6J3AtoBc77ZiWH.ZF' --data-binary "1// test{} 42" 'http://127.0.0.1:8080/api/v0/update'

If everything is OK, you should receive a HTTP 200

Create a text file with your favorite editor, with more lines, save it as "example.gts":

959861036900000/37.399192:-3.071124/214748 VehicleSpeed{VehicleSerialNumber=VF0000000001,GearboxType=PKZ} 104.59757588
959861037000000/37.399192:-3.071124/214748 EngineRpm{VehicleSerialNumber=VF0000000001,GearboxType=PKZ} 1005.25
959861037000000/37.399192:-3.071124/214748 EngineTorque{VehicleSerialNumber=VF0000000001,GearboxType=PKZ} 225
959861046900000/37.399192:-3.071124/214748 VehicleSpeed{VehicleSerialNumber=VF0000000001,GearboxType=PKZ} 103.145
959861047000000/37.399192:-3.071124/214748 EngineRpm{VehicleSerialNumber=VF0000000001,GearboxType=PKZ} 999.0
959861047000000/37.399192:-3.071124/214748 EngineTorque{VehicleSerialNumber=VF0000000001,GearboxType=PKZ} 200

Then push the file with curl (chunked is useful to push huge files that does not fit in RAM):

curl -v  -H 'Transfer-Encoding: chunked' -H 'X-Warp10-Token: 0RJlPOj4g7SbicLhRq8AAR3egfhMeoJFy7dXm_RtDz5LUGSx3HGlBYvSwa7i_0KIxEqjp9JUPHv4bgBIXJ0TfUyfmvsvQsnD3pNQJ6J3AtoBc77ZiWH.ZF' -T example.gts 'http://127.0.0.1:8080/api/v0/update'

Read all these data using your READ token.

curl -v  --data-binary "[ 'hMeJHyhK5wiHur_dsO8kccnzWLAVQO7eOfvOVhXAlmftexalPBy6cENIHwv8JDx10BSBlW0Baz.oOj1uONYYY3H5zdD_WTGePMC_cb2lRr1m_IfUgt.sZYtTdlvU0zbB' '~.*' {} NOW -1 ] FETCH" 'http://127.0.0.1:8080/api/v0/exec'

If everything is OK, you should receive an HTTP 200 OK with the most recent datapoint of every GTS in JSON format.

You may push a lot of GTS files in a single curl command. Just pipe the cat output to a chunked curl:

cat gts/*.gts | curl -T - -H 'Transfer-Encoding: chunked' -H 'X-Warp10-Token: myWriteToken'  'http://127.0.0.1:8080/api/v0/exec'

When using Docker on macOS or Windows, there is no binding between Warp 10 API address and the host (docker is running throw a Virtual Machine). To reach Warp 10 you need to replace 127.0.0.1 by the real IP address of the container. To get it, use a simple docker-machine ip default>, the container address is also shown in the Settings/Ports page of your container. If you used the shared volume between the container and the host, you can access to the virtual machine using docker-machine ssh default> and inspect the repertory /var/warp10. Don't hesitate to check on docker-machine documentation.

Pushing data with python

This small python program will take GTS text file as arguments, and will push them using requests library.

#!/usr/bin/env python
#  -*- coding: utf-8 -*-
import sys
# sudo apt install python-pip
# sudo pip install requests
import requests

if len(sys.argv) < 2:
    print ("Use: " + sys.argv[0] + " *.gtstxt")
    exit()

for filename in sys.argv[1:]:
    print("opening " + filename)
    fout = open(filename, 'rb')
    GTSfile = fout.read()
    fout.close()
    headers = {
        "X-Warp10-Token": "0RJlPOj4g7SbicLhRq8AAR3egfhMeoJFy7dXm_RtDz5LUGSx3HGlBYvSwa7i_0KIxEqjp9JUPHv4bgBIXJ0TfUyfmvsvQsnD3pNQJ6J3AtoBc77ZiWH.ZF"}
    url = "http://127.0.0.1:8080/api/v0/update"
    print(" push data to warp10... " + url)
    r = requests.post(url, headers=headers, data=GTSfile)
    print(" warp10 server answer " + str(r.status_code))

Step 9. WarpStudio interface

Warp 10 can be extended with plugins and extensions.

WarpStudio is included in the docker image, you can open http://localhost:8081/ in your favorite browser (if needed, replace localhost by the IP of the Warp 10 host).

If you are using the tar.gz, you can use the online WarpStudio, and add your local instance endpoint in the settings:

Once backend configuration is done, you can execute WarpScript on your Warp 10 instance. For example, the very first WarpScript you type in this tutorial, using curl, to read all data you pushed:

//store read token in a variable
'hMeJHyhK5wiHur_dsO8kccnzWLAVQO7eOfvOVhXAlmftexalPBy6cENIHwv8JDx10BSBlW0Baz.oOj1uONYYY3H5zdD_WTGePMC_cb2lRr1m_IfUgt.sZYtTdlvU0zbB' 'myreadtoken' STORE

//fetch the most recent point of every GTS this token can access
[ $myreadtoken '~.*' {} NOW -1 ] FETCH

Step 10. VS Code IDE for WarpScript

Visual Studio Code is a source code editor developed by Microsoft for Windows, Linux and macOS. It includes support for debugging, embedded Git control, syntax highlighting, intelligent code completion, snippets, and code refactoring. It is free and open source. We developed a plugin for WarpScript files (.mc2 extension).

After installing VS Code, launch VS Code Quick Open (Ctrl+P), paste the following command, and press enter.

ext install senx.warpscript-language

Click install and restart VS Code.

Create a test.mc2 file with your warpscript:

//store read token in a variable
'hMeJHyhK5wiHur_dsO8kccnzWLAVQO7eOfvOVhXAlmftexalPBy6cENIHwv8JDx10BSBlW0Baz.oOj1uONYYY3H5zdD_WTGePMC_cb2lRr1m_IfUgt.sZYtTdlvU0zbB' 'myreadtoken' STORE

//fetch the most recent point of every GTS this token can access
[ $myreadtoken '~.*' {} NOW -1 ] FETCH

If the extension is correctly loaded, you now have syntax highlighting, autocompletion, hover help, variable completion, local macro substitution...

By default, VS Code extension will execute WarpScript on localhost API (http://127.0.0.1:8080/api/v0/exec). To change the Warp 10 server address, open the VS Code preferences (Ctrl+Shift+P, type "settings", press enter), search for "warpscript", copy and paste a customized line from the left pane to the right pane. If you plan to use multiple instances of Warp 10, you can also specify an endpoint in the very first comment line of the file. VS Code will interpret it.

Press Ctrl+Alt+E to run the warpscript.

JSON result will be displayed on the right part of the editor. In this example, we count the number of GTS returned by the FETCH command. Up to context, JSON can context GTS or PNG images. WarpScript extension will also open GTS Preview or Image Preview tabs.

For advanced use, read our VS Code help page.

Next steps

Congratulation, you now have a fully working environment to evaluate Warp 10 and learn WarpScript.

You may setup a systemd unit to start Warp 10 automatically.

You may customize the maximum amount of RAM memory Warp 10 can use. Default value is 1gB. Edit bin/warp10-standalone.sh with your favorite text editor, customize WARP10_HEAP and WARP10_HEAP_MAX, save and restart Warp 10. The default configuration has low limits everywhere for you to discover them: feel free to raise Warp 10 limits in etc/conf.d/20-warpscript.conf.

You now need to learn WarpScript syntax and discover the functions. First, read the WarpScript concepts. Then you can follow the tutorials or you can search for functions and examples in the reference documentation. We do provide datasets in the tutorials, but the best way is to learn with your data and your use cases!

You can configure Warp 10 as an in memory instance, or configure the Warp 10 Accelerator to keep recent history in RAM.

Download The Cheat Sheet WarpScript

And don’t forget to star Warp 10 on GitHub. ;)