Loading

  1. Paradigm
    1. Example
  2. Comments
  3. Data Types
    1. Basic Types
    2. Composite Types
    3. Geo Time Series
    4. Special Types
  4. Variables
  5. Frameworks
  6. Macros
  7. Flow Control
    1. Loops
    2. Conditionals
    3. Asynchronous Transfer of Control
  8. Unit Testing
  9. Miscellaneous

WarpScript

WarpScript

WarpScript is a language specifically designed for analytics of time series data. It is the pillar of the analytics layer of the Warp 10 Platform. WarpScript makes time series first class citizens, meaning that they are a specific data structure that can be manipulated. This approach makes you more efficient when working with time series data than if you had to think of time series as numbers in tables. The time series data structure offered by WarpScript is called Geo Time Series as an optional location (latitude, longitude and elevation) can be attached to each value of a time series.

WarpScript is a data flow programming language. This differs from a query language which can fetch data and perform simple computations. WarpScript has the power of a Turing Complete programming language, with conditionals, loops, and asynchronous transfer of control.

The WarpScript Execution Engine is the part of the Warp 10 Platform which runs the WarpScript code, simply submit code via a HTTP request, the engine will execute it close to where the data reside and will return the result as a JSON object which can be integrated into an application.

The WarpScript language is also integrated in many existing data processing languages and environments such as R, Python, Zeppelin, Jupyter, Spark, Pig, Storm or NiFi, with the ability to use the same WarpScript code across all those environments. This greatly increases efficiency.

Lastly, by providing off the shelf support for time series data, WarpScript enables a greater focus on business use cases by waving the need to build or adapt low level libraries.

Paradigm

WarpScript is part of a broader family of programming languages called concatenative programming languages whose most notable members are PostScript, FORTH and Bitcoin Script.

In WarpScript, as in all concatenative programming languages, every expression is a function and juxtaposition of expressions denotes function composition, in the spirit of how Unix pipelines work.

WarpScript manages a data pipeline with which all functions interact, getting their arguments by emptying the pipeline, and filling it with their results. The data pipeline is passed from function to function during the execution of WarpScript code.

The Warp 10 Analytics Engine, the WarpScript Execution Engine of Warp 10 accessed via the /api/v0/exec endpoint, returns a JSON list which contains the elements of the data pipeline after the code execution. The first element of this list contains the most recent element of the pipeline.

Example

Here is a full example of a fictional function FUNC which is applied to two arguments 42 (a number) and 'foo' (a character string) and produces a single result which is the string representation of its arguments separated by a comma.

The execution starts with an empty data pipeline, here the pipeline is represented with 5 empty slots, in gray.

EmptyPipeline

Next the two arguments are added. Remember every expression in WarpScript is a function, this is also true for constants, so the number 42 can be seen as a function which adds the value 42 to the data pipeline and the character string 'foo' as a function which adds itself to the data pipeline.

The following WarpScript code:

42 'foo'

Therefore changes the data pipeline into:

Pipeline2

Next the FUNC function is called:

FUNC

it will consume its two arguments from the data pipeline, will concatenate their string representations and will add the result to the pipeline. The yellow pipeline slots are the arguments which will be consumed by FUNC:

PipelineArgs

The final state of the pipeline is therefore:

PipelineFinal

If executing this code via the Warp 10 Analytics Engine, the resulting JSON would be:

[ "42,foo" ]

as the data pipeline contains a single element at the end of the execution. If the call to FUNC was omitted, the result would be:

[ "foo", 42 ]

reflecting the two elements present in the data pipeline at the end of the execution.

This is really all there is to understand about the way WarpScript is executed. The rest of this article will walk you through some of the language constructs so you can easily understand the various examples present in the function documentation pages.

Comments

WarpScript code can contain comments with the following syntax:

#  Python style comments, starting with a '#' and extending to the end of the line

// Java style comments, extending to the end of the line after the '//'

/*
  C style block comments, possibly spanning multiple lines
*/

Data Types

WarpScript has support for basic and composite types.

Basic Types

The basic types are LONG (64 bits signed integer), DOUBLE (64 bits signed IEEE754 floating point), STRING (UTF-8 character string) and BOOLEAN.

There is also a null type which is produced by the NULL function.

Those types follow their commonly observed syntax. STRING values are percent encoded and enclosed in single or double quotes.

// LONG values (decimal, hexadecimal, binary)
42 0xFFFF 0b10101010 

// DOUBLE values, with full support for non numeric values
42.5 3.14E6 NaN Infinity -Infinity

// BOOLEAN values
true false T F

// STRING values
'Caf%C3%A9' "foo"

NULL

Composite Types

Composite types are mainly LISTs (arrays) and MAPs (associative arrays) and are created using the following syntax:

// A list, space separated elements enclosed between '[' and ']'
[ 'a' 42 3.141592 ]

// A map, a space separated collection of key/value pairs, enclosed in curly braces '{' and '}'
{
  // key is 'foo', value is 'bar'
  'foo' 'bar'
  // key is 1, value is 42.0
  1 42.0
}

Composite types can be nested, for example you can nest lists to simply represent matrices.

Geo Time Series

You already know that WarpScript considers Geo time Series as first class citizens, this is reflected in the very existence of the GTS type.

A GTS should be seen as a container to which data points can be added. The Warp 10 Storage Engine, Warp 10's own enhanced TSDB, manages persistent Geo Time Series, and WarpScript can create GTS either by retrieving data from a storage engine, or programmatically via the NEWGTS function.

Geo Time Series have a set of metadata attached to them, a class name which describes what kind of values are stored in the GTS, a temperature for example. They also have a set of key/values calles labels which are used to add context to a Geo Time Series, for example by recording a serial number of a device or the IP address of a server being monitored. Jointly the class and labels form the id of the Geo Time Series as managed by the Warp 10 Storage Engine. Geo Time Series' metadata can also contain another set of key/value pairs called attributes. Contrary to labels, attributes are not part of the GTS id and can be modified during its lifetime.

A Geo Time Series in WarpScript needs to have values all of the same type with supported types of LONG, DOUBLE, BOOLEAN and STRING. A hybrid type BINARY can also be stored in a GTS, a byte array encoded using ISO-8859-1 and stored in a character string.

It is also possible to store multivariate values as the value of a data point in a GTS. This type of value is really an inner Geo Time Series which is encoded and stored as BINARY. Since it is a GTS of its own, it can have multivariate value too, don't you love nesting?

Each value can have a latitude and longitude and/or elevation attached to it.

Geo Time Series is a very important data type in WarpScript, as you progress you will become more and more comfortable with it.

When pushing data to Geo Time Series managed by the Warp 10 Storage Engine, the syntax called the GTS input format should be used.

To learn more, a tutorial describes operations on GTS.

Special Types

Some special types are created by some WarpScript functions: GeoXPShape, PGraphics, byte arrays, matrices, vectors, ...

You will learn more as you go deeper into WarpScript.

Variables

Technically you could work directly with the data pipeline, but it is often more convenient to use variables, i.e. addressing values by a named symbol.

Storing a value in a variable is done using the STORE function which consumes both the name of the variable to create and the value to store into it from the data pipeline.

The value associated with a variable can be added to the data pipeline using either the $name syntax or calling the LOAD function with the name of the variable to read present in the data pipeline.

Variables are global, in order to limit the scope of a variable you can save the current context which has all the variable definitions using the SAVE function, modify or create variables and later restore the saved context using RESTORE, the variables will return to their initial state.

Frameworks

Frameworks are the companion functions to the Geo Time Series data structure. They provide common ways to interact with GTS instances. WarpScript has 6 frameworks.

  • BUCKETIZE - Make ticks evenly spaced
  • MAP - Apply a function on a sliding window around each tick
  • REDUCE - Apply a function tick by tick to multiple Geo Time Series
  • FILTER - Select Geo Time Series based on some criteria
  • APPLY - Apply an N-ary function to lists of Geo Time Series, tick by tick
  • FILL - Fill gaps in a pair of Geo Time Series

Macros

Macros allow for code factorization, they are code fragments which can be invoked from WarpScript code.

They can be deployed:

The syntax for invoking a macro is @name/of/macro.

Some WarpScript functions accept macros as arguments, the syntax for creating a macro in WarpScript is:

<%
  SAVE 'context' STORE

    // Code of the actual macro

  $context RESTORE
%>

If a variable macro contains a macro as value, it can be invoked using @macro.

Flow Control

As a Turing Complete language, WarpScript has some constructs for doing flow control.

Most of the functions related to flow control have macro arguments.

Loops

FunctionSample
FOR$from $to <% /* ACTION */ %> FOR
FORSTEP$from $to <% /* STEP */ %> <% /* ACTION */ %> FORSTEP
WHILE<% /* COND */ %> <% /* ACTION */ %> WHILE
UNTIL<% /* ACTION */ %> <% /* COND */ %> UNTIL

Conditionals

If-Then: IFT

<% /* COND macro - Must return a boolean */ %>
<% /* ACTION macro - Executed if condition is true */ %>
IFT

If-Then-else: IFTE

<% /* COND macro */ %>
<% /* ACTION macro - executed if condition returned true */ %>
<% /* ALT macro - executed if condition is false */ %> 
IFTE

Switch: SWITCH

<% /* COND 1 */ %> <% /* ACTION 1 */ %>
<% /* COND 2 */ %> <% /* ACTION 2 */ %>
<% /* DEFAULT ACTION */ %> 2 SWITCH

Asynchronous Transfer of Control

BREAK, CONTINUE, FAIL, MSGFAIL, NRETURN, RETHROW, RETURN, STOP

Try/catch TRY

<% /* ACTION */ %>
<% /* CATCH CLAUSE */ %>
<% /* FINALLY CLAUSE */ %>
TRY

Unit Testing

It is very important that any programming language has support for writing tests. WarpScript fully adheres to this principle and provides functions for writing tests:

ASSERT, ASSERTMSG

$result 'expected' == ASSERT
$result 'expected' 'Message' ASSERTMSG

Miscellaneous

We have covered the basic concepts of the WarpScript language. As you will discover in the various documentation pages, WarpScript has many more aspects which would have turned this introduction into a monument if they were all mentioned. For example WarpScript includes a large subset of the Processing drawing library which means you can create and manipulate images right from within WarpScript.

For now we encourage you to experiment with WarpScript, either in the documentation pages for the various functions, or via WarpStudio or the Warp 10 Sandbox.