Warp 10 talking to Warp 10

In this tutorial, you will learn several ways to interact between different Warp 10 instances. You do not need any external tooling to do this. You need two Warp 10 instances on your local network.

Use case

An embedded datalogger collect data on CAN, ModBus, analog inputs. It stores data locally in a local Warp 10 instance. Each minutes, it performs a subsampling of the data and push them in the company Warp 10 datalake.

In this tutorial, my embedded datalogger IP is 192.168.1.105, and my company datalake is my own computer 192.168.1.210.

On both Warp 10 instances, I have read and write tokens, named below:

• DataLakeR: read token on 192.168.1.210
• DataLakeW: read token on 192.168.1.210
• LoggerR: read token on 192.168.1.105
• LoggerW: write token on 192.168.1.105

Note you can really define a token called LoggerR. Not really secure, but very handy for a tutorial, you can copy and paste examples without any replacement. Read more here.

REXEC introduction

You build a custom application to listen on CAN, ModBus, analog inputs. This application produces GTS input format, send to the local ingress endpoint http://localhost:8080/api/v0/update. Now, you need to copy these data on your company datalake.

WarpScript has a REXEC function which allow any WarpScript to execute WarpScript on another Warp 10 instance. This function, with a few serialization tricks, is the key to easily transfer anything between two Warp 10 instances.

For this tutorial, you need to allow REXEC on the datalogger Warp 10. In your configuration file set property warpscript.rexec.enable to true. Restart the instance and test it:

RexecTest.mc2 file content:

Note the trick at the first line of my WarpScript file… I am using VSCode IDE with WarpScript language extension, and I dynamically ask VSCode to change the execution endpoint. See VSCode guide here.

The result is a stack with the number 4. What happened ?

• The top of the stack is the execution endpoint (API url).
• Just below, I put a string. This string is a WarpScript. REXEC took the WarpScript and exec it on itself (localhost).

Note: The EVAL function does exactly the same, locally. It parses and executes WarpScript in a string. A WarpScript is able to generate WarpScript and execute it.

I can ask my embedded Warp 10 to execute WarpScript on my company datalake. First, make sure your Warp 10 listens on every interfaces, not only 127.0.0.1 (default setting in a standalone version). Edit the configuration file, and set standalone.host = 0.0.0.0. Restart and test it:

The result is 5. The datalogger executed 2 3 + on the company datalake.

In the getting started tutorial, you learned to push data to the ingress endpoint /api/v0/update. The great news is that you can push data into the storage from within a WarpScript with the UPDATE function:

If you replaced LoggerW by the correct write token, you should have an empty stack result without any error, and you can read the data back from storage with the matching read token:

Serialization

You need to build a string which is a valid WarpScript, and REXEC it on your datalake with an UPDATE instruction. Here enter serialization: WarpScript offers off the shelf functions to serialize easily your data. If you didn't, it is time to go back in the learning path and read the last part of Type conversion tutorial.

Serialization is your best friend to exchange data between Warp 10 instances. First, test it:

Input is a list of GTS, output is a list of strings. Each GTS has been individually serialized. To serialize a list of string, you can turn it in a JSON.

Remote update

The above example seems correct. But it fails with error 500.

Explanations:

• Erasing your own data with UPDATE is easy. For example, you FETCH some data, apply a mean mapper, then UPDATE the result: oops, you have just erased your original data. Warp 10 has a security mechanism to prevent this: if you do not rename a GTS, you cannot UPDATE it. The 'renamed' flag is also wrapped by the serialization process.
• You FETCH some GTS on the datalogger, but you never renamed them. UPLOAD is forbidden. You need to rename them, before or after serialization. Renaming with their original name is allowed.

In the following example, the rename is done on the server side:

• LMAP is used to apply a macro on each list elements.
• Remember, WarpScript strings are URL encoded. You need to escape % character with %25.

You can check if the data is on your datalake:

SNAPSHOT the stack

You may want to serialize other elements than GTS lists. SNAPSHOT can serialize the full stack content in a valid warpscript string. Note that SNAPSHOT serializes GTS with WRAP. The SNAPSHOT output is ready to EVAL (or REXEC).

Pushing data on the datalake

Now you know how to transfer easily GTS from On the embedded Warp 10 instance, you need to create a runner every minutes.

To keep track of successful uploads, you can create a GTS 'successfullupload' on the datalogger. This GTS will contains the timestamp of the last successful update.

Enhance performances:

• Use REXECZ. In both ways, the content will be gzipped on the fly, saving bandwidth.

Enhance security:

• Use https with good certificates.
• Use tokens. Tokens allow you to do a pretty cool setup in this case: Every datalogger can have access to read and write data but not delete them. If a datalogger is compromised, the attacker won't be able to access more than this datalogger data.

Download The Cheat Sheet WarpScript