Running RedisAI on NVIDIA Jetson Nano for the first time

The hardest part of AI is not artificial intelligence itself, but dealing with AI data. The accelerated growth of data captured from the sensors in the internet of things (IoT) solutions and the growth of machine learning (ML) capabilities are yielding unparalleled opportunities for organizations to drive business value and create competitive advantage. That’s why ingesting data from many sources and deriving actionable insights or intelligence from it have become a prime objective of AI-enabled applications.

Recently, a new class of AI databases has emerged. An AI database is built with the sole purpose of speeding ML model training and model serving. Such databases help businesses optimize AI learning and training. They help you wrangle the volume, velocity, and complex data governance and management challenges associated with training ML and DL models to save time and optimize resources.

Redis is just such an AI database. The RedisAI module, which is seamlessly plugged into Redis,  is a scalable platform that addresses the unique requirements for both AI training and AI inference in one server. It provides a complete software platform that allows data scientists to easily deploy and manage AI solutions for enterprise applications. The platform combines popular open-source deep learning frameworks (PyTorch, ONNXRuntime, and TensorFlow), software libraries, and Redis modules like RedisGears, RedisTimeSeries, and more. With RedisAI, AI application developers no longer have to worry about tuning databases for performance. Requiring no added infrastructure, RedisAI lets you run your inference engine where the data lives, decreasing latency.

Adding Tensor data structure to Redis

RedisAI lets you add a tensor data structure to Redis. A tensor is an n-dimensional array and is the standard representation for data in deep learning and machine learning workloads. Like any data in Redis, RedisAI’s tensors are identified by key names, are used as inputs and outputs in the execution of models and scripts, and run well on CPUs as well as on GPUs.

The RedisAI model data structure represents a machine learning/deep learning (ML/DL) model stored in the database. RedisAI supports DL/ML identifiers and their respective backend libraries like Tensorflow, PyTorch, and ONNX. RedisAI can actually execute models from multiple frameworks as part of a single pipeline. If your tech stack is on some other language and you don’t want to introduce Python into it, as long as the language of your choice has a Redis client (very likely), you can deploy your model into RedisAI and use your Redis client to control the execution with little to no overhead. 

How RedisAI work?

RedisAI bundles together best-of-breed technologies for delivering stable and performant computation graph serving. Every DL/ML framework ships with runtime for executing the models developed with it and the common practice for serving these is building a simple server around them.

RedisAI aims to be that server, saving you from the need of installing the backend you’re using and developing a server for it. By itself that does not justify RedisAI’s existence so there’s more to it. Because RedisAI is implemented as a Redis module it automatically benefits from the server’s capabilities: be it Redis’ native data types, its robust eco-system of clients, high-availability, persistence, clustering, and Enterprise support.

Because Redis is an in-memory data structure server RedisAI uses it for storing all of its data. The main data type supported by RedisAI is the Tensor that is the standard representation of data in the DL/ML domain. Because tensors are stored in the memory space of the Redis server, they are readily accessible to any of RedisAI’s backend libraries at minimal latency.

The locality of data, which is tensor data in adjacency to DL/ML models backends, allows RedisAI to provide optimal performance when serving models. It also makes it a perfect choice for deploying DL/ML models in production and allowing them to be used by any application.

Furthermore, RedisAI is also an optimal testbed for models as it allows the parallel execution of multiple computation graphs and, in future versions, assessing their respective performance in real-time.

RedisAI provides the following data structures:

• Tensor: Represents an n-dimensional array of values
• Model: Represents a computation graph by one of the supported DL/ML framework backends
• Script: Represents a TorchScript program

DL/ML backends

RedisAI supports DL/ML identifiers and their respective backend libraries, including:

• TF: The TensorFlow backend
• TFLITE: The TensorFlow Lite backend
• TORCH: The PyTorch backend
• ONNX: ONNXRuntime backend

A complete list of supported backends is in the release notes for each version.

Backend libraries are dynamically loaded as needed, but can also be loaded during booting or at runtime. Refer to these pages for more information on loading backends:

AI.CONFIG command
Backend configuration

Please Note:

• The RedisAI module doesn’t train your data – for that you need a tool like TensorFlow and PyTorch(Two open-source projects for machine learning).
• Where RedisAI comes in is at the application layer, when it’s time to apply logic to the data(inferencing) and then serve it to the user.
• When you want to train your AI model somewhere in the Cloud. Once you want to do serving or inferencing, Redis is the right database to do that.

RedisAI delivers up to 10x more inferencing

By putting the AI serving engine inside Redis, RedisAI delivers up to 10x more inferences than other AI serving platforms, at a much lower latency, delivering up to 9x more throughput than other AI-model serving platforms. These performance improvements can drive dramatically better business outcomes for many leading AI-driven applications—including fraud detection, transaction scoring, ad serving, recommendation engines, image recognition, autonomous vehicles, and game monetization.

Why RedisAI on Jetson Nano?

Jetson Nano Developer Kit is purely an AI computer. Delivering 572 GFLOPS of computing performance, this is a small, powerful $99 computer that lets you run modern AI workloads and is highly power-efficient, consuming as little as 5 watts. It can run a wide variety of advanced networks, including the full native versions of popular ML frameworks like TensorFlow, PyTorch, Caffe/Caffe2, Keras, MXNet, and others and it comes with a GPU card inbuilt.

Pre-requisite

• Jetson Nano Board
• Power Adapter(5V 4A)
• WiFi Module
• Redis 6.x

RedisAI require Redis 6.x. You might have to compile the newer version of Redis to get it up and running.

Installing Redis 6.x

Follow the below steps:

$ wget http://download.redis.io/releases/redis-6.0.8.tar.gz
$ tar xzf redis-6.0.8.tar.gz
$ cd redis-6.0.8
$ make
$ sudo cp src/redis-server /usr/local/bin/
$ sudo cp src/redis-cli /usr/local/bin/

In case you are not able to stop Redis service, follow the below steps

redis-cli
127.0.0.1> shutdown

Steps

Clone the repository

git clone --recursive https://github.com/RedisAI/RedisAI

cd RedisAI

Download all the below scripts and place it under RedisAI directory.

• install.sh
• run.sh
• test.sh

Building RedisAI

Upload the below code to your Jetson board:

sudo apt update
sudo apt install -y git build-essential ninja-build cmake python3-pip python3-cffi redis unzip wget

git clone https://github.com/RedisAI/RedisAI.git

cd RedisAI

mkdir build

WITH_PT=0 WITH_TF=0 WITH_TFLITE=0 WITH_ORT=0 bash get_deps.sh

mv deps/linux-arm64v8-cpu deps/linux-x64-cpu

mkdir deps/linux-x64-cpu/libtorch

cd deps/linux-x64-cpu/libtorch


wget https://nvidia.box.com/shared/static/3ibazbiwtkl181n95n9em3wtrca7tdzp.whl -O torch-1.5.0-cp36-cp36m-linux_aarch64.whl
sudo apt install -y libopenblas-base

unzip torch-1.5.0-cp36-cp36m-linux_aarch64.whl
mv torch/* .

cd -

cd build

cmake -DBUILD_TF=OFF -DBUILD_TFLITE=OFF -DBUILD_TORCH=ON -DBUILD_ORT=OFF -DCMAKE_BUILD_TYPE=Release ../

make -j4 && make install

sh install.sh

It will take sometime based on your internet speed.

Running RedisAI with PyTorch

Upload the below code to your Jetson board:

# Put this script inside RedisAi main folder.
# Run with 'bash run_redisai_torch.sh'.
# Before running check that the script is executable 'chmod 755 run_redisai_torch.sh'

redis-server --loadmodule install-cpu/redisai.so

Before you execute the below script, provide sufficient permission to redisai.so

/RedisAI/install-cpu$ sudo chmod 755 redisai.so 

 sudo sh run.sh 
14438:C 20 Sep 2020 14:06:50.321 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
14438:C 20 Sep 2020 14:06:50.321 # Redis version=6.0.6, bits=64, commit=00000000, modified=0, pid=14438, just started
14438:C 20 Sep 2020 14:06:50.321 # Configuration loaded
14438:M 20 Sep 2020 14:06:50.323 * Increased maximum number of open files to 10032 (it was originally set to 1024).
                _._                                                  
           _.-``__ ''-._                                             
      _.-``    `.  `_.  ''-._           Redis 6.0.6 (00000000/0) 64 bit
  .-`` .-```.  ```\/    _.,_ ''-._                                   
 (    '      ,       .-`  | `,    )     Running in standalone mode
 |`-._`-...-` __...-.``-._|'` _.-'|     Port: 6379
 |    `-._   `._    /     _.-'    |     PID: 14438
  `-._    `-._  `-./  _.-'    _.-'                                   
 |`-._`-._    `-.__.-'    _.-'_.-'|                                  
 |    `-._`-._        _.-'_.-'    |           http://redis.io        
  `-._    `-._`-.__.-'_.-'    _.-'                                   
 |`-._`-._    `-.__.-'    _.-'_.-'|                                  
 |    `-._`-._        _.-'_.-'    |                                  
  `-._    `-._`-.__.-'_.-'    _.-'                                   
      `-._    `-.__.-'    _.-'                                       
          `-._        _.-'                                           
              `-.__.-'                                               

14438:M 20 Sep 2020 14:06:50.325 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
14438:M 20 Sep 2020 14:06:50.325 # Server initialized
14438:M 20 Sep 2020 14:06:50.325 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
14438:M 20 Sep 2020 14:06:50.326 * <ai> Redis version found by RedisAI: 6.0.6 - oss
14438:M 20 Sep 2020 14:06:50.326 * <ai> RedisAI version 999999, git_sha=unknown
14438:M 20 Sep 2020 14:06:50.326 * Module 'ai' loaded from install-cpu/redisai.so
14438:M 20 Sep 2020 14:06:50.327 * Loading RDB produced by version 6.0.6
14438:M 20 Sep 2020 14:06:50.327 * RDB age 151321 seconds
14438:M 20 Sep 2020 14:06:50.327 * RDB memory usage when created 0.80 Mb
14438:M 20 Sep 2020 14:06:50.327 * DB loaded from disk: 0.000 seconds
14438:M 20 Sep 2020 14:06:50.327 * Ready to accept connections

Testing RedisAI

Upload the below code to your Jetson board:

# Put this script inside RedisAi main folder.
# Run only after you started a redisai server (view run_redisai_torch.sh).

set -x

redis-cli -x AI.MODELSET m TORCH GPU < ./test/test_data/pt-minimal.pt

redis-cli AI.TENSORSET a FLOAT 2 2 VALUES 2 3 2 3
redis-cli AI.TENSORSET b FLOAT 2 2 VALUES 2 3 2 3

redis-cli AI.MODELRUN m INPUTS a b OUTPUTS c

redis-cli AI.TENSORGET c VALUES

sh test.sh 

+ redis-cli -x AI.MODELSET m TORCH GPU
+ redis-cli AI.TENSORSET a FLOAT 2 2 VALUES 2 3 2 3
OK
+ redis-cli AI.TENSORSET b FLOAT 2 2 VALUES 2 3 2 3
OK
....
....
ajeetraina@ajeetraina-desktop:~/redisaiscript$ 

Using RedisAI tensors

A tensor is an n-dimensional array and is the standard representation for data in DL/ML workloads. RedisAI adds to Redis a Tensor data structure that implements the tensor type. Like any datum in Redis, RedisAI’s Tensors are identified by key names.

Creating new RedisAI tensors is done with the AI.TENSORSET command. For example, consider the tensor:

tensorA

We can create the RedisAI Tensor with the key name ‘tA’ with the following command:

AI.TENSORSET tA FLOAT 2 VALUES 2 3

Copy the command to your cli and hit the on your keyboard to execute it. It should look as follows:

$ redis-cli
127.0.0.1:6379> AI.TENSORSET tA FLOAT 2 VALUES 2 3
OK

The reply ‘OK’ means that the operation was successful. We’ve called the AI.TENSORSET command to set the key named ‘tA’ with the tensor’s data, but the name could have been any string value. The FLOAT argument specifies the type of values that the tensor stores, and in this case a single-precision floating-point. After the type argument comes the tensor’s shape as a list of its dimensions, or just a single dimension of 2.

The VALUES argument tells RedisAI that the tensor’s data will be given as a sequence of numeric values and in this case the numbers 2 and 3. This is useful for development purposes and creating small tensors, however for practical purposes the AI.TENSORSET command also supports importing data in binary format.

The Redis key ‘tA’ now stores a RedisAI Tensor. We can verify that using standard Redis commands such as EXISTS and TYPE:

127.0.0.1:6379> EXISTS tA
(integer) 1
127.0.0.1:6379> TYPE tA
AI_TENSOR

Using AI.TENSORSET with the same key name, as long as it already stores a RedisAI Tensor, will overwrite the existing data with the new. To delete a RedisAI tensor, use the Redis DEL command.

RedisAI Tensors are used as inputs and outputs in the execution of models and scripts. For reading the data from a RedisAI Tensor value there is the AI.TENSORGET command:

127.0.0.1:6379> AI.TENSORGET tA VALUES
1) INT8
2) 1) (integer) 2
3) 1) (integer) 2
    1) (integer) 3

Reference:

• https://oss.redislabs.com/redisai/
• https://gist.github.com/lantiga/315ab6e10f010e908d8e4eb4c63505d1
• RedisAI Videos