Getting Started with NVIDIA Jetson Orin Nano Super – Generative AI Supercomputer

NVIDIA has just reinvented edge computing with its latest offering – the Jetson Orin Nano Super Developer Kit. This isn’t just an incremental update; it’s a significant leap forward in bringing generative AI capabilities to the edge at an unprecedented price point of $249.

Comparing Jetson Nano Vs Jetson Orin Nano Super

The NVIDIA Jetson Orin Nano Super Developer Kit is a compact, yet powerful computer that redefines generative AI for small edge devices.

It delivers up to 67 TOPS of AI performance—a 1.7X improvement over its predecessor—to seamlessly run a wide variety of generative AI models, like vision transformers, large language models, vision-language models, and more.

It provides developers, students, and makers with the most affordable and accessible platform with the support of the NVIDIA AI software and a broad AI software ecosystem to democratize generative AI at the edge. Existing Jetson Orin Nano Developer Kit users can experience this performance boost with just a software upgrade, so everyone can
now unlock new possibilities with generative AI.

Learn about Jetson AI Lab

Let’s dive into the specs and see how it compares to its predecessors:

How powerful is NVIDIA Jetson Orin Super?

The most striking aspect of the Super variant is its performance improvements:

• 1.7x increase in AI compute performance (67 TOPS vs 40 TOPS)
• 1.5x increase in memory bandwidth (102 GB/s vs 68 GB/s)
• Higher GPU and CPU clock speeds for better overall performance

Generative AI Capabilities

The NVIDIA Jetson™ platform runs the NVIDIA AI software stack, with a variety of available use-case-specific application frameworks. These include NVIDIA Isaac™ for robotics, NVIDIA Metropolis for vision AI, and NVIDIA Holoscan for sensor processing. You can save significant time with NVIDIA Omniverse™ Replicator for synthetic data generation (SDG) and NVIDIA TAO Toolkit for fine-tuning pretrained AI models from the NVIDIA® NGC™ catalog.

One of the most impressive aspects of the Orin Nano Super is its ability to run various types of generative AI models:
Large Language Models (LLMs):

Vision Language Models (VLMs):

Vision Transformers

I/O and Connectivity

Developer-Friendly Features

AI Development Tools

Getting Started with Jetson Orin Super

This guide will walk you through setting up Ollama on your Jetson device, integrating it with Open WebUI, and configuring the system for optimal GPU utilization. Whether you’re a developer or an AI enthusiast, this setup allows you to harness the full potential of LLMs right on your Jetson device.

Pre-requisite

1. Jetson Orin Nano
2. A DC power supply
3. 64GB/128GB SD card
4. WiFi Adapter
5. Wireless Keyboard
6. Wireless mouse

Software

• Download Jetson SD card image
• Raspberry Pi Imager / Etcher installed on your local system

Download Jetson SDK using this link

Preparing Your Jetson Prin Nano

1. Unzip the SD card image
2. Insert SD card into your system.
3. Bring up Raspberry Pi Imager tool to flash image into the SD card

Prerequisite

• Ensure that you have Jetpack 6.0 installed on your Jetson Orin Nano device. You can download the SDK Manager on the remote Windows or Linux and follow the tutorial from the official NVIDIA Developer site.

Step 1. Verify L4T Version

To check the L4T (Linux for Tegra) version on your NVIDIA Jetson device (e.g., Jetson Nano, Jetson Xavier), follow these steps:

Run the following command to retrieve your current L4T version.

$ head -n 1 /etc/nv_tegra_release

Here are the list of supported L4T versions:

• 35.3.1
• 35.4.1
• 35.5.0
• 36.3.0

If your L4T version does not match the supported versions listed above, you may need to re-flash the system on your NVIDIA Jetson device. You might need to use SDK Manager on another computer to re-flash the device. You can download the SDK Manager and follow the tutorial from the official NVIDIA Developer site.

Step 2. Keep apt up to date:

  $ sudo apt update && sudo apt upgrade

Step 3. Install jetpack:

 $  sudo apt install jetpack

Step 4. Add users

Add your user to the docker group and restart the Docker service to apply the change:

 $  sudo usermod -aG docker $USER && \
 $  newgrp docker && \
 $  sudo systemctl daemon-reload && \
 $ sudo systemctl restart docker

Step 5. Install jetson-examples:

 $  pip3 install jetson-examples

Step 6. Reboot system

 $  sudo reboot

Step 7. Install Ollama

  $ reComputer run ollama

Optional: If you run the above command via ssh and encounter the error command not found: reComputer, you can resolve this by executing the following command:

  $ source ~/.profile

Step 8. Run a model

The smallest LLaMA model available for download is TinyLlama, a compact 1.1 billion parameter model. Despite its reduced size, TinyLlama demonstrates remarkable performance across various tasks, making it suitable for applications with limited computational resources. You can access TinyLlama through its GitHub repository or via Hugging Face.

Let’s run the tinyllama model and perform tasks like generating Python code:

ollama run tinyllama
>>> > Can you write a Python script to calculate the factorial of a number?
Sure! Here’s the code:

def factorial(n):
    if n == 0 or n == 1:
        return 1
    else:
        return n * factorial(n - 1)

num = int(input("Enter a number: "))
print(f"The factorial of {num} is {factorial(num)}")

Step 9. Install models (e.g. llama3.2) from Ollama Library

$ ollama pull llama3.2

Step 9. Install and run Open WebUI through Docker

$ sudo docker run -d -p 3000:8080 --gpus all --add-host=host.docker.internal:host-gateway -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:cuda

Step 10. Install and run Open WebUI through docker

Once the installation is finished, you can access the GUI by visiting YOUR_SERVER_IP:3000 in your browser.

Access the API endpoints by navigating to YOUR_SERVER_IP/ollama/docs#/. For comprehensive documentation, please refer to the official resources: the Ollama API Documentation (recommended) and Open WebUI API Endpoints.

Using GPU

This installation method uses a single container image that bundles Open WebUI with Ollama, allowing for a streamlined setup via a single command. Choose the appropriate command based on your hardware setup:

sudo docker run -d -p 3000:8080 --gpus=all -v ollama:/root/.ollama -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:ollama

Using CPU only

For CPU Only: If you’re not using a GPU, use this command instead:

$ sudo docker run -d -p 3000:8080 -v ollama:/root/.ollama -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:ollama

Both commands facilitate a built-in, hassle-free installation of both Open WebUI and Ollama, ensuring that you can get everything up and running swiftly.

Once configured, Open WebUI can be accessed at http://localhost:3000, while Ollama operates at http://localhost:11434. This setup provides a seamless and GPU-accelerated environment for running and managing LLMs locally on NVIDIA Jetson devices.

Conclusion

The Jetson Orin Nano Super Developer Kit represents a significant milestone in edge AI computing. It brings datacenter-class AI capabilities to the edge at an unprecedented price point, making it an ideal platform for developers, researchers, and businesses looking to deploy advanced AI applications at the edge.

The combination of increased AI performance, enhanced memory bandwidth, and broad model support makes it a compelling choice for anyone serious about edge AI development. At $249, it’s not just a product – it’s a revolution in accessible AI computing.

References

• Jetson Orin Nano Super Quick Start Guide
• Jetson Orin JetPack SDK