Local Voice Chatbot — Riva + Llama2 on Jetson

Combines NVIDIA Riva for speech recognition/synthesis with Llama2 for conversational AI, all running locally on Jetson for privacy and low latency.

Execution model

Run one phase at a time. After each phase:

• Relay all output to the user.
• If output contains [STOP] → stop, consult the failure decision tree.
• If output ends with [OK] → tell the user "Phase N complete" and proceed.

Prerequisites

Phase 1 — Configure NGC CLI (~3 min)

cd ~ && mkdir -p ngc_setup && cd ngc_setup
wget --content-disposition https://api.ngc.nvidia.com/v2/resources/nvidia/ngc-apps/ngc_cli/versions/3.36.0/files/ngccli_arm64.zip
unzip ngccli_arm64.zip
chmod u+x ngc-cli/ngc
echo "export PATH=\"\$PATH:$(pwd)/ngc-cli\"" >> ~/.bash_profile
source ~/.bash_profile
ngc config set

Enter your NGC API Key when prompted (obtain from NGC → Account → Setup → Get API Key).

[OK] when ngc config set completes. [STOP] if download or auth fails.

Phase 2 — Install and start Riva server (~20–30 min)

cd ~ && mkdir -p riva_setup && cd riva_setup
ngc registry resource download-version nvidia/riva/riva_quickstart_arm64:2.13.1
cd riva_quickstart_v2.13.1

Edit config.sh to disable unused services:

sed -i 's/service_enabled_nlp=true/service_enabled_nlp=false/' config.sh
sed -i 's/service_enabled_nmt=true/service_enabled_nmt=false/' config.sh

Configure Docker for NVIDIA runtime:

sudo bash -c 'cat > /etc/docker/daemon.json << EOF
{
    "default-runtime": "nvidia",
    "runtimes": {
        "nvidia": {
            "path": "nvidia-container-runtime",
            "runtimeArgs": []
        }
    }
}
EOF'
sudo systemctl restart docker

Initialize and start Riva:

sudo bash riva_init.sh
sudo bash riva_start.sh

Keep this terminal alive.

[OK] when Riva server starts and shows "Riva server is ready". [STOP] if init or start fails.

Phase 3 — Install and run Llama2 via text-generation-inference (~15–20 min)

Open a new terminal:

cd ~
git clone https://github.com/dusty-nv/jetson-containers.git
cd jetson-containers
pip install -r requirements.txt
./run.sh $(./autotag text-generation-inference)

Inside the container:

export HUGGING_FACE_HUB_TOKEN=<your-huggingface-token>
text-generation-launcher --model-id meta-llama/Llama-2-7b-chat-hf --port 8899

Keep this terminal alive.

[OK] when the model loads and the server is listening on port 8899. [STOP] if OOM or auth fails.

Phase 4 — Run the voice chatbot demo (~3 min)

Open a third terminal:

cd ~
git clone https://github.com/yuyoujiang/Deploy-Riva-LLama-on-Jetson.git
cd Deploy-Riva-LLama-on-Jetson

List audio devices:

python3 local_chatbot.py --list-input-devices
python3 local_chatbot.py --list-output-devices

Launch the chatbot with your device IDs:

python3 local_chatbot.py --input-device <INPUT_ID> --output-device <OUTPUT_ID>

[OK] when the chatbot responds to voice input with spoken output.

Failure decision tree

Reference files

• references/source.body.md — Full Seeed Wiki tutorial with Riva config.sh details, screenshots, and demo video (reference only)