Local LLM Agent for Hardware Interface Control on Jetson

Use a local LLM running on Jetson to translate natural language commands into structured JSON for controlling hardware interfaces (lights, fans, thermostats, speakers) via GPIO, PWM, and I2C. The system uses Ollama with a custom prompt, FastAPI for the API layer, and confidence-based safety gating.

Execution model

Run one phase at a time. After each phase:

• Relay all output to the user.
• If output contains [STOP] → stop, consult the failure decision tree.
• If output ends with [OK] → tell the user "Phase N complete" and proceed.

Prerequisites

Phase 1 — Install Ollama (~2 min)

curl -fsSL https://ollama.com/install.sh | sh

Verify:

ollama --version

[OK] when Ollama version is printed. [STOP] if install fails — check network connectivity.

Phase 2 — Clone the project and install dependencies (~2 min)

git clone https://github.com/kouroshkarimi/llm_interface_controll.git
cd llm_interface_controll
pip install -r requirements.txt

[OK] when all packages install (FastAPI, uvicorn, etc.). [STOP] if pip fails — check Python version.

Phase 3 — Create the custom Ollama model (~2 min)

The project includes a system prompt file that constrains the LLM to output only structured JSON with fields: intent, device, action, location, parameters, confidence.

cd llm_interface_controll
ollama create jetson-controller -f models/jetson-controller.txt

Verify:

ollama list | grep jetson-controller

[OK] when jetson-controller appears in the model list. [STOP] if creation fails — ensure the base model (llama3.2:1b) is available: ollama pull llama3.2:1b.

Phase 4 — Start the FastAPI server (~1 min)

cd llm_interface_controll
uvicorn app.main:app --host 0.0.0.0 --port 8000

[OK] when uvicorn prints "Application startup complete".

Phase 5 — Test command execution (~1 min)

In a separate terminal, send a test command:

curl -X POST http://localhost:8000/command \
  -H "Content-Type: application/json" \
  -d '{"text": "Turn on the kitchen lights"}'

Expected response: JSON with intent: "control_device", device: "lights", action: "turn_on", location: "kitchen", confidence: >0.8.

Test a query:

curl -X POST http://localhost:8000/command \
  -H "Content-Type: application/json" \
  -d '{"text": "What is the temperature in the bedroom?"}'

Expected: intent: "query_status", device: "thermostat".

Test rejection of off-topic input:

curl -X POST http://localhost:8000/command \
  -H "Content-Type: application/json" \
  -d '{"text": "Write me a poem about the ocean"}'

Expected: intent: "unknown", confidence: 0.0.

[OK] when all three tests return properly structured JSON.

Failure decision tree

Reference files

• references/source.body.md — Full project documentation with architecture details, JSON schema specification, customization guide, and safety constraints (reference only)