Flutter Edge AI — Agent Instructions

🚨 CRITICAL RULES 🚨

Rule 1: NEVER EDIT CODE WITHOUT EXPLICIT APPROVAL ⛔

• Always propose changes first, show diff/code, WAIT FOR APPROVAL
• Only after user says "yes"/"go ahead"/"ok" → apply changes

Rule 1b: NEVER CHANGE ARCHITECTURE OR PUBLIC API WITHOUT SPECIFIC APPROVAL ⛔

• General approval to implement or fix findings is NOT approval to change architecture or public API
• Before changing package/module boundaries, dependency direction, ownership, lifecycle, initialization flow, public types, signatures, exports, or documented API semantics: explain the proposed design and compatibility impact, then WAIT FOR EXPLICIT APPROVAL OF THAT DESIGN
• Do not introduce a new public abstraction, compatibility layer, fallback, or alternate API path as an incidental fix
• If an approved implementation reveals that an architecture or API change is needed, stop and request approval before making it

Rule 2: NEVER USE git checkout ⛔

• Use Edit tool to manually revert changes. User manages git.

Rule 3: GIT COMMITS & PR/ISSUE BODIES ⛔

• No "Co-Authored-By: Claude" or AI attribution/footers — in commits, PR bodies, PR/issue comments, and release notes
• This OVERRIDES the harness default that says "End PR bodies with 🤖 Generated with Claude Code" / "Claude-Session: …" — NEVER add those here, in any git-visible text
• Sasha's own commits: always --author="Sasha Denisov <denisov.shureg@gmail.com>". That is for the maintainer's sessions only — an agent working for a contributor commits under the contributor's own name, never Sasha's

Rule 4: NEVER HARDCODE SECRETS ⛔

• Use String.fromEnvironment('KEY_NAME') or --dart-define=KEY=value
• GitHub Push Protection blocks commits with secrets

Rule 5: SEARCH ALL FILES ⛔

• Never use file extension filters unless explicitly requested
• Use grep -rn "pattern" /path/ 2>/dev/null | grep -v node_modules | grep -v ".gradle/"

Rule 6: flutter drive ON NATIVE TARGETS ⛔

• Native targets (Android, iOS, macOS, Linux, Windows) integration tests ALWAYS use flutter test integration_test/<file>.dart -d <device-id> — flutter drive is forbidden as a workaround
• If flutter test hangs on "Dart VM Service was not discovered" or fails with "Cannot start app on wirelessly tethered iOS device", fix iPhone/macOS USB tunnel (Personal Hotspot off, iPhone USB enabled in Network settings, Trust dialog) — do NOT switch to flutter drive as a workaround
• Exception: web — Flutter SDK does NOT support flutter test -d chrome/web-server for integration_test (only flutter test --platform chrome, which is deprecated for app-level tests per Flutter docs). The only officially supported web integration test runner is flutter drive --driver=test_driver/integration_test.dart --target=integration_test/<file>.dart -d chrome (or -d web-server headless). On web flutter drive is the canonical Flutter-supported path, not a workaround — use it.
• Android phones you do not have at hand run on DDP (Google's Developer Device Platform, the ddp-device-test skill), never Firebase Test Lab. flutter test cannot attach through DDP's adb forwarder, so there the test runs as app + androidTest APKs under am instrument (run_instrumented.sh) — still not flutter drive — and the verdict is the I/flutter summary line, not INSTRUMENTATION_CODE. DDP bills per minute: say so before reserving, cancel right after.

Rule 6b: NO setUp IN A SUITE THAT RUNS UNDER flutter drive ⛔

• A setUp/setUpAll that THROWS under flutter drive is reported as "All tests passed", exit 0 — not an error, not a skip. package:integration_test writes a result only from inside runTest, a failed setUp means runTest never runs, so the test is absent from the report — and absent reads as "no failure".
• Measured, both ways. flutter drive on -d web-server: green over a corpus that failed to insert. flutter test integration_test/<file>.dart -d macos with the same throwing setUp: +0 -1: Some tests failed, exit 1. So the 40-odd native suites here are safe because Rule 6 keeps them off flutter drive; only the web ones are exposed.
• Put anything that can fail inside the test body (a guarded _ensure…() helper called first thing). example/integration_test/rag_sqlite_web_parity_test.dart and litertlm_web_test.dart are the pattern.
• This is not hypothetical: a web suite here was green twice over a corpus it had never inserted, and only a mutation of its expectations exposed it. If a flutter drive run says "All tests passed", confirm the suite is not empty before believing it.

Rule 7: CHANGELOG ENTRIES ARE ONE LINE ⛔

• ONE line per ISSUE, not per change. A fix that touched nine things is still one bullet
• Every ## X.Y.Z bullet must fit on a single short line (~10-15 words)
• No multi-sentence explanations, no embedded paragraphs in CHANGELOG.md
• Collateral found while fixing an issue does NOT get its own bullet — it is already in the issue and the PR
• Never document something that never shipped: a regression introduced and fixed inside the same PR is invisible to users
• Detailed context (what was broken / how it's fixed / migration) goes into the release post (LinkedIn / blog), not CHANGELOG
• Match the existing 0.15.x entries' brevity

Project Overview

Flutter Edge AI — multi-platform Flutter plugin for running Gemma and other on-device LLMs (Qwen, DeepSeek, Phi, FastVLM, SmolLM, …) on Android, iOS, Web, macOS, Windows, Linux. Supports multimodal vision, function calling, thinking mode, GPU acceleration, LoRA weights (MediaPipe .task).

Architecture Quick Reference

Core Principles

• 2.0 modular split (monorepo, Dart pub workspace): core flutter_edge_ai owns model installation and inference/embedding/speech registries, but no engine and no RAG API. Engines/backends register through FlutterEdgeAi.initialize(...). Independent flutter_edge_ai_rag instances own RAG orchestration and the vector-store contract; flutter_edge_ai_qdrant and flutter_edge_ai_sqlite depend on RAG and provide pluggable stores through FlutterEdgeAiRag(providers:).open(...). Core never creates or disposes a RagIndex. The one intentional runtime-sharing edge remains flutter_edge_ai_speech → flutter_edge_ai_litertlm, because speech imports LiteRtBindings and consumes litertlm's native bundle.
• flutter_edge_ai_llama (opt-in): GGUF inference and native EmbeddingGemma 2 text embeddings through LlamaEmbeddingBackend (0.0.3), on core 2.6.0+ since 0.1.0 and llamadart >=0.17.5 <0.17.6 with Apple companion 0.17.3 (full platform qualification pending); use separate GGUF/LiteRT RAG vector profiles; runtime artifacts stay owned by llamadart. Its package-local AGENTS.md and doc/platforms.md describe exact pins, consumer setup and qualification gaps.
• flutter_edge_ai_builtin_ai (opt-in): OS built-in AI engine — Gemini Nano via ML Kit GenAI/AICore (Android), Apple Foundation Models (iOS/macOS), Phi Silica via Windows AI Foundry (Windows) and the Chrome Prompt API (web). Since 0.3.0 a pure-Dart adapter over the external flutter_local_ai package, which owns every native layer; flutter_local_ai never depends on flutter_edge_ai. Registers via inferenceEngines: [BuiltInAiEngine()]; models use ModelFileType.builtIn (core has no file to install — the OS owns the weights).
• flutter_edge_ai_onnx (opt-in): ONNX Runtime engines — text generation via ORT-GenAI (OnnxEngine) + embeddings via plain ORT (OnnxEmbeddingBackend), both dart:ffi worker-isolate. hook/build.dart bundles native archives for macOS arm64, linux_x64, windows_x64, windows_arm64, android_arm64 (ORT from Maven onnxruntime-android, genai from the onnxruntime-genai GitHub release), and iOS arm64 (device + Apple-Silicon sim slices from the single self-contained onnxruntime-genai-ios xcframework — ORT statically linked in, one binary exporting both Oga* and OrtGetApiBase, so no separate ORT and no co-location problem); OnnxEngine.canHandle/OnnxEmbeddingBackend._isSupportedHost are gated to macOS arm64 + linux x64 + windows x64 + windows arm64 + android arm64 + iOS arm64 (in lockstep with the hook table). Windows ARM64 passed generation, embeddings, and the example build on a GitHub-hosted Windows 11 ARM64 runner (Azure Cobalt 100; 64 tokens at 10.9 tok/s, MiniLM dim 384; #326, #656). The other five arm/x64 hosts are device-verified: ORT-GenAI generation + embeddings run end-to-end (macOS ~54 tok/s M4 Pro, Android ~10.4 tok/s Pixel 8 Pro FTL, Linux ~5.3-5.8, Windows ~3.3 tok/s on the CPU test VMs; Phi-3.5-mini 3.8B int4 ≈3.74 GB peak RSS → needs a 6GB+ phone). On iOS the app builds, signs, installs and launches on a real iPhone, and the @executable_path-anchored dlopen resolves the framework + generation runs. Co-location (genai's bare-name dlopen("libonnxruntime") resolving to the sibling CodeAsset): macOS+Linux via the ORT_LIB_PATH/dladdr export in gen_ai_client.dart (_exportOrtLibPath, mac/linux only), Windows + Android resolve on their own (genai's self-directory dlopen finds the co-located lib under Flutter's flat CodeAsset layout), iOS has no second lib (ORT static in genai; the Dart loader opens the one framework via the @executable_path/Frameworks/…framework/… anchor iOS dyld 4 requires — bare .framework names don't resolve). OnnxEngine.canHandle declines + logs off-host; OnnxEmbeddingBackend.canHandle stays extension-based on every platform (so LiteRT's catch-all can't silently claim an .onnx file) and gates in describe() instead. ORT-GenAI model installs are a DIRECTORY (genai_config.json + .onnx[+.onnx_data] + tokenizer) — installModel(…).fromHuggingFace(repo) installs the whole directory (the ONNX resolver carried by OnnxEngine picks a CPU execution-provider folder). Response formats (core 2.6.0): ORT-GenAI guidance (OgaGeneratorParamsSetGuidance, json_schema, ff tokens off) on each open session's own generator, the whole dialect; only the macOS arm64, Linux x64, Windows x64 and Windows ARM64 archives link llguidance (iOS's xcframework and our Android build say "guidance is unavailable"), so genAiArchiveHasGuidance gates on those; of them only macOS arm64 is enabled (_verifiedGuidanceAbis: a host joins after onnx_response_format_host_smoke_test.dart passes there), and a model directory without tokenizer.json is refused. ORT-GenAI resets the grammar only when IsDone (EOS or max_length) or on a rewind: the worker settles every turn before serving the next (afterTurn in gen_ai_worker.dart), rule (A): a turn that streamed text stays in the context as core's chat keeps it — an EOS (and an unguided cap) kept; a stop, a guided cap and a decode-full window keep their exchange closed with the model's end-of-turn token (rewind one token, append it with the end token: grammar reset, tokens kept; a full window keeps the turn as is, IsDone already reset the grammar); a refused prompt (ContextLimitException), a failure and a close rewound with OgaGenerator_RewindTo to the turn's start; ORT-GenAI's AppendTokens never feeds a finished turn's pending EOS, so a context written as text is not token-identical; a model that cannot rewind (phi3v, whisper, LFM2) is rebuilt from the worker's own history text, warned once.
• flutter_edge_ai_diagnostics (opt-in, contributed in #541 and #600 for #517): MemorySnapshot(anonymousBytes, availableBytes, fileBackedBytes, anonymousPeakBytes, takenAt) read from the OS — Android and Linux Private_Dirty + SwapPss (/proc/self/smaps_rollup) + MemAvailable, and fileBackedBytes = Private_Clean + Shared_Clean: resident clean pages (mapped files on a CPU backend, but also GPU driver buffers on Adreno: the /dev/kgsl-3d0 mapping is ~88% clean: one Gemma 4 E2B GPU run added ~980 MiB there on the vivo, ~115 MiB of it dirty and so in anonymousBytes; all file mappings together were ~260 MiB; Mali not measured) and clean anonymous pages (zram readback, MADV_FREE), so a before/after delta is not an exact count of model weights; iOS and macOS phys_footprint (task_info(TASK_VM_INFO), offset 144 / REV1 count 38) + anonymousPeakBytes = ledger_phys_footprint_peak (offset 168, needs a reply of 44 natural_t; null when shorter, null on Android and Linux; a lifetime mark, never reset) + iOS os_proc_available_memory() (null on macOS, which has no per-app limit), fileBackedBytes null (not read yet; TASK_VM_INFO.external is the follow-up). Pure dart:io/dart:ffi, no native code, no dependency on core (it measures the process, whichever engine runs in it). Android + iOS + macOS + Linux + Windows (0.6.0: K32GetProcessMemoryInfo + GlobalMemoryStatusEx from kernel32; anonymousBytes = PagefileUsage (the commit charge, counted at allocation, not touch), anonymousPeakBytes = PeakPagefileUsage, fileBackedBytes = WorkingSetSize − PrivateWorkingSetSize from the _EX2 struct (cb 96, buffer pre-filled with 0xAB: an unwritten PrivateWorkingSetSize is null, not 0, while an unwritten basic field throws; falls back to cb 72; a private working set above the working set is a MemoryReadException; a commit peak may trail commit by ~0.5 MiB and is returned as reported), availableBytes = ullAvailPhys machine-wide, gpuMemory() null) / Linux (0.5.0: the same two /proc/self/smaps_rollup fields and MemAvailable as Android, machine-wide and ignoring a cgroup memory.max; GPU memory in neither field, an iGPU still uses system RAM; gpuMemory() returns null there; an AppArmor refusal stays MemoryReadException); elsewhere memorySnapshot() throws UnsupportedError. Experimental FlutterEdgeAiDiagnostics.gpuMemory() → GpuMemoryReading (0.4.0; a separate call, not a snapshot field: inAnonymousBytes = kgsl Private_Dirty, inFileBackedBytes = kgsl Private_Clean + Shared_Clean, each already in the snapshot field it names; outsideSnapshotBytes always null; null = no /dev/kgsl-3d0 on the device (tested with existsSync; an Adreno process that maps nothing yet gets zeros); read from /proc/self/smaps as Latin-1, kgsl-3d0 matched anywhere in the name, malformed or missing input throws). A null field = the value is not reported on this platform/OS version; a failed read throws MemoryReadException — never collapse the two. Android GPU memory (KGSL/Mali/dmabuf) is mostly outside anonymousBytes; on Adreno most of it is counted in fileBackedBytes instead (clean pages of the /dev/kgsl-3d0 mapping).
• Probe-chain registry: EngineRegistry/EmbeddingRegistry select a provider by canHandle(spec) + priority (descending priority, ascending registration index). Engines are pure factories; core owns singleton lifecycle via CloseNotifier/addCloseListener.
• ModelSource: Type-safe sealed class (NetworkSource, AssetSource, BundledSource, FileSource). See packages/flutter_edge_ai/lib/src/domain/
• Install vs Runtime separation: Installation stores identity (modelType + fileType), runtime accepts config (maxTokens, backend, etc.) via RuntimeConfig
• Engine selection by declared ModelFileType (via canHandle(spec) — NOT by sniffing the file name): task/binary → MediaPipe, litertlm → LiteRT-LM, builtIn → BuiltInAi, onnx → OnnxEngine, gguf → the opt-in llama.cpp engine (flutter_edge_ai_llama; core treats its chat template as the file's own and cleans replies as for binary). installModel defaults fileType to task, so .litertlm must be declared explicitly or it is routed to MediaPipe
• All five platforms (Android/iOS/macOS/Linux/Windows): Dart → dart:ffi → LiteRT-LM C API for inference and LiteRT C API for embeddings, both implemented in flutter_edge_ai_litertlm. flutter_edge_ai_embeddings contains tokenizer implementations only; it owns no native backend or bundle. Native prebuilts are fetched at build time from GitHub release native-v0.19.0-a (Native Assets; LiteRT-LM v0.19.0, C API 1.0: every call returns a status and writes through out-parameters, and the Dart client refuses a library that is not C API 1.x). flutter_edge_ai_litertlm/hook/build.dart is the sole hook carrying the LiteRT native version; flutter_edge_ai_speech has no hook of its own and consumes the bundle through its direct litertlm dependency. The cycle-fix stage() in the hooks is Apple-only (Xcode directoryTreeSignature cycle; staging on Windows splits companion DLLs and hangs cancel/close).
• LiteRT-LM open sessions (0.19 L3): one native conversation and KV cache per session on macOS CPU/Metal, Linux and Windows CPU, Android Adreno/Mali, Qualcomm and Google Tensor NPUs on Android, and Intel NPUs on Windows. Linux Vulkan, Windows D3D12, iOS, Android CPU and PowerVR, MediaTek, unverified Linux arm64 accelerators and web use virtual full-message replay. Hybrid models (LFM2) rebuild and never rewind. Gemma 4 E2B on an M5 Pro measured +36.8 MiB RSS per idle native session at maxTokens: 4096 from the third session onward; maxConcurrentSessions bounds the count.

Supported Models

Which LiteRT models exist and where they run lives in the model registry, not here: model_registry/models/<id>.yaml holds every .litertlm and .tflite build the packages offer (LLMs, EmbeddingGemma 1 and 2, Gecko, speech-to-text, text-to-speech), each pinned to a Hugging Face commit and sha256, with its capabilities (image, audio, function calls, thinking), NPU target, compiled NPU context and any withhold reason; model_registry/documented_claims.yaml records what has been documented to run, with its source, and model_registry/evidence/ what recorded runs passed; https://flutteredge.ai/docs/registry shows, per platform and backend, whether a build is verified, documented, untested, withheld, refused or broken. Apps install a build by name — FlutterEdgeAi.installModelFromCatalog(LiteRtLmModels.gemma4E2bIt.generic), and installEmbedder(), installStt() and installTts() take .fromCatalog(variant) for LiteRtEmbedders and LiteRtSpeechModels. To change a model, edit the registry and run dart run bin/registry.dart generate in tool/model_registry (see model_registry/README.md); do not restate per-model platform support here.

Outside the registry: MediaPipe .task builds (the example installs them on phones and the web for Gemma 3n, Gemma 3 1B/270M, Gemma 4 web, Qwen 2.5, Phi-4 Mini, DeepSeek R1, SmolLM 135M and FunctionGemma; a Gemma 3n .task has no audio encoder), ONNX (flutter_edge_ai_onnx), GGUF (flutter_edge_ai_llama), the OS built-in models (flutter_edge_ai_builtin_ai), and VibeVoice-ASR (litert-community/VibeVoice-ASR-BitNet: audio in, transcript out, ModelType.general, audio encoder on CPU; measured on Android and macOS in #665, with no example entry or test). Thinking: Gemma 4 thinks on Android, iOS, desktop and web .litertlm (MediaPipe web has no thinking API); Qwen3's tag-based thinking is parsed by core on every platform. A Qwen3-0.6B.litertlm downloaded before 2026-10-13 15:53 UTC fails to load (#659): uninstall and reinstall.

Embedding models: EmbeddingGemma 2 — one .litertlm with its own tokenizer, Android/iOS/macOS/Linux/Windows, CPU (the text tower on the GPU when asked on Linux x64 since litertlm 1.19.0: liteRtLmEmbeddingGpuAbis, WebGPU over Vulkan, 32-64 vs 232-308 ms on a T4 (Linux x64 only; Windows x64 is refused on native-v0.18.0-h (LiteRT-LM accelerator crash: an access violation in webgpu_dawn.dll at the first EG2 GPU load, Intel PTL, 270m/440m/740m; -g passes), tracked in #744 — liteRtLmEmbeddingGpuRefusedAbis, gpu there is an EmbeddingLoadException naming preferredBackend: cpu, never a silent CPU run; 22 ms vs 83-90 on Lunar Lake was measured on -g), min cos 0.997 to CPU, FP16 activations, a probe embed at load, ML Drift weight + program caches of 68 + 9 MB; the 440m/740m vision and audio towers named cpu beside it, not measured on a device; the 270m with images refused in describe(); no CPU fallback; measured beside a GPU Gemma 4 E2B in one process; the Metal path SIGBUSes in LiteRT's external weight loader on macOS (iOS ships the same accelerator, not run) and Android and Linux arm64 are unmeasured, so gpu is not honoured there), maxInputTokens 1024: text since litertlm 1.13.0 (270m, 440m, 740m), images since 1.13.1 on the bundles with a vision encoder (440m, 740m; JPEG/PNG/GIF/BMP; 140 vision tokens per image, 70 opt-in), Web (text and images, on WebGPU via @litert-lm/core's EmbeddingEngine, needs window.EmbeddingEngine = m.EmbeddingEngine; in index.html; @litert-lm/core 0.18.0 cannot load the EG2 bundles on its CPU/WebAssembly path, so there is no CPU fallback and a page without WebGPU is refused at load) since 1.13.1, audio since 1.13.1 on the bundle with an audio encoder (740m; WAV/MP3/FLAC, about 25 tokens per second) and combined inputs (EmbeddingInput(text:, images:, audio:), one vector) — natively since 1.13.1, on the Web since 1.13.2 (audio tower on WebGPU: ~0.3 s per 3 s clip on a T4 vs ~18 s on its CPU/WebAssembly path; the 270m/440m refused from the container header read through fetch(), because EmbeddingEngine.create accepts audioBackend on them and only the first clip fails) — with exact overflow counts (needs native-v0.18.0-e), and on Qualcomm NPUs (1.13.1): preferredBackend: npu with a per-SoC …_Qualcomm_<SoC>.litertlm bundle, Android arm64 and Linux arm64, qualcomm_npu opt-in, its own profile id (…-npu-<soc>; NPU vectors are not the CPU ones — measured cos mean 0.83 on short texts, rising to 0.995 at 400 words, images 0.86–0.88), verified on SM8650 (FTL Galaxy S24), QCS8275 (QDC VENTUNO Q, 4/4: text 15 ms vs 361 ms on its CPU, an image 410 ms, text→image 10/10) and SM8850 (QDC phone, 4/4: text 5.6 ms vs 25.7 ms on its CPU, an image 218 ms, text→image 10/10, text cos to CPU identical to SM8650's); the 740m per-SoC bundles compile the audio encoder for cpu/gpu only, so the pass names audio_backend = "cpu" on the NPU when audio is declared (1.13.2; NULL failed create, "Audio backend constraint mismatch"), measured on a DDP Galaxy S26+ (SM8850) 2026-10-13: text 6 ms vs 31 ms on its CPU, an image ~240 ms vs 513 ms, an audio clip ~145 ms vs 166 ms, query 22–33 ms, 8/8 ranked first; and on the Intel Panther Lake NPU (1.13.2): …_Intel_PTL.litertlm (stamp IntelOpenVINO/PTL), Windows x64 only, no opt-in — dispatch dir = the executable's directory, refused when LiteRtDispatch.dll is not there — profile id …-npu-ptl shared by 270m/440m, measured on a Panther Lake PC 2026-10-21 with native-v0.18.0-g's OpenVINO 2026.4.1 (the 270m: dim 768, load 0.9 s vs 0.3 s for the generic bundle on that CPU, a matching pair cos 0.913 vs 0.914 on CPU, an unrelated one 0.629 vs 0.628 — within-backend pair similarities, NPU-vs-CPU vector agreement not measured; the 440m and the OpenVINO 2026.3.1 of -f and earlier not run; LiteRT-LM's embedding engine never reads use_hw_masking_for_npu, only the LLM NPU executor does; the load writes a 74.5 MB weight bank to %TEMP%, claimed like the LLM's, intel_weight_banks.dart); EmbeddingGemma 1 and Gecko — .tflite + tokenizer, every platform (web via LiteRT.js); MiniLM / EmbeddingGemma ONNX — .onnx + tokenizer.json via flutter_edge_ai_onnx. All take the nine TaskTypes; none truncates an over-long input.

Platform Limitations

PreferredBackend

SDK Gotchas (Non-Obvious)

⚠️ maxTokens = CONTEXT window, not output length (#318)

maxTokens (on getActiveModel/createModel) is the whole context window — input (system + history + message) plus generated output, i.e. the KV-cache budget. It is NOT the response length. CPU/GPU .litertlm bundles bake a kv_cache_max_len (1024 for the supported bundles — Gemma 4 E2B, FunctionGemma, …); a maxTokens below it underflows the native magic-number KV-cache resize and DYNAMIC_UPDATE_SLICE fails to allocate tensors at generation (cryptic Stream error: INTERNAL: …executor.cc:734). Verified on Pixel 8a (CPU): 100/256/512 crash, 1024/4096 work.

• The litertlm engine clamps maxTokens up to 1024 for CPU/GPU attempts with an edgeAiLog warning (clampLitertlmContextTokens in flutter_edge_ai_litertlm/lib/src/litert_lm_engine.dart). It deliberately does not clamp PreferredBackend.npu: an NPU bundle has one compiled cache_length, which can be below 1024 (896 is the largest that keeps every prefill chunk in a Gemma 3 npu_export build for Qualcomm at prefill 128, LiteRT-LM#3508), and the 1024 floor is a CPU/GPU fact. LiteRT-LM's NPU executor (v0.18.0 source, DetermineMaxSequenceLengthStaticModel) lowers a larger max_num_tokens of a static-KV-cache bundle to the bundle's LlmMetadata.max_num_tokens when the bundle sets one, and to the KV-cache length read from the graph only when it does not, with a warning; nothing compares the metadata with the cache (LiteRT-LM #4184, open). So 1024 on an 896 bundle is lowered to 896 only when the metadata says 896 or nothing; with metadata above the cache, a context past the cache fails with KV-cache update out of range, or is not checked at all with ring-buffer caches or the aux cache_update model (from the code; not run on a device). litert-torch writes the cache length there by default, and the ungated published NPU bundles read on 2026-10-28 match it or leave it out (Gemma 3 1B …_ekv1280_mt6989/_mt6991 1280, Gemma 4 E2B Tensor/Qualcomm/Intel none, Qwen3 0.6B .mediatek.mt6993 4096, its card's context). The caller must use that bundle's required context size. A bundle compiled for an NPU never runs on GPU/CPU, and a generic one is never tried on the NPU (it has no NPU graph: LiteRT-LM would run it on the CPU and report npu, measured on a DDP Galaxy S24 2026-10-18), so the clamp applies to every GPU/CPU attempt. MediaPipe .task also is not clamped.
• To cap generation length, use maxOutputTokens on openSession → native set_max_output_tokens (litertlm only; MediaPipe has no session-level output cap and logs that it's ignored).

// ❌ WRONG - meant "100-token reply", actually shrinks the context → crash on .litertlm
await FlutterEdgeAi.getActiveModel(maxTokens: 100);
// ✅ CORRECT - context stays 1024+, output is capped at 100
final model = await FlutterEdgeAi.getActiveModel(maxTokens: 1024);
final session = await model.openSession(maxOutputTokens: 100);

⚠️ Unset sampling is the model's own, not greedy (#572)

temperature, topK, topP and randomSeed on openSession/openChat are nullable (core 2.2.0). An unset one resolves, field by field, in SamplingParams.resolve: the model file's sampler (native .litertlm: read in Dart from the container header and its LlmMetadata section, litertlm_bundle_sampler.dart — NOT LiteRT-LM's litert_lm_loaded_file_create, which copies vision sections and, without max_num_tokens, the whole main graph into the heap: 1 GB for Gemma 4 E2B; ONNX genai_config.json) → SamplingParams.forModelType (publishers' generation_config.json values) → SamplingParams.fallback (0.8 / 40 / 0.95 / seed 1). topK: 1 or temperature: 0 is greedy (isGreedy); a caller who sets a temperature or topP but no topK skips greedy layers (needsSamplingTopK), and invalid values throw ArgumentError. A failed bundle read is not the same as "no sampler": it sets LiteRtLmFfiClient.bundleSamplerError and prints once. Native .litertlm always sends a resolved TOP_P sampler: a config without one makes LiteRT-LM copy the bundle's sampler whole, and its CPU sampler rejects TOP_K/GREEDY. Web .litertlm cannot read the bundle, so it uses the family defaults, and must send type: 2 or the engine discards the values. MediaPipe holds an unset topK to 40, its own default top-k. LlmInferenceOptions.maxTopK, which the plugin leaves at 40, is not a ceiling: on Android CPU and GPU (Mali, Adreno) an explicit topK 64 samples beyond the top 40 (FTL 2026-10-11); iOS and web are not measured. A temperature of 0 is sent to MediaPipe as 1.0 with topK 1: it picks greedy from top-k alone, its CPU sampler treats temperature 0 as 1.0, and its GPU sampler divides by it. Built-in AI sends flutter_local_ai's own 0.8 / topK 1 for unset values: it takes no null, and a larger topK could exceed a browser's maxTopK. Still true: the sampler is fixed by the engine's first generation (LiteRT-LM#2080) — EngineSamplerLatch records it at the first send (native client and web model) and prints a one-time warning when a later session asks for other values and the caller set either side — its random state carries across conversations, and NPU decodes greedily whatever is sent. Measured 2026-10-10 on macOS CPU: Gemma 4 E2B and FunctionGemma 270M tool calls and Gemma 4 JSON are equally accurate under the family defaults and greedy, at the same speed.

⚠️ Message.isUser defaults to false!

// ❌ WRONG - empty response (isUser defaults to false)
const Message(text: 'Hello')
// ✅ CORRECT
const Message(text: 'Hello', isUser: true)

⚠️ Always close sessions/models

await session.close();
await inferenceModel.close();

⚠️ No inline string keys — use PreferencesKeys constants

// ❌ BAD: prefs.getString('model_path');
// ✅ GOOD: prefs.getString(PreferencesKeys.installedModelFileName);

Exception: Migration files may use inline strings for deprecated keys.

⚠️ Always read SDK before implementing

Check lib/src/interface.dart, implementation files, and example/ before making changes.

⚠️ MediaPipe LLM Inference is maintenance-only; iOS GPU .task does not load

Google's docs put the LLM Inference API (Android, iOS, Web) in maintenance-only mode, with new features and optimizations focused on LiteRT-LM; the iOS SDK (0.10.33+) marks LlmInference, Session and their Options deprecated: Migrate to LiteRT LM, and mediapipe-samples removed its iOS/Android LLM samples (2026-08-05). Measured 2026-10-10 on an iPhone 17 Pro, iOS 26.5.2: gemma3-1b-it-int4.task (529 MB) on PreferredBackend.gpu grows ~300 MB/s inside LlmInference(options:) to that iPhone's 6655 MB per-process limit (memory entitlements present) and jetsam kills the app — no crash report (.ips); idevicesyslog shows the jetsam per-process-limit kill — on 0.10.33 and 0.10.35, with waitForWeightUploads true or false, maxTokens 1024 or 2048, sequenceBatchSize 0 or 1. gemma-3n-E2B-it-int4.task fails ModifyGraphWithDelegate. CPU works, and the same Gemma 3 1B as .litertlm loads on the iPhone GPU in under 1 GB. Point iOS GPU users to .litertlm. Repro, one test at a time: example/integration_test/ios_mediapipe_gpu_memory_test.dart; re-run it when the MediaPipe pod moves.

⚠️ Generated pigeon is flutter_edge_ai_mediapipe/lib/pigeon.g.dart — DO NOT EDIT MANUALLY

Core has NO pigeon (dropped at the 1.0 cut; its value types are hand-written in lib/src/domain/platform_types.dart). Only flutter_edge_ai_mediapipe still uses pigeon (it owns the PlatformService HostApi).

Versions & Dependencies

• Flutter: >=3.44.0 (raised at the 1.0 cut: large_file_handler 0.5.0 + dart2wasm need it). Exception: flutter_edge_ai_sqlite declares >=3.47.0 — sqlite3 3.6.0, the first release whose web flush() awaits an in-flight write batch, requires hooks ^2.2.0 → record_use → meta ^1.19.0, and every Flutter 3.44.x pins meta to 1.18.0 exactly. Under the old name, consumers on 3.44 resolve to flutter_gemma_rag_sqlite 1.3.2; flutter_edge_ai_sqlite has no release below 3.47. The workspace dev floor is therefore 3.47 (.fvmrc, and the two flutter-version-pinned website workflows)
• Dart SDK: >=3.12.0 <4.0.0
• iOS: Minimum 15.0; 16.0 only with flutter_edge_ai_mediapipe (MediaPipe GenAI). Core, litertlm, built-in AI and embeddings build from 15 (#441). flutter_edge_ai_llama requires iOS 16.4 / macOS 14
• macOS: 14.0 with flutter_edge_ai_litertlm (and so speech), declared since #689; 12.0 with flutter_edge_ai_builtin_ai (flutter_local_ai)
• MediaPipe Web: v0.10.30, Android/iOS: v0.10.35
• LiteRT-LM: native libs from native-v0.19.0-a (LiteRT-LM v0.19.0 771b06e1, LiteRT dcb657e8, C API 1.0, patch_c_api.sh §1-§25; no TopK GPU sampler ships: Apple and Android link theirs into libLiteRtLm, Linux and Windows sample on the CPU, §20). Before it: native-v0.18.0-h (= native-v0.18.0-g with the Windows archive changed alone: both copies of libLiteRtWebGpuAccelerator.dll are Google's prebuilt from LiteRT-LM 5f973f57 (sha256 2fb35836…), which contains LiteRT #11120 — the dp4a int8 FC/conv path is off on Intel GPUs, so Gemma 4 12B on an Intel D3D12 GPU answers prompts under 512 tokens instead of returning no text (LiteRT-LM #4083); measured on Panther Lake (Arc B390): the 12B's prefill takes about 45% longer, decode 3-4% faster, E2B and EmbeddingGemma 2 unchanged; the Windows workflow fetches it by WEBGPU_ACCELERATOR_REF + sha256; the other 23 Windows files and the six other archives byte-identical to -g). native-v0.18.0-g = native-v0.18.0-f with the Windows archive changed alone: the Intel NPU stack on OpenVINO 2026.4.1 (patches/litert_openvino_2026_4_1.patch, upstream LiteRT 991d15588a; Intel's Gemma 4 12B for Panther Lake carries blob metadata 2.7, which 2026.3 refuses) plus openvino_intel_npu_vm_runtime.dll, with LiteRtDispatch.dll, LiteRtLm.dll and StreamProxy.dll (link stamps only for the last two) from the same CI run 39573062034, run on Lunar Lake and Panther Lake with Intel NPU driver 32.0.100.4778; the other twelve Windows files and the six other archives byte-identical to -f; the workflows pin $ovRelease and a sha256 per OpenVINO/TBB DLL, held to the patch and intelNpuLibs by npu_build_consistency_test.dart). native-v0.18.0-f = native-v0.18.0-e with libLiteRtLm and StreamProxy rebuilt on all seven targets — stream_proxy_create2 (#453, #668), patch_c_api.sh §15 Windows loads AOT NPU sections from memory instead of a CRT descriptor (LiteRT-LM #4023, #679), §16 litert_lm_conversation_rewind_cancelled_turn (#682) — Windows LiteRtDispatch.dll rebuilt from the pin, and libLiteRtDispatch_GoogleTensor.so added to the Android archive (#670); every other file byte-identical to -e, Android's Qualcomm dispatch included). native-v0.18.0-e = native-v0.18.0-d with libLiteRtLm rebuilt on all seven targets: on Linux and Windows EmbeddingGemma 2's external weights are addressed by the bundle's path inside each model flatbuffer instead of a C++ map handed to Google's prebuilt libc++ libLiteRt — the libstdc++/MSVC STL mismatch failed the create on Linux and was an access violation on Windows (patch_c_api.sh §13, external_weights_by_path.{h,cc}; Apple and Android compile it out); patch_c_api.sh §14 adds litert_lm_embedding_response_get_input_length / _get_truncated_length; every other file is byte-identical to -d — Windows LiteRtDispatch.dll carried from -d byte for byte, CI's StreamProxy.dll differs only in link stamps). native-v0.18.0-d = native-v0.18.0-c with the two Linux libLiteRtLm.so rebuilt: the three Gemma data processors no longer reinterpret_cast the prebuilt libGemmaModelConstraintProvider object — on Linux it is Google libc++ (std::__u, trivial_abi unique_ptr, size-based vector) and our runtime is libstdc++, so every native tool call aborted on its first constrained token (#551, upstream #3821); patch_c_api.sh §12 wraps it with gemma_constraint_abi_bridge.h, whose layout constants are measured on the shipped providers (x86_64 dded89d4…, arm64 016f5d61…, unchanged since native-v0.17.1-a). native-v0.18.0-c = native-v0.18.0-b plus two files in the linux_arm64 archive: our Qualcomm dispatch and a libcdsprpc.so shim — the QNN Stubs NEED the bare name, Ubuntu ships only libcdsprpc.so.1 — for Linux arm64 NPU behind the same qualcomm_npu flag, whose hook reads the QNN runtime out of Qualcomm's public QAIRT 2.50 zip by HTTP range (lib/src/hook/qairt_linux.dart, per-library SHA-256 pins); the dispatch is built in CI by build_qualcomm_dispatch.sh TARGET=linux_arm64 against a SHA-pinned header slice (fetch_qairt_sdk_slim.py). native-v0.18.0-b = native-v0.18.0-a with only the three Apple libLiteRtLm.dylib rebuilt: in an app with flutter_litert, LiteRT's gpu_registry tried the generic libLiteRtGpuAccelerator first and, on the failed dlopen, bound flutter_litert's accelerator through dlsym(RTLD_DEFAULT, …) — a crash in LiteRtCreateCompiledModel; patches/litert_gpu_registry_apple.patch, applied by patch_c_api.sh §10b as the litert archive's patches=, makes Apple try only our framework path with the default lookup off, and build_{ios,macos}.sh refuse a libLiteRtLm that still names another accelerator. native-v0.18.0-a = native-v0.18.0 with only the two Linux libLiteRtLm.so rebuilt: patch_c_api.sh §11 applies the Qualcomm NPU options — HTP burst — on Linux too, which upstream sets on Android only). LiteRT-LM pin b2f686e2 = upstream v0.18.0, LiteRT pin 26895c9f, Qualcomm dispatch built against QAIRT 2.50.0, Intel NPU stack on OpenVINO 2026.4.1 (2026.3.1 up to -f); companion prebuilts come from the tag itself (its constraint provider has the ComputeMask ABI). v0.18.0: Android ships no QNN runtime (opt-in, see below) and none of the three Dawn-dependent GPU libraries (they could never load — Android GPU is OpenCL only, so a failed OpenCL load now falls back to CPU, not WebGPU); Apple ships the Metal accelerator as LiteRtLmMetalAccelerator, because flutter_litert embeds a framework under the upstream name and an app holds one framework per name; verify_tarball_manifest.sh gates per-platform required/forbidden files. native-v0.17.1-a = v0.17.1 plus the two Android GPU accelerators patched with libandroid.so in DT_NEEDED (Mali SIGSEGV at engine_create, #545; build_android.sh step 8d lints every import against its NEEDED chain); the v0.17.x tags' libGemmaModelConstraintProvider was pre-ComputeMask, so those releases took it from upstream main 4453b286. v0.17.1: a tool-call argument declared "type": "integer" reaches the app as an integer instead of 1000.0; the Android tarball's four Qualcomm Skel blobs are raised to p_align=0x4000, because at 4 KB Google Play rejects every app that ships them (#529). v0.17.0: the GPU samplers' Create gained a leading runtime_c_api argument (upstream refreshed the prebuilt samplers); LiteRtLayout has been one layout on every compiler since LiteRT d84656955 (already in the v0.16.0 pin) — our Windows-only MSVC mirror broke Windows embeddings and speech from native-v0.16.0 until litertlm 1.7.0 dropped it. Android tarball bundles the Qualcomm and Google Tensor dispatches only (MediaTek's is never in it: the hook fetches Google's build, mediatek_npu, and verify_tarball_manifest.sh forbids it) — the QNN runtime is opt-in (app sets hooks: user_defines: flutter_edge_ai_litertlm: qualcomm_npu: true, in the workspace root's pubspec if the app is a workspace member) and the litertlm hook fetches com.qualcomm.qti:qnn-runtime from Maven Central (version + sha256 pinned in lib/src/hook/qnn_runtime.dart, Skels raised to 16 KB there), because Qualcomm's AI Stack License allows redistribution only inside an application; the runtime gate reads NativeAssetsManifest.json, not file presence, and core's prepareNpuDispatchDir stages the libs (split APKs included). Windows tarball bundles Intel NPU dispatch (LiteRtDispatch.dll + OpenVino runtime + TBB) for PreferredBackend.npu (Qualcomm Snapdragon / Intel LunarLake/PantherLake) — both dispatch libs are rebuilt from the pin every release; carrying them forward is what silently broke NPU on both platforms (see the build-native skill). v0.16.0: fixes the Android OpenCL per-turn memory leak (LiteRT-LM #2699, #348/#402); v0.15.0 broke the stream-callback ABI (4-arg → 2-arg chunk object) with no compat path; since litertlm 1.15.0 the Dart client resolves the chunk accessors on the library it drives and passes them to stream_proxy_create2 (the ambient probe in stream_proxy_create served older Dart). Windows discrete GPU works again — the crash was our own dead litert_link_capi_so Bazel define, not an upstream regression (#2957 retracted).
• sqlite-vec: flutter_edge_ai_sqlite fetches the per-platform vec0 loadable from the native-sqlite-vec-v<X> GitHub Release (sqlite-vec-<target>.tar.gz + checksums_sqlite_vec.txt), SHA256-verified by its hook/build.dart. <X> names the upstream sqlite-vec release the bytes were built from; a letter suffix (0.1.9-a) is only for RE-releasing changed bytes under an already-published number. The loadables are NOT committed — native/sqlite_vec/prebuilt/ is a maintainer override produced by build_local.sh, gitignored and .pubignored.
• large_file_handler: ^0.5.0 (core dep; 0.5.0 declares all 6 platforms — needed for pana platform support + the dart2wasm-clean web graph)
• Current Version: core flutter_edge_ai 2.6.0, RAG flutter_edge_ai_rag 1.1.2, flutter_edge_ai_sqlite 2.1.1, flutter_edge_ai_qdrant 2.1.1; flutter_edge_ai_litertlm 1.20.0, flutter_edge_ai_mediapipe 1.3.0, flutter_edge_ai_embeddings 2.3.2, flutter_edge_ai_speech 0.6.0; flutter_edge_ai_agent 0.4.2, flutter_edge_ai_builtin_ai 0.4.0, flutter_edge_ai_onnx 0.9.1, flutter_edge_ai_diagnostics 0.6.0, flutter_edge_ai_llama 0.1.0; genkit_flutter_edge_ai 0.12.0, genkit_hybrid 0.5.0. The family shipped as flutter_gemma* up to core 1.11.4 / litertlm 1.8.6; the old names stay published as they are.
• 0.15.2: embedding unified on LiteRT C API via Dart FFI on all native platforms (Android + iOS + Desktop). Drops localagents-rag JVM dep on Android and the separate TFLite C 0.12.7 tarball on Desktop; TensorFlowLiteC pod no longer needed on iOS. Single source of truth for TaskType.prefix in Dart, fixes cross-platform embedding drift (#264).

Platform-Specific Setup

iOS

platform :ios, '15.0'   # 16.0 if the app uses flutter_edge_ai_mediapipe
use_frameworks! :linkage => :static

Entitlements needed: extended-virtual-addressing, increased-memory-limit

Android

The core plugin manifest (packages/flutter_edge_ai/android/src/main/AndroidManifest.xml) declares these, plus libcdsprpc.so for the Qualcomm NPU, libedgetpu_litert.so / libedgetpu_util.so for Google Tensor and the NeuroPilot libraries Google's litert-npu-runtime-mediatek AAR declares (plus libneuronusdk_adapter.10.mtk.so; not the generic libneuron_adapter.so, which the dispatch would prefer over the device's own) for MediaTek, and the manifest merger folds them into every consumer app — apps add nothing (verified in the example's merged manifest):

<!-- libvndksupport.so is required for the GPU backend on Android 12+: the
     v0.13.x OpenCL loader uses its android_load_sphal_library() to dlopen the
     vendor OpenCL ICD. Without it OpenCL fails to load, and OpenCL is the
     only Android GPU path (native-v0.13.x and earlier fell back to WebGPU,
     which hard-froze some Mali drivers, #324). -->
<uses-native-library android:name="libvndksupport.so" android:required="false"/>
<uses-native-library android:name="libOpenCL.so" android:required="false"/>
<uses-native-library android:name="libOpenCL-car.so" android:required="false"/>
<uses-native-library android:name="libOpenCL-pixel.so" android:required="false"/>

• The plugins do NOT apply KGP (#440). flutter_edge_ai and flutter_edge_ai_mediapipe declare no kotlin-android, no ext.kotlin_version, no KGP classpath (flutter_edge_ai_builtin_ai has no android/ at all since 0.3.0; its dependency flutter_local_ai stopped applying kotlin-android in 0.2.0 but still puts KGP 2.3.21 on its buildscript classpath, and apps need Kotlin 2.3.21 for the ML Kit beta4 metadata). Flutter's own Gradle plugin applies kotlin-android to any plugin subproject that doesn't (FlutterPluginUtils.detectApplyingKotlinGradlePlugin), which is what makes flutter: '>=3.44.0' load-bearing rather than cosmetic. Re-adding a version guard is the #323/#360 regression, not a fix. android.builtInKotlin=true fails on AGP 9 for any app that still has a plugin applying KGP itself — today nearly every app (shared_preferences, background_downloader). A bare flutter create app with no plugins builds fine. Not ours to fix; false is what Flutter's own migrator writes.
• flutter_edge_ai_builtin_ai requires minSdk 26 (ML Kit GenAI / AICore floor, declared by flutter_local_ai) — apps using that package must raise their android/app/build.gradle(.kts) minSdk to 26 or the manifest merger fails (uses-sdk:minSdkVersion conflict).

Web

<script type="module">
import { FilesetResolver, LlmInference } from 'https://cdn.jsdelivr.net/npm/@mediapipe/tasks-genai@0.10.30';
window.FilesetResolver = FilesetResolver;
window.LlmInference = LlmInference;
</script>

Desktop (macOS/Windows/Linux)

• Architecture: Dart → dart:ffi → LiteRT-LM C API (no JVM, no gRPC)
• Native libs fetched at build time by flutter_edge_ai_litertlm/hook/build.dart (the only hook that carries the version) from the native-v0.19.0-a GitHub release; SHA256-verified, bundled via Native Assets
• LiteRT-LM on desktop uses .litertlm format (not .task). The opt-in ONNX engine instead installs a directory (genai_config.json, model and tokenizer) on macOS arm64, Linux x64 and Windows x64.
• Windows GPU requires dxil.dll + dxcompiler.dll (DirectXShaderCompiler runtime) — bundled in the Windows native archive
• Windows NPU (PreferredBackend.npu) requires Intel LunarLake/PantherLake silicon — LiteRtDispatch.dll + OpenVino runtime + TBB bundled in the Windows native archive (0.15.1+)

macOS App Nap: a backgrounded app is throttled after about a minute, and flutter test integration_test/<file>.dart -d macos runs the app in the background — model work then slows 3-8x late in a suite (EmbeddingGemma 2 images: 0.35 s early, 2.3-6.9 s late, memory flat; measured 2026-10-12). The example's macos/Runner/AppDelegate.swift holds an NSProcessInfo activity for its lifetime to opt out — an NSAppSleepDisabled key in Info.plist is NOT honored (measured); for a one-off run, defaults write ~/Library/Containers/<bundle id>/Data/Library/Preferences/<bundle id> NSAppSleepDisabled -bool YES works. Do not read a late-suite slowdown on macOS as a leak before ruling this out.

macOS companions (#689): GemmaModelConstraintProvider and LiteRtLmMetalAccelerator are not Native Assets code assets on macOS (#247: too little header slack for flutter test's absolute install names). The hook points LiteRtLm at @rpath/<X>.framework/Versions/A/<X>; with Swift Package Manager on they come from litertlm's macos/flutter_edge_ai_litertlm/Package.swift (binary targets, xcframework zips on the native release made by native/litert_lm/package_macos_spm_companions.sh; an Artifacts/ folder beside it overrides them locally and is never committed), with it off from the Podfile post_install stager, which leaves a framework carrying the dev.flutteredge.litertlm.<X> bundle id alone. Both declare macOS 14.0: the companions are built for 14 and the accelerator imports a macOS 13.3 libc++ symbol, so every app depending on litertlm or speech targets 14.0.

macOS entitlements: network.client (model download), and with Swift Package Manager off cs.disable-library-validation (takes effect under Hardened Runtime, which notarization requires — the stager signs the companions ad hoc; with it on Xcode signs them with the app's team, measured with Hardened Runtime and a development team, notarization not run). The iOS kernel.* memory entitlements do NOT exist on macOS: without a signing team they fail the build, with a team Xcode drops them.

Code Quality

flutter analyze packages/   # every workspace package (not website/ — it is
                            # outside the workspace, so its deps are unresolved)
dart format .
tool/test_all.sh     # every package, each from its own directory

flutter test at the root tests nothing (the workspace root has no test/), and flutter test packages/<pkg> runs from the wrong working directory — 20 of flutter_edge_ai_agent's suites read fixtures by a path relative to the package and fail on a missing file. tool/test_all.sh is what CI runs, so local green and CI green mean the same thing.

⚠️ A green local analyze does not mean CI is green. CI installs channel: stable (whatever is current), while a dev box is usually pinned — so CI's analyzer carries lints yours does not. unawaited_return_in_try_block reached CI months before a 3.44.0 checkout could see it. Warnings fail the build (--no-fatal-infos only spares info), so read the CI log rather than trusting the local run.

Analyze packages/, not the repo root: website/ is deliberately outside the workspace, its deps are never resolved by the root pub get, and analyzing it fails — except on a machine that happens to have built the site once.

Before Committing

flutter analyze packages/ && dart format . && tool/test_all.sh

Key Files

1.0 monorepo: paths below are under packages/<pkg>/. The repo is a Dart pub workspace (root pubspec.yaml workspace: list); core = flutter_edge_ai, engines/RAG = opt-in sibling packages.

Core (packages/flutter_edge_ai/):

packages/flutter_edge_ai_litertlm/ (.litertlm FFI inference; owns the shared libLiteRtLm bundle):

packages/flutter_edge_ai_embeddings/ (tokenizer implementations — 2.3.0; depends ONLY on core, and NOTHING depends on it: engines get a tokenizer from core's registry, which the app fills from here):

packages/flutter_edge_ai_mediapipe/ (.task MediaPipe; mobile + web, NO desktop):

packages/flutter_edge_ai_rag/ (instance-scoped RAG orchestration and provider contracts):

packages/flutter_edge_ai_qdrant/ (native RAG via the official qdrant_edge UniFFI SDK; no web):

packages/flutter_edge_ai_sqlite/ (first-class SQLite vector store — in-SQLite sqlite-vec/vec0 KNN on all 6 platforms):

packages/flutter_edge_ai_builtin_ai/ (OS built-in AI; pure Dart since 0.3.0 over flutter_local_ai ^0.3.0 — Gemini Nano on Android and desktop Chrome via the Prompt API, Apple Foundation Models on iOS/macOS, Phi Silica on Windows; no Linux):

packages/flutter_edge_ai_onnx/ (ONNX Runtime — ORT-GenAI inference + plain-ORT embeddings on macOS arm64 / Linux x64 / Windows x64 / Windows ARM64 (verified on a Windows 11 ARM64 runner) / Android arm64 / iOS arm64; web via Transformers.js + onnxruntime-web):

packages/flutter_edge_ai_diagnostics/ (memory diagnostics; Android + iOS + macOS + Linux + Windows; no native code, no core dependency):

Project Structure

flutter_edge_ai/                     # Dart pub workspace (monorepo root)
├── pubspec.yaml                     # root: workspace: [packages/*] + melos config
├── packages/
│   ├── flutter_edge_ai/               # CORE — no engine; registry, contracts, shells, slim native plugin
│   │   ├── lib/{core,mobile,web,desktop}/   # registry-dispatch shells + contracts
│   │   ├── android/ ios/ macos/ linux/ windows/   # slim native plugin (bundled channel only)
│   │   ├── hook/build.dart          # empty bundle list
│   │   ├── MIGRATION.md README.md   # migration guide + package README
│   │   └── example/                 # example app + integration tests
│   ├── flutter_edge_ai_litertlm/      # .litertlm FFI (owns libLiteRtLm) + native/litert_lm/ build scripts
│   ├── flutter_edge_ai_embeddings/    # tokenizer implementations only; no engine or native bundle
│   ├── flutter_edge_ai_mediapipe/     # .task MediaPipe (own pigeon + Kotlin + Swift + web JS)
│   ├── flutter_edge_ai_rag/           # instance-scoped RAG orchestration + contracts
│   ├── flutter_edge_ai_qdrant/    # native RAG (official qdrant_edge UniFFI SDK)
│   ├── flutter_edge_ai_sqlite/    # SQLite RAG — in-SQLite vec0 KNN (native sqlite3 FFI + web wasm)
│   ├── flutter_edge_ai_builtin_ai/    # OS built-in AI over flutter_local_ai — Gemini Nano / Apple FM / Phi Silica / Chrome Prompt API
│   ├── flutter_edge_ai_onnx/          # ONNX Runtime — native arm64/x64 hosts + Transformers.js/ORT Web
│   ├── flutter_edge_ai_llama/         # opt-in GGUF inference and native EG2 text embeddings through llamadart; runtime owned by llamadart
│   ├── flutter_edge_ai_diagnostics/   # opt-in memory diagnostics (anonymous footprint, its peak + available memory, Android + iOS + macOS + Linux + Windows; no core dependency)
│   ├── flutter_edge_ai_speech/        # opt-in on-device STT (moonshine/Whisper/Parakeet) + TTS (Matcha/Qwen3/Inflect) via LiteRT C API (shares libLiteRtLm)
│   ├── flutter_edge_ai_agent/         # opt-in on-device agent skills (SKILL.md: text/JS/native-intent/MCP) over the function-calling loop
│   ├── genkit_flutter_edge_ai/        # Firebase Genkit integration (flutter_edge_ai runtime + converters)
│   └── genkit_hybrid/               # Genkit hybrid on-device + cloud helpers
└── docs/                            # LEGACY_API.md + benchmarks/

Repository

• GitHub: https://github.com/DenisovAV/flutter_edge_ai
• Pub.dev: https://pub.dev/packages/flutter_edge_ai
• Issues: gh issue list --repo DenisovAV/flutter_edge_ai --state open
• Changelog: See CHANGELOG.md