muesli

> **Post-compaction recovery:** PreCompact hooks auto-generate context handover files at `Context/handoff-summary-YYYY-MM-DD-<slug>.md`. After compaction, read the latest handoff file in `Context/` to restore session memory and resume work.

# Muesli

Local-first macOS app for **dictation** and **meeting transcription** on Apple Silicon. Speech-to-text runs on-device via CoreML/Neural Engine by default; users can explicitly opt into BYOK OpenAI Realtime dictation. Native Swift/AppKit — no Electron or Python runtime.

**Status:** Live and public. Available at [GitHub Releases](https://github.com/Muesli-HQ/muesli/releases). Signed, notarized, stapled.

## What It Does

- **Dictation:** Hold hotkey → speak → release → text pasted at cursor (~0.13s with Parakeet)
- **Meeting transcription:** Captures mic (You) + system audio (Others) → VAD-driven chunking → speaker diarization → AI-powered meeting notes
- **Meeting export:** Export notes or transcript as PDF (paginated US Letter) or Markdown via `MeetingExporter.swift`
- **Screen context:** Accessibility API captures app name + text around cursor for dictation context-awareness (opt-in, off by default)
- **ASR models:** Parakeet v3/v2, Whisper Tiny/Small/Medium/Large Turbo, Cohere Transcribe, Nemotron 3.5 Multilingual, SenseVoice Small, Qwen3 ASR, Bodhan Core and Flex (each with FP16/INT8 weights)
- **3 summarization backends:** OpenAI API key, OpenRouter API key, ChatGPT OAuth (subscription-based)
- **Camera-based meeting detection:** Requires mic + camera + recognized meeting app (camera alone won't trigger)
- **Join & Transcribe:** Extract meeting URLs from calendar events (Zoom, Meet, Teams, Webex, Chime, FaceTime), split button with "Join & Transcribe" / "Join Only" / "Transcribe Only", platform icons in notifications
- **Google Calendar integration:** Coming Up section, status bar, pre-meeting countdowns, event-driven notifications via `EKEventStoreChangedNotification`
- **Meeting templates:** Built-in and custom templates for structured meeting notes

## Building

Bodhan uses a CoreML encoder and native MLX decoder in both precisions. Its `mlx-swift` dependency requires Swift 6.3; use Xcode 26.6 on macOS 26 to build. See `AGENTS.md` for current build and cache instructions.

### Production build (signed, installed to /Applications)
```bash
./scripts/build_native_app.sh
```

### Dev/test build (isolated from production)
```bash
./scripts/dev-test.sh                         # Build MuesliDev.app (separate bundle ID, separate data)
./scripts/dev-test.sh --lane A                # Build MuesliDevA.app for a parallel worktree
./scripts/dev-test.sh --lane B                # Build MuesliDevB.app for another parallel worktree
./scripts/dev-test.sh --lane A --local-only   # Explicitly omit iCloud/APNs entitlements
./scripts/dev-test.sh --lane A --reset        # Re-run onboarding for lane A, keep lane data
./scripts/dev-test.sh --reset                 # Re-run onboarding for default MuesliDev, keep data
./scripts/dev-seed-from-prod.sh               # Copy production DB/config into MuesliDev safely
```

MuesliDev uses bundle ID `com.muesli.dev` and stores data at `~/Library/Application Support/MuesliDev/`. Named lanes use fixed identities: `MuesliDevA` / `com.muesli.dev.a` / `~/Library/Application Support/MuesliDevA`, then B and C with matching suffixes. Named lane executable/process names also match the lane app name. Production data is never touched.

Named lanes default to local-only signing through `scripts/MuesliLocalOnly.entitlements`, which omits iCloud and APNs entitlements for non-sync feature work. Use `--cloud-entitlements` only when the lane has a matching Apple Developer provisioning profile and the test actually needs iCloud/APNs behavior.

### SwiftPM build artifacts in worktrees
SwiftPM can write build artifacts to `native/MuesliNative/.build` inside the active worktree. That can consume several GB per worktree. Local scripts now resolve a shared SwiftPM scratch path through `scripts/muesli_spm_cache.sh`:

- Explicit `MUESLI_SWIFTPM_SCRATCH_PATH` wins.
- `MUESLI_SWIFTPM_SCRATCH_CHANNEL` overrides the channel segment under the resolved cache root.
- `MUESLI_EXTERNAL_SPM_CACHE_ROOT` overrides the default `/Volumes/MuesliBuildCache/muesli-spm` external cache root.
- If `/Volumes/MuesliBuildCache/muesli-spm` is mounted, scripts use that external APFS cache.
- Otherwise scripts fall back to `~/Library/Caches/muesli-spm`.
- `MUESLI_DISABLE_SWIFTPM_SCRATCH_PATH=1` intentionally opts out and uses SwiftPM's package-local `.build`; this takes precedence over all scratch path settings.

The preferred local cache is an APFS sparse bundle stored on the external SSD at `/Volumes/eSSD/MuesliBuildCache.sparsebundle`. Mount it before build-heavy local work:

```bash
hdiutil attach /Volumes/eSSD/MuesliBuildCache.sparsebundle
```

That sparse-bundle path is the maintainer's local SSD path. Contributors can substitute their own volume path or skip the attach step; scripts fall back to `~/Library/Caches/muesli-spm` when the external cache is not mounted.

Default script channels:

```bash
./scripts/dev-test.sh                 # /Volumes/MuesliBuildCache/muesli-spm/worktrees/<worktree>/dev when mounted
./scripts/build_native_app.sh release # /Volumes/MuesliBuildCache/muesli-spm/release when mounted
./scripts/release-preprod.sh          # /Volumes/MuesliBuildCache/muesli-spm/preprod when mounted
./scripts/release-alpha.sh            # /Volumes/MuesliBuildCache/muesli-spm/alpha when mounted
```

For parallel PR/worktree work, use isolated paths:

```bash
MUESLI_SWIFTPM_SCRATCH_PATH="/Volumes/MuesliBuildCache/muesli-spm/worktrees/pr182/dev" ./scripts/dev-test.sh
swift test --package-path native/MuesliNative --scratch-path "/Volumes/MuesliBuildCache/muesli-spm/worktrees/pr182/test"
```

The build script passes the resolved path to SwiftPM as `--scratch-path`, so multiple worktrees do not each grow their own `.build`. Caveat: do not run concurrent builds from different worktrees into the same scratch path; use separate paths per channel, agent, or simultaneous build. Deleting a scratch path only removes rebuildable SwiftPM artifacts, not installed apps or app data. Set `MUESLI_DISABLE_SWIFTPM_SCRATCH_PATH=1` only when you intentionally want package-local `.build`.

### Parallel dev lanes
Use fixed lanes for concurrent local testing instead of creating branch-named app identities:

```bash
./scripts/dev-test.sh --lane A
./scripts/dev-test.sh --lane B
./scripts/dev-test.sh --lane C
```

Each lane installs a separate app bundle under `/Applications/`, keeps a separate support directory under `~/Library/Application Support/`, and has a separate macOS permission identity. Grant permissions once per lane. Do not copy or reset TCC permissions unless explicitly testing permission prompts.

### Tests
```bash
swift test --package-path native/MuesliNative    # 1,148 @Test declarations across 120 suites
```

### Onboarding testing
```bash
# Reset onboarding without losing data:
./scripts/dev-test.sh --reset
./scripts/dev-test.sh --lane A --reset

# Reset macOS permissions only when intentionally re-granting TCC:
./scripts/dev-reset-permissions.sh
./scripts/dev-reset-permissions.sh --bundle-id com.muesli.dev.a --process-name MuesliDevA --app-path /Applications/MuesliDevA.app

# Then:
./scripts/dev-test.sh
./scripts/dev-test.sh --lane A
```
Note: config JSON uses snake_case keys (`has_completed_onboarding`, not `hasCompletedOnboarding`).

## CI/CD Pipeline

### Pull Requests
- **CI workflow** (`.github/workflows/ci.yml`): macOS 15 runners
  - `changes` → `build` → `cli-smoke` → `ci-gate` (required check)
- **Claude Code Review** — reviews every PR automatically
- **Greptile** — reviews every PR automatically
- **Vercel** — scoped to `site/` only
- **Concurrency** — stale CI runs auto-cancelled on new pushes

### Releases
```bash
./scripts/release.sh                   # Auto-increments version
./scripts/release.sh 1.0.0             # Explicit version
```
**Critical:** Staple the app bundle BEFORE creating the DMG, otherwise Gatekeeper rejects.

### Signing & Notarization
- Developer ID: `Pranav Hari Guruvayurappan (58W55QJ567)`
- Bundle ID: `com.muesli.app`
- Notary profile: `MuesliNotary` (Keychain)

## Key Architecture

```
native/MuesliNative/Sources/
├── MuesliNativeApp/              # Main app (~50 Swift files)
│   ├── MuesliController.swift    # Central orchestrator — dictation, meetings, onboarding, state
│   ├── TranscriptionRuntime.swift # Routes to ASR backends, post-processing, VAD + diarization
│   ├── FluidAudioBackend.swift   # Parakeet TDT on ANE
│   ├── Qwen3AsrBackend.swift     # Qwen3 ASR on ANE (macOS 15+)
│   ├── Qwen3PostProcessor.swift  # On-device GGUF LLM for dictation cleanup (opt-in)
│   ├── WhisperKitBackend.swift   # Whisper on CoreML/ANE via WhisperKit
│   ├── ScreenContextCapture.swift # AX-based app context for dictation + meetings
│   ├── MeetingExporter.swift     # PDF/Markdown export with NSPrintOperation
│   ├── OnboardingView.swift      # 7-step onboarding with real permission polling + dictation test
│   ├── OnboardingProgress.swift  # Crash-safe onboarding state persistence
│   ├── MeetingSession.swift      # Meeting lifecycle + diarization + screen context
│   ├── MeetingSummaryClient.swift # OpenAI / OpenRouter / ChatGPT summarization
│   ├── SystemAudioRecorder.swift # ScreenCaptureKit SCStream for system audio
│   ├── ChatGPTAuthManager.swift  # OAuth PKCE + WHAM API
│   ├── HotkeyMonitor.swift       # Global hotkey detection (modifier keys)
│   ├── MeetingDetector.swift     # Camera + mic + app detection for meetings
│   ├── MeetingNotificationController.swift # Join & Transcribe notification panel with platform icons
│   └── PasteController.swift     # Clipboard-preserving Cmd+V paste
├── MuesliCore/                   # Shared library (SQLite, paths, models)
│   ├── DictationStore.swift      # SQLite3 C API — dictations + meetings CRUD
│   └── MuesliPaths.swift         # App-identity-aware path resolution
└── MuesliCLI/                    # Agent-friendly CLI (JSON over stdout)
```

## Data Storage

- **Config:** `~/Library/Application Support/{AppName}/config.json` (snake_case keys)
- **Database:** `~/Library/Application Support/{AppName}/muesli.db` (SQLite WAL)
- **Models:** `~/Library/Application Support/FluidAudio/Models/` (shared across app identities)
- **Onboarding progress:** `~/Library/Application Support/{AppName}/onboarding-progress.json` (deleted on completion)
- **ChatGPT tokens:** macOS Keychain (`com.muesli.app.chatgpt-auth`)
- **Whisper models:** `~/.cache/muesli/models/`

`{AppName}` is `Muesli` for production, `MuesliDev` for dev, `MuesliCanary` for alpha — controlled by `MuesliSupportDirectoryName` in Info.plist.

## macOS Permissions

| Permission | What Uses It | API |
|---|---|---|
| Microphone | Dictation + meeting mic | AVAudioRecorder, AVAudioEngine |
| Accessibility | Paste text + screen context capture | CGEvent Cmd+V, AXUIElement |
| Input Monitoring | Hotkey detection | NSEvent global monitors |
| Screen Recording | System audio capture | ScreenCaptureKit SCStream |
| Camera (implicit) | Meeting detection | CoreMediaIO property listeners |
| Calendar (optional) | Upcoming meetings | EKEventStore, Google Calendar API |

**Critical:** Accessibility permission requires an app restart to take effect. The onboarding flow handles this with an automatic restart after the hotkey configuration step.

**Important:** `CGWindowListCreateImage` (screenshots) conflicts with active `SCStream` sessions — causes `RPDaemonProxy: connection INTERRUPTED` and breaks system audio capture. Never take screenshots during meeting recording. See `Context/handoff-2026-04-16-coreaudio-tap-migration.md` for the planned fix.

## Onboarding Flow

7 steps: Welcome → Model → Permissions → Hotkey → **[app restart]** → Dictation Test → Meeting Summaries → Google Calendar

Key implementation details:
- Real OS permission polling every 1s (not fake timers) via `AXIsProcessTrusted()`, `CGPreflightListenEventAccess()`, etc.
- Uses proper request APIs: `AXIsProcessTrustedWithOptions`, `CGRequestScreenCaptureAccess`, `CGRequestListenEventAccess`
- Hotkey, calendar, and mic monitors are **deferred until after onboarding completes** to prevent premature permission prompts
- App restart via detached shell: `/bin/sh -c "sleep 1; open -- \"$1\"" -- <bundlePath>` then `NSApp.terminate(nil)`
- Progress saved on every step transition to `onboarding-progress.json` (schema-versioned, atomic writes)
- Dictation test step uses the real hold-to-talk hotkey flow with `dictationTestCallback` routing. It never pastes, but intentionally keeps the floating waveform visible so onboarding exercises the same recording feedback as normal dictation; releasing the hotkey leaves the active state immediately.
- `OnboardingView.dictationTestStep` (static Int = 4) — hotkey monitor only starts when resuming at this step or later

## Screen Context (opt-in, `enableScreenContext` in config)

**Dictation:** `DictationContextCapture.capture()` — synchronous Accessibility API call:
- App name + bundle ID via `NSWorkspace.shared.frontmostApplication`
- Text before cursor via `kAXSelectedTextRangeAttribute` + `kAXStringForRangeParameterizedAttribute` (falls back to `kAXValueAttribute` suffix for apps that don't support parameterized attributes)
- Selected text via `kAXSelectedTextAttribute`
- Browser URL via `kAXDocumentAttribute`
- Only runs when BOTH `enableScreenContext` AND `enablePostProcessor` are true
- Context injected into Qwen3 post-processor prompt as `<APP-CONTEXT>` tags
- Stored in existing `app_context` column in `dictations` table

**Meetings:** `MeetingScreenContextCollector` (actor) — periodic AX capture every 60s:
- Uses same `DictationContextCapture.capture()` (no screenshots — `CGWindowListCreateImage` conflicts with `SCStream`)
- Deduplicated, aggregated, injected into meeting summary prompt as "Visual context" section
- OCR-based capture (`ScreenContextCapture.captureOnce()`) exists in code but is unused until CoreAudio migration

## Meeting Export

`MeetingExporter.swift` — export menu in `MeetingDetailView` content toolbar:
- Two menu items: "Export Notes"/"Export Transcript" (contextual to active tab) + "Export Full Meeting"
- Format (PDF/Markdown) chosen via `ExportFormatAccessory` popup in NSSavePanel
- PDF: `NSPrintOperation` with paginated US Letter pages (612x792pt, 1" margins)
- Markdown: atomic write with metadata header (title, date, duration, word count, template)
- NSSavePanel presented via `beginSheetModal(for:)` — never `runModal()` (deadlocks in SwiftUI)
- File auto-opens in default app after save via `NSWorkspace.shared.open(url)`

## Development Workflow

1. **Feature work:** Create branch → implement → `./scripts/dev-test.sh` → push → PR
2. **PR review:** Claude Code + Greptile review automatically. Fix P1s before merge.
3. **Merge to main** via squash merge
4. **Release:** `./scripts/release.sh` → notarize → GitHub Releases

## Calendar Notification Pipeline

Event-driven architecture for meeting notifications:

- **Primary trigger:** `EKEventStoreChangedNotification` — macOS pushes calendar changes (add/move/delete) instantly via `NotificationCenter`. Immune to App Nap timer suspension in LSUIElement apps.
- **Fallback:** 60s `Timer` polls Google Calendar API (sync token for efficiency) and checks the 5-minute notification window for time-based triggers.
- **Dedup:** Composite key `id|startDate` — rescheduled events generate fresh notifications. Stale entries pruned hourly.
- **Per-event timers:** `meetingStartingNowTimers: [String: Timer]` — concurrent events get independent "starting now" timers.
- **Suppression:** After user acts on a calendar notification (Join Only, Dismiss), mic/camera detection is suppressed for the remaining event duration.
- **Meeting URL extraction:** EventKit (`event.url`, `location`, `notes` via regex) + Google Calendar API (`hangoutLink`, `conferenceData.entryPoints[type=video]`). `mergeEvents` backfills Google URL when EventKit duplicate has none.

**macOS 26 App Nap behavior (LSUIElement apps):** All timer mechanisms (`Timer.scheduledTimer`, `DispatchSourceTimer`, `Task.sleep`, `Thread.sleep`, `DispatchQueue.asyncAfter`, POSIX `nanosleep`) get suspended by aggressive power management. Only `NotificationCenter` observers (system IPC) are immune. The 60s fallback timer may not fire reliably — `EKEventStoreChangedNotification` is the critical path. Users with Google Calendar synced to macOS Calendar (System Settings > Internet Accounts) get reliable notifications via EventKit. OAuth-only users depend on the 60s timer.

## Known Limitations

- **Nemotron 3.5 Multilingual (`nemotron35`):** Supported local Nemotron ASR backend. Ships the FluidInference `multilingual/2240ms` variant (~665 MB, vocab 13087, blank 13087). Multilingual incl. Hindi/Chinese/Japanese + 100+ locales via `prompt_id`. In-app **language picker** (`Nemotron35Language` enum → `prompt_id`; config key `nemotron35_language`, default `auto`=101): the controller pushes the selected prompt id to the coordinator (`setNemotron35PromptId`), which applies it to the actor on load/select. Picker UI lives in the Models tab card (mirrors the Cohere language card). A **model-update check** records the HF repo commit sha at download (`<cache>/.revision`) and surfaces an "Update" affordance in the Models tab when the repo's `main` advances (never auto-downloads). Native punctuation (in-vocab). Append-only/no corrections, weak on very short dictations. 2240ms chunk latency (35840 samples). Cache shape `[1,24,42,1024]`. Uses shared `RNNTStreamState` helpers through the `NemotronStreamingTranscribing` protocol; `StreamingDictationController` takes a `chunkSamples` override. Model cached at `~/.cache/muesli/models/nemotron35-multilingual-2240ms/`. Offered in onboarding under "Other models". **Dictation modes:** hold-to-talk uses the normal record→transcribe-file path (`transcribeWithNemotron35`); double-tap uses live streaming (`StreamingDictationController`). `handleStart` allows hold-to-talk; prepare/arm pre-warm stays skipped for streaming backends (`isStreamingDictationBackend`) so the double-tap detection window stays clean.
- **Qwen3 ASR:** 2-3s latency (autoregressive decoder). First run after launch has ~30s CoreML compilation warmup.
- **ChatGPT OAuth:** Uses reverse-engineered WHAM API. Could break if OpenAI changes the API.
- **Speaker diarization:** Post-processing only. Runs after meeting stops.
- **Screen context OCR disabled during meetings:** `CGWindowListCreateImage` conflicts with `SCStream`. AX-based context used instead. Planned fix: migrate to CoreAudio tap for system audio (see `Context/handoff-2026-04-16-coreaudio-tap-migration.md`).
- **NSSavePanel:** Must use `beginSheetModal(for:)` in SwiftUI, never `runModal()`. `NSAttributedString(html:)` deadlocks on main thread — build attributed strings manually.
- **App restart during onboarding:** Uses `exit(0)` via detached shell. `NSApp.terminate(nil)` inside SwiftUI animation context can crash.
- **macOS 26 App Nap:** LSUIElement apps have all timers suspended by aggressive power management. Calendar notifications rely on `EKEventStoreChangedNotification` (immune). The 60s Google Calendar poll timer may not fire. See Calendar Notification Pipeline section.
- **"Meeting starting now" after Join Only/Dismiss:** The scheduled timer is not cancelled when the user clicks Join Only or Dismiss on the "Upcoming meeting" notification. A redundant "Meeting starting now" fires at event start time. Fix: pass notification key into `handleUpcomingMeeting` so callbacks can cancel it.

## Upcoming Work

1. **Cancel "starting now" timer on Join Only/Dismiss** — Pass notification key into `handleUpcomingMeeting` so `onJoinOnly`/`onDismiss` callbacks can cancel `meetingStartingNowTimers[key]`.
2. **CoreAudio tap migration** — Replace ScreenCaptureKit with CoreAudio aggregate device for system audio. Unblocks OCR during meetings + friendlier "System Audio" permission (not "Screen Recording"). See `Context/handoff-2026-04-16-coreaudio-tap-migration.md`.
3. **Google OAuth verification** — Pending Google approval (~4 weeks from April 12). Once approved, embed credentials with `verified: true`.
4. **Post-processor fine-tune** — Collect `postproc-pairs.jsonl` from canary testers, train v3 model for better implicit list formatting.

More agent context in Muesli-HQ/muesli

4 other files this repository gives its agents.

Also found in one other repository

The same file, byte for byte, in the weekly crawl of public GitHub.

• pHequals7/muesli1.4k