Running several AI models within a desktop memory budget

Start with the competing workloads.

Doppel uses an MLX language-and-vision model alongside smaller embedding, ranking, extraction and speech models. Those jobs have different urgency: a foreground answer and a background indexing pass do not need the same scheduling policy.

The implementation uses a shared model-memory coordinator. Services reserve capacity before loading. Admission checks both estimated model residency and the process’s observed physical footprint. When capacity is tight, eligible idle models can be evicted in a deterministic order.

Tracking both quantities matters: removing a model reference and seeing its memory return to the operating system are separate events. Crediting the same released memory twice can admit more work than the process can safely hold.

Choose compute placement per model.

The language-and-vision path uses MLX. Doppel’s Core ML embedding and speech backends are configured for CPU and Neural Engine execution, keeping those paths separate from the MLX GPU heap. Core ML decides placement within the permitted compute units.

Conversion tooling produces explicit input shapes for common workloads. Jina embeddings have 128-, 256- and 512-token variants. The speech bundle has separate audio buckets and a stateful decoder contract. Each shape and runtime path needs its own checks against the source implementation.

File size alone cannot tell us the full inference footprint. We also need to account for model state, temporary buffers and simultaneous work.

Reuse state with explicit boundaries.

Repeated chat turns often share a token prefix. Doppel keeps a bounded prefix-cache registry and validates token ledgers and model-family cache geometry before reuse. Conversation lineage determines whether an existing state can be restored or a matching prefix forked.

The canonical transcript remains separate from the working context sent to the model. Compaction creates a disposable summary; source messages, tool results and their identities remain stored. Edits and branch changes invalidate summaries whose source revision is stale.

That keeps performance work tied to a clear consistency contract: cached state must belong to the input actually being processed.

Make speech failure recoverable.

Core ML state can be expensive to allocate repeatedly. The ASR backend reuses decoder state, explicitly clears its named buffers between utterances and drains temporary allocations around inference steps.

Recognition work is bounded and cancellable between native predictions. Supported Objective-C exceptions are contained at a native boundary. A native exception disables that recognizer for the process and exposes restart guidance; it does not repeatedly reload the same potentially damaged runtime.

The meeting pipeline preserves the original audio after a processing failure and supports an explicit retry. Resource control and durable input recovery address different parts of reliability.

Qualify the complete operation.

A memory policy for a 16 GB Mac defines scheduling and admission limits. Qualification on physical 16 GB hardware establishes how the full workload behaves there.

For a useful comparison, keep the model revision, input set and generation settings fixed, then report:

• Time to the first useful response and end-to-end completion latency.
• Peak physical memory during overlapping chat, speech and indexing work.
• Behavior under cancellation, model eviction and memory pressure.
• Output parity after conversion, and quality on the product’s evaluation cases.

Measure the experience with the surrounding application running. A fast isolated model can still produce a slow or fragile product when its residency and background work compete with everything else.

See the product context.

Doppel combines these runtime mechanisms with hybrid search, meeting review and checks on supporting sources.