Local AI Voice Assistant
Jarvis is a 100% local, low-latency voice assistant and autonomous agent running fully on-device. Developed under direct guidance, the system ensures complete user privacy while orchestrating tools and system files asynchronously using the Model Context Protocol (MCP).
Project Overview
Jarvis is a fully private, voice-enabled assistant capable of interacting with a local computer system. Developed under the guidance of Mr. Yashod Gayashan and under the direct supervision of C-Clarke International Institute of Digital Sciences, this project integrates lightweight local models for speech-to-text, LLM inference, and text-to-speech without relying on cloud integrations or third-party APIs.
To make it extensible, the system utilizes a tool-calling layer and a Model Context Protocol (MCP) bridge, giving the LLM secure access to read files, run search queries, and manage local workspaces.
Project Architecture
1. Input Pipeline
Mic stream → WebRTC VAD → Faster-Whisper
2. Agent Orchestrator
Ollama LLM ↔ Tool Registry ↔ MCP Bridge
3. Output Pipeline
Sentence Splitter → Piper ONNX TTS → Audio Player
Data Flow Cycle:
1. Voice input is sliced into 10/20/30ms frames and analyzed by VAD to identify silence boundaries.2. Faster-Whisper transcribes raw PCM audio bytes to text and forwards it to the Async Agent loop.
3. The agent queries Ollama and decides whether to invoke local tools or external MCP servers.
4. Generated stream chunks are parsed by the sentence splitter, feeding audio blocks to Piper TTS to play speech immediately.
How the Project Evolved
Phase 1: Streaming Text REPL
Development began with a basic text-based chat loop in a CLI console. HTTPX was integrated to stream responses token-by-token from a local Ollama server running the Qwen model.
Phase 2: Push-to-Talk Speech Input
The interface was expanded into a voice-capture pipeline using sounddevice
to capture raw mic inputs, webrtcvad
to detect silence boundaries, and Faster-Whisper to process the PCM data.
Phase 3: Real-Time Speech Synthesis
To eliminate response latency, a custom SentenceSplitter
was built. Instead of waiting for the full LLM answer, the incoming text stream is split at punctuation marks,
feeding completed sentences to a local Piper ONNX Text-to-Speech system immediately.
Phase 4: Tool Registry & Agent Dispatch
A custom tool dispatcher registry was implemented using Python decorators. The decorator uses python inspection capabilities to introspect docstrings and method signatures, generating JSON Schemas to advertise function specs back to Ollama.
Phase 5: The Model Context Protocol (MCP) Bridge
The codebase was migrated to an asynchronous loop using asyncio. A
stdio client bridge was then built using the Model Context Protocol (MCP) to spawn and connect external
tool servers dynamically (like local filesystem management tools).
Challenges Faced & Solutions
1. Clipping of Initial Spoken Words
During early voice testing, the VAD engine took around 200–300 ms to detect speech inputs, clipping the first word of commands. To resolve this, a sliding 300 ms ring buffer was implemented to continually record audio. When speech is detected, the assistant appends the pre-buffered audio to the front of the recording, resolving the clipping issue entirely.
2. Thread-Blocking Tasks in Async Event Loops
Integrating audio recording and Whisper transcription blocked the main Python asyncio
loop, causing stuttering audio and delayed speech output. This was solved by wrapping blocking library
calls in executors (run_in_executor),
offloading CPU-heavy audio models to separate threads while keeping the main async IO stream free.
Future Improvements
- Add a local wake-word engine (like Picovoice Porcupine) to move from push-to-talk to fully hands-free activation.
- Integrate multi-turn tool calling logic to allow the LLM to write a file and execute it in a single response cycle.
- Implement Docker containerization to simplify the deployment of the python modules, Whisper, and Piper dependencies.
Project Metadata
- Target Role AI Developer
- Status Source Available
- Target Hardware Consumer GPU / CPU
- Execution Model Asynchronous Event Loop