My Product2026AI / AutomationFeatured

J.A.R.V.I.S. - Personal AI Assistant System

A privacy-first personal AI assistant with 9 LLM providers (Claude, GPT-4, Gemini, Groq, Ollama, OpenRouter, Cursor, GitHub Copilot, Claude Code CLI), 80+ tools, Palace Memory with Knowledge Graph, ambient voice monitoring with GPU transcription, a self-improving learning loop, and 4 fully-featured client apps: Flutter Android app, Next.js web dashboard, system tray daemon, and CLI.

80+ integrated tools

tools

9 LLM providers

llms

CLI, Web, Android, Daemon

interfaces

Technical Implementation

J.A.R.V.I.S. (Just A Rather Very Intelligent System) is a fully async Python 3.12 / FastAPI backend with four client frontends. The Python core runs an event-yielding AgentLoop that emits ThinkingEvent, ToolCallEvent, ToolResultEvent, and ResponseEvent, streaming every agent action to all connected UIs in real time. A CommandRouter classifies each request into domains (fast-path for single-domain > 0.8 confidence, plan-path for multi-domain), a ModelRouter dispatches to 9 LLM providers by prefix, and a ToolRegistry semantically selects 5–8 relevant tools per turn (saving ~6,500 tokens per request). The FastAPI backend exposes 25+ route modules (chat, jobs, tasks, workflows, health, voice, meetings, code workspace, ambient, SLI, location, Matrix, WebSocket terminals, Prometheus metrics, MCP servers, and more). The Flutter Android app (Dart, Riverpod) connects over Tailscale or LAN via REST (Dio + JWT AuthInterceptor with single-flight refresh) and 3 WebSocket channels (chat, events, terminal); it has 10 screens — Chat, Terminal, Tasks, Jobs, Workflows, Activity, Health, Locations, Settings — plus Porcupine wake-word voice (STT, Whisper upload, TTS), biometric lock (local_auth), offline DB (Drift/SQLite), flutter_map location, fl_chart analytics, and xterm.js terminal emulation. The Next.js 14 web dashboard (Tailwind CSS, Radix UI, Zustand, TanStack Query) runs on port 3001 and proxies to FastAPI on 8501; it includes a Monaco-powered code workspace (file explorer, Git panel, search, editor tabs), a live meetings view (audio level meter, transcript, summaries), and a Settings shell with 16 panels (Profile, Model, Health, Integrations for Jira/ClickUp/Google/Azure DevOps, Social Media, Ambient Voice, Personal Intelligence, System Log Intelligence, and more). The Palace Memory organises long-term memory into Wings, Rooms, and Drawers (PostgreSQL + SQLite fallback), with a Knowledge Graph for temporal subject→predicate→object triples and a PalaceConsolidator that auto-extracts high-signal facts from every conversation with zero LLM cost. An ambient voice listener (sounddevice → 3-stage VAD → 3-tier diarization → faster-whisper → 8-layer junk filter) runs always-on in the background and sends session transcripts to an LLM to extract action items, decisions, reminders, and summaries. A self-improving loop (ReflectionAgent, SkillExtractor, SkillStore, FailureMemory, LearningWorker) runs every 6 hours, scoring sessions and consolidating reusable skill patterns.

Key Features

Multi-LLM routing: Claude API, Claude Code CLI, GPT-4o (OpenAI), Gemini 2.0, Groq, Ollama (local), OpenRouter, GitHub Copilot CLI, Cursor — prefix dispatch + auto-model fallback chain (Groq → OpenRouter → paid)

Event-yielding AgentLoop: Think-Act-Observe cycle streams ThinkingEvent, ToolCallEvent, ToolResultEvent, ResponseEvent to all UIs in real time via WebSocket

CommandRouter: intent + domain classifier, fast-path (>0.8 confidence, direct route) vs plan-path (multi-domain → Planner DAG decomposition)

ToolRegistry: semantic keyword matching loads only 5–8 relevant tools per turn — saves ~6,500 tokens per request

Multi-agent orchestrator: Planner decomposes requests into DAG waves; Specialist Agents (Coding, Debugging, Research, Automation, Communication, Knowledge, System) run in parallel with asyncio.gather()

BudgetController: per-turn caps on tokens, tool steps, wall-clock time, and cost; ContextBuilder loads 10 context layers in parallel

80+ tool modules: shell, filesystem, git/git-workflow, Docker, web, browser/scraper, clipboard, screen capture, code, document, data export

Communication tools: email (SMTP/IMAP), Gmail watcher, Outlook, Slack, Microsoft Teams, WhatsApp (WAHA), Matrix/Instagram/Facebook (mautrix-meta), ntfy push notifications

Business tools: Jira (webhooks + issue management), ClickUp, MiHCM HR (auto clock-in/out), Hubstaff (activity tracking), LinkedIn, Upwork, calendar, Google Calendar, reminder, schedule

DevOps tools: deployment, IIS management, Docker, n8n workflow triggers, OpenClaw, code intelligence, PR review rubric, dev utilities

Intelligence tools: research, reasoning, sentiment, knowledge graph, summary, project signals, project intelligence, unified search, tool chains, composition engine

Task tools: task CRUD, task intelligence (v2: dependency graph, blocking detection, Kanban, smart suggestions), task sync, task forecast, income workspace

Memory/learning tools: memory (Palace CRUD + KG queries), self-improve, goal management, recent plans, orchestrator

System tools: Windows Control (17 categories: services, Wi-Fi, Bluetooth, registry, env vars, winget packages, power plans, startup, printers, Windows Update, storage, display, sound, firewall), system apps, system intelligence, SLI

Live meeting tools: meeting (capture + summarise), meeting-live (real-time audio + transcript), briefing + briefing resilience

Three distinct task systems: Project Tasks (personal Kanban, jarvis_tasks), Background Jobs (on-demand async AgentLoop execution, jarvis_jobs), Autonomous Tasks (cron-scheduled recurring automation, jarvis_autonomous_tasks)

Task Intelligence v2: dependency validation, blocking detection with auto-unblock notifications, smart auto-prioritisation, visual dependency graph, Kanban board, AI suggestions; 100+ tests

Palace Memory: hierarchical Wings → Rooms → Drawers in PostgreSQL/SQLite; PalaceKnowledgeGraph stores temporal subject→predicate→object triples; PalaceConsolidator auto-extracts memories from every turn with zero LLM cost

5-layer memory stack: PalaceMemory (hierarchical long-term) + PostgresMemory (tsvector full-text) + RAG Store (pgvector embeddings via nomic-embed-text / Ollama) + Redis hot cache + Local SQLite offline fallback

Self-improving loop: ReflectionAgent scores sessions post-task; SkillExtractor auto-creates reusable skills; SkillStore (Postgres + pgvector) + SkillMatcher 3-stage retrieval; LearningWorker prunes and consolidates every 6 hours

Personal Intelligence: style_model + preferences + personal_memory learn your communication patterns, personal facts, and preferences — injected into every agent prompt

Context compression: Summarizer + RollingSummary + FactExtractor keep long conversations within token budget without losing important context

Ambient Voice Listener: always-on mic (sounddevice) → 3-stage VAD (Silero neural + WebRTC + RMS fallback) → 3-tier diarization (pyannote/speaker-diarization-3.1 → resemblyzer → single-turn fallback)

faster-whisper large-v3-turbo (int8, GPU) transcription with 8-layer junk filter (no_speech_prob, avg_logprob, non-Latin script, hallucination phrases, min words, repetition, gibberish, domain keywords)

Session analysis: LLM extracts action items (→ communications.db, morning brief), decisions, reminders (APScheduler → Windows toast + ntfy), summaries, topics; Android app streams ambient audio via ingest API

System Log Intelligence (SLI): reads Windows Event Log, PowerShell history, VS Code activity, Git activity, performance counters, WER crash reports — feeds patterns to agent context

Monitors: disk usage, Docker container health, Git repo changes, process status, URL reachability — all feed into health and proactive alerts

Goal System: GoalStore + GoalTracker + GoalNudge — track personal and financial goals with milestone linking and proactive reminders

Flutter Android app (Dart 3, Riverpod 2, go_router): 10 screens — Chat, Terminal, Tasks, Jobs, Workflows, Activity (MiHCM + Hubstaff), Health, Locations, Settings, Connection (PIN pairing)

Flutter auth: PIN pairing (POST /api/v1/auth/pair), JWT tokens with single-flight refresh via AuthInterceptor, flutter_secure_storage; Dio REST client + 3 WebSocket channels (chat, events, terminal)

Flutter voice: Porcupine wake-word detection, device STT (speech_to_text), Whisper backend STT (record audio → POST /api/v1/voice/transcribe), flutter_tts playback; VoiceForegroundService keeps voice active in background

Flutter features: fl_chart analytics, xterm terminal emulation, flutter_markdown rendering, Drift (SQLite) offline DB, flutter_map (OpenStreetMap) + geolocator location tracking, NSD service discovery, biometric lock (local_auth fingerprint/face), flutter_local_notifications

Next.js 14 web dashboard (port 3001) proxies to FastAPI (port 8501): pages — Chat (streaming agent events), Terminal (xterm.js), Code Workspace, Meetings, Projects, Settings, Login

Code Workspace: Monaco editor, file explorer (FileExplorer.tsx), Git panel, search panel, resizable editor tabs — full browser-based IDE connected to server filesystem

Meetings view: real-time AudioLevelMeter, AudioPlayer, MeetingControls, live TranscriptView, MeetingSummary; wired to FastAPI /api/meetings WebSocket

Settings dashboard (16 panels): Profile, Config, Model (LLM selection), Health, Activity, Projects, Meetings, Tasks (with Kanban + dependency graph), Workflows, Terminal, Reflection (learning loop), Integrations (Jira, ClickUp, Google, Azure DevOps), Social Media, Ambient Voice, Personal Intelligence, System Log Intelligence, Backup, Passes

Next.js tech: Tailwind CSS, Radix UI primitives, Monaco Editor (@monaco-editor/react), xterm.js, Zustand state, TanStack Query v5, react-markdown + remark-gfm, react-resizable-panels, react-dropzone

SafetyGate: ALLOW / WARN / CONFIRM / BLOCK policies — blocks rm -rf /, DROP TABLE without WHERE, force-push to main, and other destructive patterns

Privacy modes: STRICT (Ollama only, no cloud), GUARDED (cloud allowed, secrets auto-redacted), RELAXED — secrets never sent to cloud in STRICT/GUARDED

4-layer security: Cloudflare Access JWT → rate limiter → web login session → JWT Bearer tokens; Tailscale-aware CORS for remote Android access

Observability: structlog JSON logs, TraceManager (per-request cost + token accounting), Prometheus /metrics endpoint, Grafana dashboards, OpenTelemetry traces

Docker Compose: Postgres + pgvector, Redis, Prometheus, Grafana, n8n, WAHA (WhatsApp), mautrix-meta (Matrix/Instagram/Facebook), Synapse (Matrix homeserver)

Work-day automation: auto clock-in via MiHCM, Hubstaff activity simulation, morning WhatsApp brief, Jira/ClickUp sync, proactive EOD digest, standup messages, meeting alerts

Architecture & Patterns

Python 3.12 FastAPI backend — fully async, zero blocking calls in the event loop; 25+ route modules covering every feature

ModelRouter: own Message/ToolCall/ToolSpec types decouple tools from all 9 vendor SDKs; prefix routing + auto-model fallback chain

AgentLoop: Think-Act-Observe cycle, yields typed events, TurnContext tracks tool call budget; ContextBuilder loads 10 context layers in parallel

ToolRegistry: one Tool subclass per file, semantic keyword matching, loads only 5–8 tools per turn vs exposing 80+ at once

Multi-agent orchestrator: Planner (LLM + rule-based fallback) decomposes query to DAG; Specialist Agents run in topological waves with parallel asyncio.gather()

Palace Memory: PalaceStore (Postgres + SQLite fallback) + PalaceKnowledgeGraph (temporal triples); PalaceConsolidator extracts memories from turns with zero LLM cost

Ambient voice pipeline: sounddevice (1% CPU) → 3-stage VAD → 3-tier diarization → faster-whisper GPU → 8-layer junk filter → session LLM analysis → APScheduler reminders

Autonomous task system: Postgres priority queue → WorkerPool → GeneralWorker / CoderWorker per job role; Kanban board with dependency graph

Memory stack (5 layers): PalaceMemory → PostgresMemory (tsvector) → RAG Store (pgvector) → Redis hot cache → Local SQLite offline

Self-improving loop: ExperienceStore + SkillStore + FailureMemory in Postgres; SkillMatcher 3-stage retrieval; LearningWorker consolidates every 6 hours

Flutter app (Dart 3 + Riverpod 2): Presentation → Services (WS, Auth, Voice, Location, Discovery) → Data (Dio repos, Drift SQLite) → Core; 3 WS channels (chat, events, terminal)

Next.js 14 dashboard: proxy architecture (3001 → 8501), Monaco code workspace, live meetings view, 16-panel Settings shell, xterm.js terminal, Zustand + TanStack Query

Observability: structlog JSON logs, TraceManager per-request cost/token accounting, Prometheus /metrics, Grafana dashboards, 4-layer security stack

Project Highlights

Four fully-featured client apps: Flutter Android, Next.js web dashboard (Monaco IDE + live meetings), system tray daemon, and CLI9 LLM providers — tools never touch vendor-specific types; transparent switching and auto-fallback chainPalace Memory with Knowledge Graph: hierarchical long-term memory that auto-extracts facts from every conversation with zero LLM costAlways-on ambient voice: GPU-accelerated faster-whisper, 3-tier speaker diarization, 8-layer junk filter, auto action-item extractionSelf-improving AI: ReflectionAgent + SkillExtractor + LearningWorker learn from every session — no manual tuning neededPrivacy-first: runs 100% locally with Ollama; STRICT mode never sends any data to cloud; GUARDED mode auto-redacts all secretsReal-time event bus: every agent thought, tool call, and result streams live to all UIs simultaneouslyFull Windows OS control: 17 operation categories via natural language (Wi-Fi, Bluetooth, registry, winget, power plans, and more)Complete work-day automation: morning brief, MiHCM clock-in, Hubstaff tracking, Jira/ClickUp sync, WhatsApp digests, EOD summary

Technology Stack

Python 3.12FastAPIasyncioPydanticstructlogFlutterDartRiverpodDiogo_routerDriftxtermNext.js 14React 18TypeScriptTailwind CSSRadix UIMonaco EditorZustandTanStack QueryPostgreSQLpgvectorSQLiteRedisClaude APIClaude Code CLIOpenAI APIGemini APIGroqOllamaOpenRouterGitHub Copilot CLICursorfaster-whisperpyannotePorcupinesounddeviceDockern8nPrometheusGrafanaAPSchedulerTailscalemautrix-meta

Interested in This Project?

Let's discuss how I can help bring similar solutions to your business.

Get in Touch View All Projects