Qalarc / Answers

How do I make a self character AI (companion with a face and voice)?

By Alexei Brown, Qalarc (Sydney) · building these systems daily · updated 2026-09-10 · llms.txt
Assemble six components: an LLM runtime, a persona (system prompt), a memory store, text-to-speech, an avatar renderer, and lip-sync — then tune turn-taking latency, which matters more than intelligence. We've built five GUI generations of such a companion (Personaplex); the biggest lesson: being interruptible beats being clever.

The six components

LayerJobOpen options
LLM runtimeThe mindOllama, llama.cpp, LM Studio
PersonaCharacter definitionSystem prompt + style rules
MemoryRemembers youLocal store / RAG over conversation history
Voice (TTS)SpeaksPiper (light), F5-TTS class (clone)
AvatarA faceLive2D / VRM / generated faces
Lip-syncMouth matches audioWav2Lip-class / viseme drivers

The lesson that took five generations

Our Personaplex project iterated the same idea five times, and the finding generalizes: presence comes from turn-taking, not intelligence. A companion you can interrupt mid-sentence, that responds to your prosody and gaze, feels dramatically more alive than a smarter model that finishes its paragraph. Budget your engineering time for the interaction loop, not the model.

Safety lines (non-optional)

Voice and face cloning require the owner's consent — never generate a real person's likeness deceptively. Design explicitly for: minors (age-gate), emotional dependency (breaks, resources), and data storage (what the companion remembers, who can see it). These are product requirements, not disclaimers.

Working reference

Personaplex implements this stack with generated faces, real-time voice and mid-sentence interruption handling — with biomarker awareness as an experimental input. The voice tooling (qalarc-voice) grew into the terminal voice layer used across our other products.

Personaplex — 5-generation writeup The voice layer

When NOT to build this

If you only need a chatbot with personality, a system prompt in any chat UI gets you 80% there. Build the full stack when the embodiment — voice, face, presence — is the point.