Voice Agent

A voice session orchestrates VAD, STT, LLM, and TTS as one pipeline. The SDK owns microphone capture, turn detection, and playback; the app consumes a stream of typed events.

Overview

Creating a session

One call names the three models and hands back a VoiceSession. The session downloads and loads what it was given, makes sure a VAD is resident, and wires the pipeline.

func createSession(
    stt: ModelRef,
    llm: ModelRef,
    tts: ModelRef,
    vad: VadOptions? = nil,
    turnHandling: TurnHandlingOptions? = nil,
    generation: LlmOptions? = nil,
    downloadIfNeeded: Bool = true
) async throws -> VoiceSession

let session = try await RunAnywhere.voice.createSession(
    stt: ModelRef(id: "sherpa-onnx-whisper-tiny.en"),
    llm: ModelRef(id: "lfm2-350m-q4_k_m"),
    tts: ModelRef(id: "vits-piper-en_US-lessac-medium")
)

ModelRef is ExpressibleByStringLiteral, so a bare string works:

let session = try await RunAnywhere.voice.createSession(
    stt: "sherpa-onnx-whisper-tiny.en",
    llm: "lfm2-350m-q4_k_m",
    tts: "vits-piper-en_US-lessac-medium"
)

The tts ref takes an optional voice, which selects a voice inside the TTS model rather than naming a model:

tts: ModelRef(id: "vits-piper-en_US-lessac-medium", voice: "en_US-lessac-medium")

Set downloadIfNeeded: false to fail rather than download when a model is missing.

VoiceSession

var events: AsyncThrowingStream<VoiceEvent, Error> { get }
func start() throws
func say(_ text: String) async throws
func interrupt()
func close() async

start() is the only thing that opens the microphone. Subscribing to events on its own is safe, so you can wire the UI before capture begins.

let events = session.events

let eventTask = Task {
    do {
        for try await event in events {
            handle(event)
        }
    } catch {
        print("Session failed: \(error.localizedDescription)")
    }
}

try session.start()

say(_:) speaks a line outside the turn loop, useful for a greeting. interrupt() stops the agent mid-utterance. close() stops capture, tears down the pipeline, and releases native resources.

VoiceEvent

enum VoiceEvent: Sendable {
    case userTranscribed(text: String, isFinal: Bool)
    case agentStateChanged(AgentState)
    case agentResponse(text: String)
    case speechStarted
    case speechEnded
    case error(message: String, recoverable: Bool)
}

AgentState is .listening, .thinking, or .speaking. .userTranscribed arrives repeatedly with isFinal: false while the user is mid-sentence, then once with isFinal: true. .agentResponse arrives per token, so append rather than replace. Recoverable component trouble comes through .error with recoverable: true and the session keeps running. An unrecoverable failure throws into the stream instead.

Turn handling

var turnHandling = TurnHandlingOptions()
turnHandling.endpointing = Endpointing(minDelayMs: 500, maxDelayMs: 3000)
turnHandling.interruption = Interruption(enabled: true, minDurationMs: 500)

let session = try await RunAnywhere.voice.createSession(
    stt: "sherpa-onnx-whisper-tiny.en",
    llm: "lfm2-350m-q4_k_m",
    tts: "vits-piper-en_US-lessac-medium",
    turnHandling: turnHandling
)

Only endpointing.minDelayMs reaches the native compose ABI today. maxDelayMs and the interruption settings are accepted, logged, and ignored until the ABI carries them.

Generation options

Pass LlmOptions to shape what the agent says.

var generation = LlmOptions()
generation.systemPrompt = "You are a concise voice assistant. Answer in one or two sentences."
generation.maxOutputTokens = 120
generation.temperature = 0.6

let session = try await RunAnywhere.voice.createSession(
    stt: "sherpa-onnx-whisper-tiny.en",
    llm: "lfm2-350m-q4_k_m",
    tts: "vits-piper-en_US-lessac-medium",
    generation: generation
)

Keeping maxOutputTokens low matters here in a way it does not for chat: every token becomes speech the user waits through.

Complete voice assistant

import SwiftUI
import RunAnywhere

@MainActor
@Observable
final class VoiceAssistant {
    enum SessionState: Equatable {
        case disconnected, connecting, listening, processing, speaking
        case error(String)
    }

    private(set) var state: SessionState = .disconnected
    private(set) var transcript = ""
    private(set) var assistantResponse = ""

    private var session: VoiceSession?
    private var eventTask: Task<Void, Never>?

    func start() async {
        guard case .disconnected = state else { return }

        state = .connecting
        transcript = ""
        assistantResponse = ""

        do {
            let session = try await RunAnywhere.voice.createSession(
                stt: "sherpa-onnx-whisper-tiny.en",
                llm: "lfm2-350m-q4_k_m",
                tts: "vits-piper-en_US-lessac-medium"
            )
            self.session = session

            let events = session.events
            eventTask = Task { [weak self] in
                do {
                    for try await event in events {
                        await MainActor.run { self?.handle(event) }
                    }
                } catch {
                    await MainActor.run { self?.state = .error(error.localizedDescription) }
                }
            }

            try session.start()
            state = .listening
        } catch {
            state = .error(error.localizedDescription)
        }
    }

    func stop() async {
        eventTask?.cancel()
        eventTask = nil
        state = .disconnected

        let closing = session
        session = nil
        await closing?.close()
    }

    func interrupt() {
        session?.interrupt()
    }

    private func handle(_ event: VoiceEvent) {
        switch event {
        case .agentStateChanged(let agentState):
            switch agentState {
            case .listening: state = .listening
            case .thinking:  state = .processing
            case .speaking:  state = .speaking
            }

        case .userTranscribed(let text, let isFinal):
            transcript = text
            if isFinal { assistantResponse = "" }

        case .agentResponse(let text):
            assistantResponse += text

        case .speechStarted:
            state = .listening

        case .speechEnded:
            state = .processing

        case .error(let message, let recoverable):
            if !recoverable { state = .error(message) }
        }
    }
}

struct VoiceAssistantView: View {
    @State private var assistant = VoiceAssistant()

    var body: some View {
        VStack(spacing: 20) {
            if !assistant.transcript.isEmpty {
                Text(assistant.transcript)
                    .frame(maxWidth: .infinity, alignment: .trailing)
            }

            Text(assistant.assistantResponse)
                .frame(maxWidth: .infinity, alignment: .leading)

            Spacer()

            Button {
                Task {
                    if case .disconnected = assistant.state {
                        await assistant.start()
                    } else {
                        await assistant.stop()
                    }
                }
            } label: {
                ZStack {
                    Circle().fill(buttonColor).frame(width: 80, height: 80)
                    Image(systemName: isActive ? "stop.fill" : "mic.fill")
                        .font(.title)
                        .foregroundStyle(.white)
                }
            }

            Text(statusText)
                .font(.caption)
                .foregroundStyle(.secondary)
        }
        .padding()
        .onDisappear {
            Task { await assistant.stop() }
        }
    }

    private var isActive: Bool {
        if case .disconnected = assistant.state { return false }
        return true
    }

    private var buttonColor: Color {
        switch assistant.state {
        case .disconnected: return .blue
        case .connecting:   return .gray
        case .listening:    return .red
        case .processing:   return .orange
        case .speaking:     return .green
        case .error:        return .gray
        }
    }

    private var statusText: String {
        switch assistant.state {
        case .disconnected:      return "Tap to start"
        case .connecting:        return "Connecting"
        case .listening:         return "Listening"
        case .processing:        return "Processing"
        case .speaking:          return "Speaking"
        case .error(let reason): return reason
        }
    }
}

Session lifecycle

Hold one session at a time. start() is idempotent, so a second call on a running session does nothing, but two sessions mean two microphone drivers. Cancel the event task before closing:

eventTask?.cancel()
eventTask = nil
await session?.close()

close() never throws and is safe to call more than once.

Error handling

do {
    let session = try await RunAnywhere.voice.createSession(
        stt: "sherpa-onnx-whisper-tiny.en",
        llm: "lfm2-350m-q4_k_m",
        tts: "vits-piper-en_US-lessac-medium"
    )
    try session.start()
} catch let error as SDKException {
    switch error.code {
    case .notInitialized:
        print("Call RunAnywhere.initialize first")
    case .modelNotFound:
        print("One of the models is not registered: \(error.message)")
    case .modelLoadFailed:
        print("A model could not load: \(error.message)")
    case .invalidState:
        print("The session is closed")
    default:
        print("Error: \(error.localizedDescription)")
    }
}

Session creation does the downloading and loading, so most failures surface here rather than during the conversation.

Best practices

Create the session ahead of the tap

createSession downloads and loads three models. Doing that when the user taps the microphone is a visible stall. Create it during onboarding or on view appear.

Handle audio interruptions

Calls and other apps interrupt the audio session. Close the session and create a new one when the interruption ends.

Close on disappear

Tear the session down in onDisappear. A live microphone driver outlives the view otherwise.

VAD

Voice activity detection

STT

Speech-to-text

Getting Started

Swift SDK

Kotlin SDK

React Native SDK

Flutter SDK

Web SDK

Vibe Coding

Overview

Creating a session

VoiceSession

VoiceEvent

Turn handling

Generation options

Complete voice assistant

Session lifecycle

Error handling

Best practices

VAD

STT

​Overview

​Creating a session

​VoiceSession

​VoiceEvent

​Turn handling

​Generation options

​Complete voice assistant

​Session lifecycle

​Error handling

​Best practices

VAD

STT

Overview

Creating a session

VoiceSession

VoiceEvent

Turn handling

Generation options

Complete voice assistant

Session lifecycle

Error handling

Best practices