How Handsfree works

Choose the computer and conversation on screen, then talk to the coding agent working in that conversation's project folder. You can also type. Project switching is a screen action in the current app; voice supplies requests to the selected session.

This guide describes the implementation in this iPhone checkout, reviewed September 7, 2026. It does not establish what is currently running on your Mac or your coworker's phone. For an illustrated setup tour, start with Connect your computer. For a practical work tour, continue to Your first conversation. For installation commands, use Mac + iPhone setup.

What runs where?#

Handsfree gives you a phone interface to an agent running on your own computer. The computer reads files, edits code and runs commands. The installed iPhone app recognizes your speech on device, shows the conversation and speaks replies using its own voice. Only text goes between the phone and host.

flowchart TB
    Phone["Your iPhone<br/>Choose a session, speak or type<br/>See progress and hear replies"]
    subgraph Mac[Your computer - awake and online]
        Host[Handsfree host]
        Agent[Claude Code agent]
        Files[Your selected project folder]
    end
    Claude[Claude service]
    Phone <-->|Text over private connection| Host
    Host <--> Agent
    Agent <-->|Read, edit, run commands| Files
    Agent <--> Claude

The host is the computer-side service that accepts paired phone connections, remembers sessions and passes work to the agent. A separate background agent process lets accepted work continue when the phone disconnects or the connection service restarts.

The local setup recipe starts the agent host without desktop speech dependencies. The iPhone uses Apple’s on-device English recognition and native voice. It uses Tailscale to give your phone a private HTTPS route to the Mac. Claude still uses its online model service; handling speech on the phone does not make the whole system offline.

The iPhone app bundles its own interface. The optional hosted browser companion supplies an interface too, but your computer still supplies the host and project files. You do not need to run the website development server to use the installed iPhone app. This implementation does not require Convex or a cloud database.

Three choices: host, project, session#

For example, a host called “My Mac” could have these sessions:

My Mac
├── /Users/you/Projects/website
│   ├── Session: understand the homepage
│   └── Session: investigate the login bug
└── /Users/you/Projects/mobile-app
    └── Session: plan the next release

That is a conceptual map. The phone uses one choice per screen: New session → computer → project → agent → Start talking. Steps with one available option are skipped. Continue a session shows recent conversations on the connected computer; with multiple computers, choose another computer to see its sessions. There is no sidebar to navigate.

The host offers named projects configured with HF_PROJECTS. Without that list it offers its default folder and available folders from saved sessions. It also has a default folder (WORKDIR) and a configured set of allowed folders (HF_ALLOWED_ROOTS). You can create sessions in different allowed folders without restarting it. Adding a folder outside those allowed roots requires changing the host configuration. The folder must already exist on the host; entering a path does not clone a repository.

Each session remembers its original folder. To work in another project, select that project's existing session or choose that project for a new session. Use another project folder provides an advanced path-entry fallback. Changing the host's default does not move old sessions.

Two sessions in the same folder have separate conversations but share the same files and Git checkout. Creating a session does not create a branch, worktree or isolated copy. Agent edits appear in the files you open later on the Mac.

Which desktop conversations will I see?#

Recent sessions comes from this Handsfree host's saved sessions. Pairing does not automatically discover every Claude terminal conversation or every task in the Codex desktop app.

Existing sessions from the original Handsfree host can be imported by an operator using the migration procedure in the host guide. This is an explicit import from the old session registry, not a general desktop-thread picker. Live desktop takeover is disabled.

The phone offers only ready, configured agents and skips the agent screen when there is just one. The production host currently supplies Claude Code. Codex transport experiments exist in this checkout, but production Codex dispatch is still pending, so Codex does not appear as an available agent. See the provider checkpoint for that work.

What is by hand, and what is voice?#

There is no built-in voice navigation command such as “switch to my website project.” Those words are sent to the current agent as ordinary conversation. Use the session controls to change where subsequent requests go.

Voice messages are transcribed and submitted automatically; there is no draft-transcript approval step. In hands-free mode, a pause ends an utterance and starts processing it. Expect a sequence of spoken turns, with transcription and agent processing time between them. The agent is instructed to narrate brief progress updates and give conversational summaries, though exact replies vary.

How much can it do without me touching the phone?#

Once the correct session is selected and listening is enabled, you can ask questions, hear results and give follow-up instructions by voice. For example: “Explain the login flow,” then “Which part would you change?”, then “Make that change and run the relevant tests.”

The real host is configured for unattended Claude tools. Requests can change files and execute commands under the host's OS account without a phone approval dialog for each operation. Allowed project roots limit session selection; they are not a filesystem sandbox. This is why the setup recipe starts in a disposable folder.

Turning off listening or disconnecting the phone does not cancel accepted work. Switching sessions also leaves the old session's work on the host, and turns off listening locally. Use Stop agent in the relevant session when you intend to interrupt work. It also cancels that session's queued requests; it does not undo edits already made.

How does this relate to the Android original?#

This project is an iOS port of the original Handsfree system. It retains an adapted Claude agent daemon and adds its own authenticated host connection, shared interface and native iPhone audio engine. The upstream notes identify the original source and retained behavior.

The broad idea is shared: a phone conversation drives an agent working on a computer. Your coworker's exact Android screens, project picker, setup and everyday reliability cannot be established from this checkout. Unmodified older Android clients cannot connect to this host's authenticated protocol.

Keep the iPhone app open while talking; it keeps the screen awake. Leaving it pauses listening and discards unfinished speech. Recognition quality still needs a check on your actual phone. The browser companion supports foreground voice and needs fresh pairing after a page reload.

Where to go next#

• Your first conversation: setup landmarks, pairing and a sample phone session.
• Mac + iPhone setup: exact installation, private connection and startup instructions.
• Host guide: operator configuration, imported sessions and recovery.

Implementation references for maintainers: screen controls, native session and voice actions, browser client, host session handling, and agent behavior.

For the implementation behind this picture, read Phone → host → phone: speech models, TTS timing, prompt instructions, summaries and recovery.