Designing how people talk to agents

When we started adding LLM capabilities to Composer, the obvious first question was: where does the chat go? A sidebar felt like a commitment. A modal felt like an interruption. A separate page felt like leaving. None of these sat right for an app built around side-by-side multitasking, where you’re already looking at the thing you want to talk to the agent about.

Over the course of about a year, the team and I worked through several iterations of what it means for an agent to be present in the app without taking over. Each step was small, but looking back they add up to a pretty different interaction model from where we started.

A stub and a keybinding

The first version was intentionally bare. I added a chat dialog to the AutomationPlugin, bound to Cmd+K, that floated above whatever you were working on. Just an input and a thread. The important decision wasn’t the UI — it was the keybinding. It established that the agent would always be one gesture away and that it would appear in context, not in a dedicated pane.

I made it resizable shortly after. A chat thread that can only show a few lines is frustrating when the response is long, but a full-height dialog is heavy for a quick question. Drag-to-resize let users expand the dialog as the conversation grew and collapse it when they were done.

Giving the agent context

The ambient dialog gave us a universal entry point, but it had no idea what you were looking at. When someone asked a question while editing a document, the agent couldn’t see the document. To fix this, I implemented companion chat — a chat thread bound to a specific document that automatically receives the document’s content as context.

This changed how the team used the assistant almost immediately. Questions went from generic (“how do I format a table in Markdown?”) to specific (“summarize the key points in this document”). The idea of giving the LLM not just a system prompt but the actual content the user is working with became a pattern we kept coming back to.

Rich responses

As the assistant became more capable, its responses needed to be more than plain text. If an agent mentions a document or a contact, the user should see it rendered the way they’d see it elsewhere in the app. I integrated the ChatThread with Composer’s card surface system so agent messages could embed transclusions of structured objects, rendered by the same plugins that handle them everywhere else.

Meanwhile, Rich Burdon refactored the underlying AiSession and AiChatProcessor to support tool use, streaming text, and session management. Dmytro Maretskyi built the blueprints and conductor system for composing multi-step agent flows. Josiah Witt wired up the observer pattern for chat state and ensured the assistant worked across web and mobile, falling back to Edge Cloud AI on platforms where the local Ollama sidecar wasn’t available.

Configuring the prompt

As we added more kinds of context — selected objects, space contents, agent-specific configurations — the prompt itself needed a richer interface. I designed a prompt options modal that lets users attach context objects to their message and configure which tools the agent should have access to.

Into the editor itself

The companion chat was useful but still lived alongside the document, not inside it. I worked on CodeMirror extensions that brought agent interactions directly into the editor, so that asking about a specific paragraph or requesting a rewrite happened right where the content lives.

Dmytro later built a more comprehensive AI-assisted editor with a Typewriter component that supports tone adjustments, custom instructions, and a proofreading linter that surfaces inline diagnostics with one-click fixes. All of it runs through the same AiService pipeline that powers the chat thread.

Keyboard access

Throughout all of this, I worked to make sure the assistant was fully keyboard-accessible. Invoking the dialog, navigating the conversation, attaching context, submitting a prompt — it all needed to work without a mouse. This mattered for accessibility, but also just for the kind of fast, keyboard-driven workflow that the rest of Composer supports.

What’s next

The system keeps growing. The team has built agent planning and task management, conversation branching, a conductor DSL for multi-step flows, code execution sandboxes, and initiatives — event-driven pipelines that let agents respond to changes in the user’s data. What started as a floating input has become a network of touchpoints woven into the app.

Each step along the way was small — a stub dialog, then resizing, then context, then structured responses, then inline editing, then keyboard access — but each one made the agent feel less like a chatbot you visit and more like a collaborator who’s already in the room.