When initially experimenting with LLMs and agentic AI, software engineers at Notion AI applied advanced code generation, complex schemas, and heavy instructioning.
Quickly, though, trial and error taught the team that it could get rid of all of that complicated data modeling. Notion’s AI engineering lead Ryan Nystrom and his team pivoted to simple prompts, human-readable representations, minimal abstraction, and familiar markdown formats. The result was dramatically improved model performance.
Applying this re-wired approach, the AI-native company released V3 of its productivity software in September. Its notable feature: Cutomizable AI agents — which have quickly become Notion’s most successful AI tool to date. Based on usage patterns compared to previous versions, Nystrom calls it a “step function improvement.”
“It’s that feeling of when the product is being pulled out of you rather than you trying to push,” Nystrom explains in a VB Beyond the Pilot podcast. “We knew from that moment, really early on, that we had something. Now it’s, ‘How could I ever use Notion without this feature?’”
‘Rewiring’ for the AI agent era
As a traditional software engineer, Nystrom was used to “extremely deterministic” experiences. But a light bulb moment came when a colleague advised him to simply describe his AI prompt as he would to a human, rather than codify rules of how agents should behave in various scenarios. The rationale: LLMs are designed to understand, “see” and reason about content the same way humans can.
“Now, whenever I’m working with AI, I will reread the prompts and tool descriptions and [ask myself] is this something I could give to a person with no context and they could understand what’s going on?” Nystrom said on the podcast. “If not, it’s going to do a bad job.”
Stepping back from “pretty complicated rendering” of data within Notion (such as JSON or XML) Nystrom and his team represented Notion pages as markdown, the popular device-agnostic markup language that defines structure and meaning using plain text without the need for HTML tags or formal editors. This allows the model to interact with, read, search and make changes to text files.
Ultimately, this required Notion to rewire its systems, with Nystrom’s team focusing largely on the middleware transition layer.
They also identified early on the importance of exercising restraint when it comes to context. It’s tempting to load as much information into a model as possible, but that can slow things down and confuse the model. For Notion, Nystrom described a 100,000 to 150,000 token limit as the “sweet spot.”
“There are cases where you can load tons and tons of content into your context window and the model will struggle,” he said. “The more you put into the context window, you do see a degradation in performance, latency, and also accuracy.”
A spartan approach is also important in the case of tooling; this can help teams avoid the “slippery slope” of endless features, Nystrom advised. Notion focuses on a “curated menu” of tools rather than a voluminous Cheesecake Factory-like menu that creates a paradox of choice for users.
“When people ask for new features, we could just add a tool to the model or the agent,” he said. But, “the more tools we add, the more decisions the model has to make.”
The bottom line: Channel the model. Use APIs the way they were meant to be used. Don’t try to be fancy, don’t try to overcomplicate it. Use plain English.
Listen to the full podcast to hear about:
-
Why AI is still in the pre-Blackberry, pre-iPhone era;
-
The importance of “dogfooding” in product development;
-
Why you shouldn’t worry about how cost effective your AI feature is in the early stages — that can be optimized later;
-
How engineering teams can keep tools minimal in the age of MCP;
-
Notion’s evolution from wikis to full-blown AI assistants.
Subscribe to Beyond the Pilot on Apple Podcasts, Spotify, and YouTube.
Credit: Source link
