How AI Chatbots are built (behind the scenes look)
Understand what makes up 90%+ of AI chat apps
📣 Let’s rethink the way we build (Sponsor)
Software development has come a long way in the past decade—so why haven’t our tools?
Let go of the clutter, distractions, and stress with Atono, so your team can focus on what matters: building better software together.
A clean, focused experience: Minimalist UI with in-context tools ensuring everything is where you need it, when you need it
User stories at the center: Stories are first-class objects in Atono that highlight customer value, set clear goals, and keep your team aligned
Do more, juggle less: Bring roadmapping, story writing, feature flagging, bug reporting, and usage tracking together in one place
Use code YEARONUSM25 and get a year free for 25 users!
Thanks, Atono, for sponsoring this week’s article 🙏
Today, we’re covering AI chatbots and how they’re built.
Recently, Sidwyn and I built our version of an AI chatbot in WriteEdge—it lets you get feedback on your technical design doc via a chat sidebar.
Here’s what it looks like:
We wanted to share the core AI development lessons we learned along the way with you, particularly because it serves as the foundation for so many AI apps today.
Note: This article isn’t intended to be a step-by-step tutorial for building an AI chatbot. Instead, it will teach you about how most AI chatbots are built so you know how they work behind the scenes. It should still be much easier to build your own after this.
1) Existing tools and libraries
We, along with most AI chatbots and GPT “wrappers,” use the Vercel AI SDK. This SDK provides a set of APIs we can use in React and on the Node backend to easily build AI features.
On the client, we manage chat state using the useChat hook from “ai-sdk/react.”
On the backend, we stream chat messages using Vercel’s streaming APIs.
And we’d be lying if we said we didn’t use the Vercel AI Chatbot template as a foundation. If you’re starting from scratch, this is without a doubt the best way to get the ground running immediately.
However, we had an existing codebase, so it required more work to figure out how chatbots are built to make it work for us.
In the next few sections, we’ll share what we learned.
2) Data entities
If we exclude upvoting and downvoting messages, there are three core entities: Chat, Message, and User.
Let’s write out the schema in Typescript, but assume this is in your DB tables.
interface Chat {
id: string;
title: string;
userId: string;
createdAt: string;
}
interface Message {
id: string;
chatId: string;
content: Json;
role: string;
createdAt: string;
}
interface User {
id: string;
... other fields
}Note that you might make tiny tweaks depending on your application. For example, we added a `tddId` to the `Chat` table so we could link a chat to a specific design doc.
So far, everything is straightforward except the `content` on the `Message` table. Why is it a JSON object instead of a string? In short, for flexibility.
The most common case where JSON format is helpful is for storing message text and tool call results. For brevity, we’ll cut out extensive detail of tool calls in this article. However, to show you why storing in JSON is valuable, consider the following messages we want to store:
// assistant text response
{
type: "text",
text: "Overall, the design document is well-structured and covers..."
}
// assistant tool response
{
type: "tool-result",
toolCallId: '12345',
toolName: 'getWeather',
result: "{ degreesFahrenheight: 75, degreesCelsius: 24 }"
}Since we use JSON, both of these objects can be stored in `Message.content`. On the client, we’d handle both of these data structures in how we render the UI.
For example:
// React code on the client
messages.map(message => {
if (message.type === "text") {
return <TextMessage text={message.text} role={message.role} />
} else if (message.type === "tool-result") {
if (message.toolName === 'getWeather') {
return <WeatherDisplay result={message.result} />
}
}
})Note: This is moreso pseudo-code to communicate the idea. The newly recommended way to stream text and tool calls is to use a “parts” field, but it can get lengthy to discuss and is an implementation detail. For a fully detailed example, check these docs.
The key takeaway here is that you need three models: Chat, Message, and User. In Message, store the content as JSON for flexibility of message types.
3) Handling messages and streaming responses
When we receive a new message from the frontend client, there are two things that need to happen:
Storage: Store the message so we don’t lose it.
Response: Generate a response using the context of the conversation, tools, and prompt. We also need to stream the response so the client can see immediate feedback.
Both are pretty straightforward. Here’s the pseudo-code!
Storing the messages and chat data
For storage, have the following at your /chat route:
if (!chatExists) {
createChat(); // Make call to db
}
saveMessageToChat(); // Make call to db
// stream response from AI, save response to db - see next sectionGenerating a response and streaming it
For streaming, we use Vercel’s streaming APIs to respond to the client. The code looks like this:
import { createDataStreamResponse, streamText } from 'ai';
// (1) immediately start streaming
return createDataStreamResponse({
// (2) Gives us a dataStream callback to write to the stream
execute: (dataStream) => {
// (3) Call OpenAI for a response
const result = streamText({
model,
system: systemPrompt, // customize this to your liking
messages,
onFinish: async ({ response, reasoning }) => {
// (4) Saves our final response to the DB
await saveMessages({
messages: response.messages,
chatId,
});
},
});
// (5) Ensures that if client disconnects, the stream is still consumed and we save the response message
result.consumeStream();
// (6) Streams the result of the GPT call into the response stream
result.mergeIntoDataStream(dataStream, {
sendReasoning: true,
});
},
});The step-by-step process here is:
Kick off streaming via `createDataStreamResponse`
Get a callback to write to the stream via `execute`
Make an API call to OpenAI (or any provider) via `streamText`
Ensure the final message is saved via `onFinish`
Call `consumeStream` so that if the client disconnects, we will self-consume the stream. In this case, if the user comes back after disconnecting, the message will be there
Write to the `dataStream` from (2) with the result of `streamText` from (3).
Doing these steps results in the message coming in the chunks you see when you use ChatGPT like this:
4) Displaying the messages on the client
So, if the messages come in chunks like the above screenshot, how do we display them in real time with nice formatting?
The react-markdown library handles it! You can send your text content to it, and it will convert it to the appropriate HTML elements that format it like Markdown.
Let’s take a look at this “Performance metrics” text:
When we receive text on the client, it comes in 2 chunks:
[chunk 1]
“**Performance ”
[chunk 2]
“Metrics**: ”
As we render the chunks coming in, we first actually display “**Performance ” with the ** at the beginning. Once the second chunk of “Metrics **” comes in, the react-markdown component renderer reads the full thing, “**Performance Metrics**” and switches it to an HTML element with the content inside.
The next question you might ask is, where does the “font-semibold” class come from, and how can I customize that?
It’s all part of how you customize the react-markdown library behavior. You define mappers for each markdown element and tell it exactly how to convert it to HTML.
Here’s our mapper for the above example:
const components = {
...
strong: ({ node, children, ...props }) => {
return (
<span className='font-semibold' {...props}>
{children}
</span>
);
},
... other mappers like h1, h2, h3, etc.
}React Markdown views text wrapped in “**” as “strong”, then calls the function provided to get the transformed HTML element.
Then you pass these mappers into the `ReactMarkdown` component:
<ReactMarkdown components={components}>
{yourStreamedText}
</ReactMarkdown>
5) Managing client chat state
Okay, we know how to display a message, but how can we keep track of all the messages and actually request the response from the backend?
The useChat hook does that for us. You tell it which API endpoint to hit, and it handles the rest. It gives you all messages, handlers for the text input, submit handlers, and the response state.
Our code looks roughly like this:
const {
messages,
input,
setInput,
handleSubmit,
isLoading,
} = useChat({
id,
api: "/api/docs/feedback",
body: { id, tddId },
initialMessages,
onError: (error) => {
toast.error("An error occured, please try again!");
},
});Then, we wire up the result of `useChat` to our UI.
`messages` gets mapped over and rendered to various components based on the shape of the data
`input` and `setInput` are wired up to the text input component
`handleSubmit` is triggered on enter key press and pressing the submit button
`isLoading` controls whether the loading state is displayed
With that, you have everything you need to build an AI chatbot! You now know how 90%+ of the AI chatbots out there are built, and have a good understanding of how you could build your own.
📖 TL;DR
Use existing tools like the Vercel AI SDK and the AI Chatbot Template to give yourself a foundation with examples.
The two core entities are Chat and Message. Store message contents as JSON for increased flexibility.
On the backend, when you receive a message, store it, respond, and store the response. Stream using the Vercel streaming APIs.
Render the messages via react-markdown and customize the markdown → HTML mappers as needed
Store chat state and hit the backend via the useChat hook from the Vercel AI SDK.
👏 Shout-outs of the week
How to be a force multiplier on
by Steve Huynh — 7 highly actionable tips with stories to boot. Highly recommend!- by Wes Kao — Great advice on reordering your “but”, or using “and” to avoid miscommunication and soften the blow on hard feedback.
And finally, a special shout-out to Artiom Dashinky on his new book, “Staff Product Designer,” which mentioned multiple High Growth Engineer articles! It was so cool to see the “Becoming a go-to person” article takeaways make it into the book.
Thank you for reading and being a supporter in growing the newsletter 🙏
Let’s get to 90k by next week! We’re just a few hundred readers away!
You can also hit the like ❤️ button at the bottom of this email to support me or share it with a friend to earn referral rewards. It helps me a ton!
 | A guest post by
|
Great breakdown of the chat and messaging system in WriteEdge!