Streaming Responses in a Website Chatbot Widget
Part 13's widget waits for the entire real answer before displaying anything — the exact non-streaming pattern the AI Fundamentals series' own streaming part already flagged as a real UX gap. This part fixes it, bringing genuine, real-time token display to the actual website widget.
Real, streaming-enabled backend using Server-Sent Events
# widget_api.py
from fastapi import FastAPI
from fastapi.responses import StreamingResponse
from shared.ai_client import get_client
app = FastAPI()
def generate_stream(question: str, context: str):
client = get_client()
with client.messages.stream(
model="claude-sonnet-5",
max_tokens=512,
system=f"Answer using ONLY this real information:\n\n{context}",
messages=[{"role": "user", "content": question}],
) as stream:
for text in stream.text_stream:
yield f"data: {text}\n\n"
@app.post("/api/chat-stream")
def chat_stream(request: ChatRequest):
relevant_docs = retrieve_context(request.question) # part 6's real retrieval
context = "\n\n".join(relevant_docs)
return StreamingResponse(
generate_stream(request.question, context),
media_type="text/event-stream",
)Server-Sent Events (SSE) is a real, standard web technology for a server to push a real, ongoing stream of data to a browser over one open connection — genuinely well-suited to this exact use case, and this backend directly reuses the AI Fundamentals series' own streaming technique, now wrapped in a real, standard web protocol a browser can consume natively.
The real, updated frontend JavaScript
<script>
async function askStreaming(question) {
const response = await fetch("/api/chat-stream", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ question }),
});
const reader = response.body.getReader();
const decoder = new TextDecoder();
const messageEl = document.createElement("p");
messages.appendChild(messageEl);
while (true) {
const { done, value } = await reader.read();
if (done) break;
const chunk = decoder.decode(value);
const lines = chunk.split("\n").filter((l) => l.startsWith("data: "));
for (const line of lines) {
messageEl.textContent += line.replace("data: ", "");
}
}
}
</script>This real, updated function replaces part 13's single fetch().then() call — reader.read() is called in a real, genuine loop, appending each real chunk of text to the message element the instant it arrives, rather than waiting for one complete real response.
This directly demonstrates the AI Fundamentals series' own explanation of how ChatGPT generates answers as real, visible, working behavior — a Bright Leaf Coffee customer using this widget now sees the exact, real token-by-token generation process this entire site has explained conceptually since the very first AI series, happening live in their own browser.
Why this matters more for a real, embedded widget specifically
A full-page chat interface: a visitor has already committed to
waiting for an AI response
A real, small floating widget: competing directly with the rest of
a real, busy product page for a visitor's real, limited attention
— a long, silent wait feels disproportionately worse in this
smaller, more casual real contextThis is a genuinely real, practical UX reason streaming matters even more here than in a dedicated chat application — the perceived responsiveness gain (the AI Projects series' own honest framing of streaming's real benefit) has outsized real value in a lightweight, embedded widget context.
A real, graceful fallback for connection issues
try {
await askStreaming(question);
} catch (error) {
messageEl.textContent = "Sorry, we're having trouble responding right now.";
}This directly extends part 13's own error-handling discipline — a real, dropped SSE connection (a genuine, real network issue) needs the same honest, graceful degradation the AI Fundamentals series' own error-handling coverage established for the backend, applied here on the actual frontend.
A real, honest limitation: SSE is one-directional
SSE genuinely streams FROM server TO browser — it does not support
the browser sending additional real data back over the same open
connection mid-streamFor this widget's real, simple question-and-streamed-answer pattern, this is a non-issue. A genuinely more complex, bidirectional real-time feature (like the Claude Agent SDK's real capabilities covered in this site's own news) would need WebSockets instead — a real, different technology worth knowing about, though beyond this specific widget's actual needs.
Next: building a real knowledge-base chatbot — applying this series' full RAG and streaming stack to GreenDesk's genuinely larger, internal document set.