When interacting with AI chat services like ChatGPT, Claude, or Gemini, you observe the AI's response being streamed character by character onto the screen. This provides a fluid and interactive experience. How is this real-time, bit-by-bit output functionally implemented?
Fundamentally, web services operate on the HTTP protocol, which follows a unidirectional communication model: the client sends a request, and the server returns a single, complete response. However, for AI chat, we need the server to dispatch AI message tokens to the client as soon as they are generated, a scenario not well-suited for traditional HTTP request-response pairs.
Can We Use WebSockets?
As an alternative, one might consider WebSockets, but this approach presents several drawbacks:
- Streaming messages from server to client does not inherently require a bidirectional channel; a multi-chunk response to a single client request is sufficient.
- Since WebSockets are not pure HTTP, they cannot inherently leverage existing HTTP features out of the box, such as authentication (via cookies or tokens), CORS policies, caching mechanisms, or comprehensive logging.
- Scaling becomes more challenging. Load balancer configuration is particularly complex. In typical web services, load balancers distribute traffic efficiently across multiple servers. With WebSockets, however, a specific server must maintain a persistent connection with the client, necessitating "Sticky Sessions." This hinders horizontal scaling by preventing even traffic distribution across multiple backend servers.
The Answer is Server-Sent Events (SSE)
The solution to this problem can actually be found within the HTTP protocol itself, utilizing **Server-Sent Events (SSE)**.
A typical HTTP response sends its entire payload at once, signaling completion with a Content-Length header. The browser then closes the connection once it receives the specified amount of data.
In contrast, an SSE response adopts HTTP/1.1 200 OK but crucially employs Transfer-Encoding: chunked and Content-Type: text/event-stream.
This configuration instructs the browser to assume the response is ongoing. It keeps the connection open indefinitely and processes data incrementally as each chunk arrives. By leveraging this mechanism, we can maintain adherence to the standard HTTP protocol while enabling the server to send a response in multiple segments for a single request. This is how AI message streaming is achieved.