第 25 期 | LangSmith 进阶：Prompt 管理与 A/B 测试 (EN)

⏱ Est. reading time: 20 min Updated on 5/7/2026

🎯 Learning Objectives

Welcome, future AI masters, to the first installment of the LangChain Masterclass! Starting today, we embark on a journey to build production-grade AI applications. As our opening chapter, we are skipping the fluff and diving straight into the core, ensuring you thoroughly master the foundation of LangChain applications.

By the end of this lesson, you will:

  1. Deeply understand the three core components of LangChain Core: The roles and collaboration mechanisms of LLM Models, Prompt Templates, and Output Parsers.
  2. Master LangChain Expression Language (LCEL): Learn how to elegantly chain these core components using the pipe operator | to build a foundational LLM workflow.
  3. Lay a solid foundation for the Intelligent Support Copilot project: Hands-on build the first "brain" of our support assistant, enabling it to perform basic Q&A based on your instructions.
  4. Avoid common beginner pitfalls: Understand API Key security, the Prompt Engineering mindset, and LangChain performance considerations to save yourself from unnecessary headaches.

📖 Concept Breakdown

Why LangChain? Can't we just call OpenAI or Anthropic APIs directly? Sure! But that's like reinventing the wheel by smelting your own steel every time. LangChain is like a set of Lego blocks; it modularizes the repetitive, tedious, yet absolutely essential parts of your code. It is not just a library for calling LLMs; it is an orchestration framework for LLM application workflows.

Imagine our "Intelligent Support Knowledge Base" project: It needs to do more than just answer questions. It needs to understand user intent, retrieve information from a massive knowledge base, summarize answers, and even interact with external tools. If you write this logic from scratch every time, you will quickly sink into a swamp of spaghetti code. LangChain was created to solve this exact pain point. It provides standardized interfaces and components, allowing you to build complex LLM applications quickly and flexibly, just like snapping Legos together.

LangChain Core is the heart of LangChain. It defines the most fundamental and universal abstractions and functionalities for all LLM applications. In this lesson, we focus on three core components:

  1. LLM (Large Language Model):

    • The Concept: This is the "brain" of our support copilot, responsible for understanding, reasoning, and generating text. LangChain provides a unified interface wrapper for various LLMs—whether it's OpenAI's GPT series, Anthropic's Claude, or open-source models on Hugging Face, you can call them using a similar approach. This drastically lowers the cost of switching models, letting you focus on business logic rather than adapting to different APIs.
    • The Practice: In LangChain, LLMs are generally divided into two types: BaseLLM (for text completion, e.g., text-davinci-003) and BaseChatModel (for conversations, e.g., gpt-3.5-turbo). For conversational scenarios like customer support, we almost exclusively use BaseChatModel because it excels at handling chat history, understanding roles, and managing multi-turn interactions.

  2. Prompt Templates:

    • The Concept: Prompts are the soul of your LLM application. They transform your AI from an omniscient but clueless "blank slate" into a highly targeted, specialized expert. A good prompt guides the LLM to output exactly what you want—whether that's role-playing as a support agent, summarizing text, or generating specific data formats. For our support copilot, the prompt is key to defining its persona, tone, and domain expertise.
    • The Practice: LangChain offers powerful PromptTemplate and ChatPromptTemplate classes. ChatPromptTemplate is ideal for chat models, allowing you to define messages for different roles (SystemMessage, HumanMessage, AIMessage). This helps the LLM better understand context and role-play. For instance, we can use a SystemMessage to tell the LLM: "You are a professional support agent. Please answer questions in a friendly, concise tone."

  3. Output Parsers:

    • The Concept: LLMs spit out free-form text, but we often need structured data, such as a JSON object, a list, or a boolean value. An Output Parser acts as a "translator," taking unstructured LLM text and converting it into our predefined format. This is crucial for downstream programmatic logic.
    • The Practice: LangChain provides various parsers, from the simple StrOutputParser (returns a raw string) to the complex PydanticOutputParser (parses into Pydantic model objects). For our support copilot, if we want it to not only answer but also tag the question category (e.g., "Billing Issue"), an Output Parser becomes indispensable.

The "glue" that connects these components is LangChain Expression Language (LCEL).

LCEL

  • The Concept: LCEL sounds a bit fancy, but essentially, it's a superpower LangChain gives you to elegantly chain LLM modules together using the pipe operator |, just like building blocks. It not only makes code clean and readable, but more importantly, it provides advanced features out-of-the-box like asynchronous execution, streaming, parallel processing, and fallback mechanisms. This lays the groundwork for building high-performance, robust production applications.
  • The Practice: In LCEL, similar to Unix pipes, you pass the output of one component as the input to the next. For example, prompt | llm | output_parser forms a complete chain.

Now, let's use a Mermaid diagram to visualize how these LangChain Core components work together:

graph TD
    subgraph LangChain Core - The Skeleton of LLM Apps
        A[Prompt Template
Defines intent & format] --> B[LLM Model
Intelligence core, generates response];
        B --> C[Output Parser
Structured output, ensures format];
    end
    U[User Input
Raw question] --> A;
    C --> D[Structured/Formatted Output
Copilot response];

    style A fill:#e0f7fa,stroke:#00796b,stroke-width:2px,color:#000,font-weight:bold
    style B fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px,color:#000,font-weight:bold
    style C fill:#fff3e0,stroke:#ef6c00,stroke-width:2px,color:#000,font-weight:bold
    style U fill:#fce4ec,stroke:#ad1457,stroke-width:2px,color:#000,font-weight:bold
    style D fill:#f0f4c3,stroke:#9e9d24,stroke-width:2px,color:#000,font-weight:bold
    linkStyle 0 stroke-width:2px,stroke:red;
    linkStyle 1 stroke-width:2px,stroke:blue;
    linkStyle 2 stroke-width:2px,stroke:green;
    linkStyle 3 stroke-width:2px,stroke:purple;

This diagram clearly illustrates the entire flow from user input to final output. The user's question is first formatted and wrapped by the Prompt Template, then sent to the LLM Model for processing. The raw response generated by the LLM is then structured by the Output Parser, ultimately yielding our desired support answer. This workflow is the very first "brain" of our Intelligent Support Copilot!

💻 Hands-on Code Drill (Application in the Support Project)

Alright, the theory is clear. It's time to roll up our sleeves and apply these concepts to our "Intelligent Support Knowledge Base" project. In this session, we will build a foundational Q&A bot. It will receive user questions and respond in the persona of a professional support assistant.

Prerequisites

First, ensure your development environment is ready. We need to install the LangChain library and the SDK of your chosen LLM provider. Here, we will use OpenAI as an example.

# Python Environment
pip install langchain langchain-openai python-dotenv

# TypeScript/JavaScript Environment
npm install langchain @langchain/openai dotenv

Don't even think about hardcoding your API Key in your code! That's a massive red flag. We use python-dotenv or dotenv to manage environment variables. Create a .env file in your project root and paste your OpenAI API Key:

# .env file
OPENAI_API_KEY="sk-your-openai-api-key-here"

Python Implementation

import os
from dotenv import load_dotenv
from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate, HumanMessagePromptTemplate, SystemMessagePromptTemplate
from langchain_core.output_parsers import StrOutputParser
from langchain_core.messages import HumanMessage, SystemMessage

# 1. Load environment variables
load_dotenv()

# 2. Initialize the LLM model
# We choose ChatOpenAI because it's a chat model, better suited for support scenarios
# model_name can be chosen based on your needs, e.g., "gpt-4" for higher quality but higher cost
llm = ChatOpenAI(model_name="gpt-3.5-turbo", temperature=0.7)

# 3. Define the Prompt Template
# This is the "persona" and "instructions" for our support copilot
# SystemMessagePromptTemplate defines the background and behavioral guidelines
# HumanMessagePromptTemplate receives the user's question
prompt_template = ChatPromptTemplate.from_messages(
    [
        SystemMessagePromptTemplate.from_template(
            "You are a professional intelligent support assistant. Your task is to answer user questions about our product (Intelligent Support Knowledge Base) concisely and accurately. "
            "Please maintain a friendly and helpful tone. If a question is beyond your knowledge, politely inform the user that you cannot answer and suggest they contact human support."
        ),
        HumanMessagePromptTemplate.from_template("{user_question}"), # {user_question} is a placeholder
    ]
)

# 4. Initialize the Output Parser
# The simplest output parser, returns the LLM response directly as a string
output_parser = StrOutputParser()

# 5. Chain components using LCEL to build the support copilot chain
# The pipe operator `|` passes Prompt output to LLM input, and LLM output to Output Parser input
support_copilot_chain = prompt_template | llm | output_parser

# 6. Simulate user questions and get responses
def get_copilot_response(question: str) -> str:
    """
    Simulate the support copilot receiving a question and returning an answer.
    """
    print(f"\n--- User Question ---")
    print(f"User: {question}")
    
    # Invoke the chain to get the response
    # The key in the input dict must match the placeholder name defined in prompt_template
    response = support_copilot_chain.invoke({"user_question": question})
    
    print(f"\n--- Copilot Response ---")
    print(f"Copilot: {response}")
    return response

if __name__ == "__main__":
    print("Welcome to the Support Copilot! Type 'exit' to end the conversation.")
    while True:
        user_input = input("You: ")
        if user_input.lower() == "exit":
            print("Thank you for using the copilot. Goodbye!")
            break
        
        get_copilot_response(user_input)
        
        # We can try some edge cases
        if user_input.lower() == "test edge cases":
            get_copilot_response("What is your product's refund policy?")
            get_copilot_response("Tell me a joke.") # Should be restricted by System Prompt
            get_copilot_response("How do I configure the knowledge base retrieval strategy?")
            get_copilot_response("What is the weather like today?") # Should be restricted by System Prompt

TypeScript Implementation

import 'dotenv/config'; // Ensure environment variables are loaded at the top of the file
import { ChatOpenAI } from '@langchain/openai';
import { ChatPromptTemplate, HumanMessagePromptTemplate, SystemMessagePromptTemplate } from '@langchain/core/prompts';
import { StringOutputParser } from '@langchain/core/output_parsers';
import { RunnableSequence } from '@langchain/core/runnables';

// 1. Environment variables are loaded via 'dotenv/config'

// 2. Initialize the LLM model
// Use process.env.OPENAI_API_KEY to ensure the API Key is loaded correctly
const llm = new ChatOpenAI({
  modelName: "gpt-3.5-turbo",
  temperature: 0.7,
  openAIApiKey: process.env.OPENAI_API_KEY, // Explicitly pass the API Key
});

// 3. Define the Prompt Template
const promptTemplate = ChatPromptTemplate.fromMessages([
  SystemMessagePromptTemplate.fromTemplate(
    "You are a professional intelligent support assistant. Your task is to answer user questions about our product (Intelligent Support Knowledge Base) concisely and accurately. " +
    "Please maintain a friendly and helpful tone. If a question is beyond your knowledge, politely inform the user that you cannot answer and suggest they contact human support."
  ),
  HumanMessagePromptTemplate.fromTemplate("{user_question}"), // {user_question} is a placeholder
]);

// 4. Initialize the Output Parser
const outputParser = new StringOutputParser();

// 5. Chain components using LCEL to build the support copilot chain
// In TypeScript, RunnableSequence is the common way to build chains
const supportCopilotChain = RunnableSequence.from([
  promptTemplate,
  llm,
  outputParser,
]);

// 6. Simulate user questions and get responses
async function getCopilotResponse(question: string): Promise<string> {
  console.log(`\n--- User Question ---`);
  console.log(`User: ${question}`);

  // Invoke the chain to get the response
  const response = await supportCopilotChain.invoke({ user_question: question });

  console.log(`\n--- Copilot Response ---`);
  console.log(`Copilot: ${response}`);
  return response;
}

// Simulate interaction
async function main() {
  console.log("Welcome to the Support Copilot! Type 'exit' to end the conversation.");
  const readline = require('readline').createInterface({
    input: process.stdin,
    output: process.stdout
  });

  for await (const line of readline) {
    const userInput = line.trim();
    if (userInput.toLowerCase() === "exit") {
      console.log("Thank you for using the copilot. Goodbye!");
      readline.close();
      break;
    }

    await getCopilotResponse(userInput);

    // We can try some edge cases
    if (userInput.toLowerCase() === "test edge cases") {
      await getCopilotResponse("What is your product's refund policy?");
      await getCopilotResponse("Tell me a joke."); // Should be restricted by System Prompt
      await getCopilotResponse("How do I configure the knowledge base retrieval strategy?");
      await getCopilotResponse("What is the weather like today?"); // Should be restricted by System Prompt
    }
  }
}

if (require.main === module) {
  main();
}

Code Walkthrough:

  1. Load environment variables: This is a security best practice! Never write sensitive information directly into your code.
  2. Initialize LLM: We instantiated ChatOpenAI. model_name (or modelName) dictates which specific model to use. temperature controls the randomness of the generated text (0.0 is the most deterministic, 1.0 is the most random; 0.5-0.7 is usually the sweet spot for support scenarios).
  3. Define Prompt Template:
    • ChatPromptTemplate.from_messages is the recommended way to build conversational prompts.
    • SystemMessagePromptTemplate is used to set the AI's "persona" and "rules of engagement." This is the key to defining our support copilot! It lets the AI know it's a professional agent, how it should speak, what it can say, and what it shouldn't.
    • HumanMessagePromptTemplate receives the actual user input. {user_question} is a placeholder that LangChain will replace with the actual user question during invoke.
  4. Initialize Output Parser: Here we used the basic StrOutputParser (Python) or StringOutputParser (TypeScript), which simply returns the LLM's output directly as a string. In future lessons, we will introduce more complex parsers to extract structured data.
  5. Build the Chain: This is the magic of LCEL!
    • In Python, we directly use the pipe operator | to chain prompt_template, llm, and output_parser together.
    • In TypeScript, we use RunnableSequence.from to build the exact same chain. Regardless of the language, the core idea is: the output of the previous component serves as the input to the next.
  6. Invoke the Chain: By calling support_copilot_chain.invoke({"user_question": question}) (Python) or await supportCopilotChain.invoke({ user_question: question }) (TypeScript), we pass the user question into the chain and receive the final copilot response. The invoke method takes a dictionary (or JavaScript object) where the keys must exactly match the placeholder names defined in the Prompt Template.

Run this code, and you will see your first Support Copilot chatting with you right in your terminal! Try asking questions about the "Intelligent Support Knowledge Base," and also try asking out-of-scope questions to see if it politely declines as instructed by the System Prompt. That is the power of Prompt Engineering!

🚧 Pitfalls & Best Practices

As a senior AI architect, I've seen too many beginners stumble here. You must know these "pitfalls" upfront to go further:

  1. API Key Security is Paramount:

    • Pitfall: Hardcoding OPENAI_API_KEY in your code or pushing it to a public GitHub repository. This is equivalent to walking outside with your bank PIN taped to your forehead.
    • Solution: Always use environment variables! python-dotenv or dotenv are your best friends. In production environments, use secret management services provided by cloud vendors (like AWS Secrets Manager, Azure Key Vault, Google Secret Manager). During local development, ensure your .env file is ignored by .gitignore.

  2. Prompt Engineering is Both Art and Science:

    • Pitfall: Thinking you can just throw a question at the LLM without spending time crafting the prompt. The result: The LLM gives chaotic or irrelevant answers.
    • Solution: The prompt is your only bridge to communicating with the LLM. It needs to be clear, explicit, and structured.
      • Persona Definition: Clearly tell the LLM what role it should play ("You are a professional support assistant").
      • Task Description: Clarify what the task is ("Answer user questions about the product").
      • Constraints: Set limits on output format, tone, length, and fallback strategies for unknown situations ("If beyond your knowledge, politely inform the user").
      • Iterative Refinement: Prompts aren't built in a day; they require continuous testing, tweaking, and optimization.

  3. LLM Model Selection and Cost Considerations:

    • Pitfall: Blindly using the most powerful model (like gpt-4) from day one, leading to skyrocketing costs and slow response times. Or, using an underpowered model to save money, resulting in poor quality.
    • Solution: Choose the right model based on your business needs and budget.
      • gpt-3.5-turbo: High cost-performance, fast speed, suitable for most daily support Q&A.
      • gpt-4: Stronger reasoning capabilities, suited for complex problems and multi-step reasoning, but more expensive and slower.
      • Open-source models (via Ollama/HuggingFace): Lowest cost (self-hosted), but requires infrastructure management. Performance might lag behind commercial models. Suitable for scenarios with strict data privacy requirements or tight budgets.
      • Temperature: Setting it to 0.0 makes the model output more deterministic and repetitive; setting it to 1.0 makes it more creative and random. For support scenarios, 0.5-0.7 is usually ideal—maintaining flexibility without hallucinating facts.

  4. LLM "Hallucinations" and Fact-Checking:

    • Pitfall: Blindly trusting everything the LLM generates without any fact-checking, especially in critical business scenarios.
    • Solution: LLMs can "confidently spout nonsense," a phenomenon known as "hallucination." For a support copilot, this means it might invent product features or refund policies.
      • Prompt Constraints: Explicitly demand in the prompt: "Answer ONLY based on known information. If you don't know, say so."
      • RAG (Retrieval-Augmented Generation): This is the core technology we will dive deep into in upcoming lessons. By retrieving relevant information from an external knowledge base and providing it as context to the LLM, we can drastically reduce hallucinations and improve accuracy. This is mandatory for production-grade support bots!

  5. LCEL Async and Streaming:

    • Pitfall: Using synchronous LLM calls in scenarios with high traffic or where real-time feedback is needed, leading to slow responses and poor user experience.
    • Solution: LCEL natively supports asynchronous ainvoke() and streaming astream(). For support applications, users expect fast answers, so async and streaming outputs are must-have skills. We will gradually introduce these in future lessons. Knowing this now puts you ahead of the curve.

📝 Lesson Summary

Congratulations! In this first installment of the LangChain Masterclass, you didn't just deeply understand the three cornerstones of LangChain Core: LLM Models, Prompt Templates, and Output Parsers. More importantly, you used LangChain Expression Language (LCEL) to build your very first Intelligent Support Copilot.

We gave our support copilot its initial "brain" and "persona," enabling it to handle basic Q&A with a professional attitude based on your instructions. This seemingly simple chain—Prompt | LLM | Output Parser—is the starting point for all complex LLM applications.

You also learned the crucial pitfalls and best practices every senior developer must know when building LLM apps, including API Key security, the Prompt Engineering mindset, model selection, and understanding the limitations of LLMs.

This is just the first step of a long journey. In the upcoming lessons, we will continuously iterate on this foundation. We will add memory to our "Intelligent Support Knowledge Base," integrate external knowledge bases (RAG), call tools, implement multi-turn conversations, and progressively transform it into a true production-grade AI application.

Ready? See you in the next lesson!