serving

# 生产服务指南 在生产环境中部署 TensorRT-LLM 的综合指南。 ## 服务器模式 ### trtllm-serve (推荐) **特性**: - 兼容 OpenAI API - 自动下载并编译模型 - 内置负载均衡 - Prometheus 指标监控 - 健康检查 **基础用法**: bash trtllm-serve meta-llama/Meta-Llama-3-8B --tp_size 1 --max_batch_size 256 --port 8000 **高级配置**: bash trtllm-serve meta-llama/Meta-Llama-3-70B --tp_size 4 --dtype fp8 --max_batch_size 256 --max_num_tokens 4096 --enable_chunked_context --scheduler_policy max_utilization --port 8000 --api_key $API_KEY # 可选认证 ### Python LLM API (用于嵌入式服务) python from tensorrt_llm import LLM class LLMService: def __init__(self): self.llm = LLM( model="meta-llama/Meta-Llama-3-8B", dtype="fp8" ) def generate(self, prompt, max_tokens=100): from tensorrt_llm import SamplingParams params = SamplingParams( max_tokens=max_tokens, temperature=0.7 ) outputs = self.llm.generate([prompt], params) return outputs[0].text # 在 FastAPI, Flask 等中使用 from fastapi import FastAPI app = FastAPI() service = LLMService() @app.post("/generate") def generate(prompt: str): return {"response": service.generate(prompt)} ## 兼容 OpenAI 的 API ### 聊天补全 (Chat Completions) bash curl -X POST http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{ "model": "meta-llama/Meta-Llama-3-8B", "messages": [ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Explain quantum computing"} ], "temperature": 0.7, "max_tokens": 500, "stream": false }' **响应**: { "id": "chat-abc123", "object": "chat.completion", "created": 1234567890, "model": "meta-llama/Meta-Llama-3-8B", "choices": [{ "index": 0, "message": { "role": "assistant", "content": "Quantum computing is..." }, "finish_reason": "stop" }], "usage": { "prompt_tokens": 25, "completion_tokens": 150, "total_tokens": 175 } } ### 流式传输 bash curl -X POST http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{ "model": "meta-llama/Meta-Llama-3-8B", "messages": [{"role"}]