Adapter System

# Adapter and Interceptor System NeMo Evaluator uses an adapter system to process requests and responses between the evaluation engine and model endpoints. The `nemo-evaluator` core library provides built-in interceptors for common use cases. ## Architecture Overview ``` ┌───────────────────────────────────────────────────────────────┐ │ Adapter Pipeline │ │ │ │ Request ───► [Interceptor 1] ───► [Interceptor 2] ───► │ │ │ │ │ │ │ ▼ │ │ ┌───────────────────────────────────┐ │ │ │ Endpoint Interceptor │ │ │ │ (HTTP call to Model API) │ │ │ └───────────────────────────────────┘ │ │ │ │ │ ▼ │ │ Response ◄─── [Interceptor 3] ◄─── [Interceptor 4] ◄─── │ │ │ └───────────────────────────────────────────────────────────────┘ ``` Interceptors execute in order for requests, and in reverse order for responses. ## Configuring Adapters The adapter configuration is specified in the `target.api_endpoint.adapter_config` section: ```yaml target: api_endpoint: model_id: meta/llama-3.1-8b-instruct url: https://integrate.api.nvidia.com/v1/chat/completions api_key_name: NGC_API_KEY adapter_config: interceptors: - name: system_message config: system_message: "You are a helpful assistant." - name: caching config: cache_dir: "./cache" - name: endpoint - name: reasoning config: start_reasoning_token: "" end_reasoning_token: "" ``` ## Available Interceptors ### System Message Interceptor Injects a system prompt into chat requests. ```yaml - name: system_message config: system_message: "You are a helpful AI assistant. Think step by step." ``` **Effect**: Prepends a system message to the messages array. ### Request Logging Interceptor Logs outbound API requests for debugging and analysis. ```yaml - name: request_logging config: max_requests: 1000 ``` ### Caching Interceptor Caches responses to avoid repeated API calls for identical requests. ```yaml - name: caching config: cache_dir: "./evaluation_cache" reuse_cached_responses: true save_requests: true save_responses: true max_saved_requests: 1000 max_saved_responses: 1000 ``` ### Endpoint Interceptor Performs the actual HTTP communication with the model endpoint. This is typically added automatically and has no configuration parameters. ```yaml - name: endpoint ``` ### Reasoning Interceptor Extracts and removes reasoning tokens (e.g., `` tags) from model responses. ```yaml - name: reasoning config: start_reasoning_token: "" end_reasoning_token: "" enable_reasoning_tracking: true ``` **Effect**: Strips reasoning content from the response and tracks it separately. ### Response Logging Interceptor Logs API responses. ```yaml - name: response_logging config: max_responses: 1000 ``` ### Progress Tracking Interceptor Reports evaluation progress to an external URL. ```yaml - name: progress_tracking config: progress_tracking_url: "http://localhost:3828/progress" progress_tracking_interval: 10 ``` ### Additional Interceptors Other available interceptors include: - `payload_modifier`: Transforms request parameters - `response_stats`: Collects aggregated statistics from responses - `raise_client_errors`: Handles and raises exceptions for client errors (4xx) ## Interceptor Chain Example A typical interceptor chain for evaluation: ```yaml adapter_config: interceptors: # Pre-endpoint (request processing) - name: system_message config: system_message: "You are a helpful AI assistant." - name: request_logging config: max_requests: 50 - name: caching config: cache_dir: "./evaluation_cache" reuse_cached_responses: true # Endpoint (HTTP call) - name: endpoint # Post-endpoint (response processing) - name: response_logging config: max_responses: 50 - name: reasoning config: start_reasoning_token: "" end_reasoning_token: "" ``` ## Python API Usage You can also configure adapters programmatically: ```python from nemo_evaluator.adapters.adapter_config import AdapterConfig, InterceptorConfig from nemo_evaluator.api.api_dataclasses import ApiEndpoint, EndpointType adapter_config = AdapterConfig( interceptors=[ InterceptorConfig( name="system_message", config={"system_message": "You are a helpful assistant."} ), InterceptorConfig( name="caching", config={ "cache_dir": "./cache", "reuse_cached_responses": True } ), InterceptorConfig(name="endpoint"), InterceptorConfig( name="reasoning", config={ "start_reasoning_token": "", "end_reasoning_token": "" } ) ] ) api_endpoint = ApiEndpoint( url="http://localhost:8080/v1/chat/completions", type=EndpointType.CHAT, model_id="my_model", adapter_config=adapter_config ) ``` ## OpenAI API Compatibility NeMo Evaluator supports OpenAI-compatible endpoints with different endpoint types: ### Chat Completions ```yaml target: api_endpoint: type: chat # or omit, chat is default url: http://endpoint/v1/chat/completions ``` ### Text Completions ```yaml target: api_endpoint: type: completions url: http://endpoint/v1/completions ``` ### Vision-Language Models ```yaml target: api_endpoint: type: vlm url: http://endpoint/v1/chat/completions ``` ## Error Handling Configure error handling via the `log_failed_requests` option: ```yaml adapter_config: log_failed_requests: true interceptors: - name: raise_client_errors # ... other interceptors ``` ## Debugging ### Enable Logging Interceptors Add request and response logging to debug issues: ```yaml adapter_config: interceptors: - name: request_logging config: max_requests: 100 - name: endpoint - name: response_logging config: max_responses: 100 ``` ### Common Issues **Issue: System message not applied** Ensure the `system_message` interceptor is listed before the `endpoint` interceptor. **Issue: Cache not being used** Check that `reuse_cached_responses: true` is set and the cache directory exists: ```yaml - name: caching config: cache_dir: "./cache" reuse_cached_responses: true ``` **Issue: Reasoning tokens not extracted** Verify the token patterns match your model's output format: ```yaml - name: reasoning config: start_reasoning_token: "" # Must match model output exactly end_reasoning_token: "" ``` ## Custom Interceptor Discovery NeMo Evaluator supports discovering custom interceptors via the `DiscoveryConfig` within `AdapterConfig`. You can specify modules or directories where your custom interceptors are located: ```yaml adapter_config: discovery: modules: - "my_custom.interceptors" - "my_package.adapters" dirs: - "/path/to/custom/interceptors" interceptors: - name: my_custom_interceptor config: custom_option: value ``` Custom interceptors must implement the standard interceptor interface expected by `nemo-evaluator`. ## Additional AdapterConfig Options Beyond interceptors, `AdapterConfig` supports these additional fields: | Field | Description | |-------|-------------| | `discovery` | Configure custom interceptor discovery | | `post_eval_hooks` | List of hooks to run after evaluation | | `endpoint_type` | Default endpoint type (e.g., "chat") | | `caching_dir` | Legacy option for response caching | | `generate_html_report` | Generate HTML report of results | | `log_failed_requests` | Log requests that fail | | `tracking_requests_stats` | Enable request statistics | | `html_report_size` | Number of request-response pairs in report | ## Notes - The interceptor chain order matters - request interceptors run in order, response interceptors run in reverse - Interceptors can be enabled/disabled via the `enabled` field in `InterceptorConfig` - For complex custom logic, consider packaging as a custom container with your interceptors pre-installed