Post Training Torchforge – Troubleshooting

# torchforge Troubleshooting Guide ## GPU Resource Issues ### Issue: Not Enough GPUs **Symptoms**: "Insufficient GPU resources" error **Solutions**: 1. **Reduce service requirements**: ```yaml services: generator: procs: 1 with_gpus: true trainer: procs: 1 with_gpus: true # Remove ref_model or use CPU ``` 2. **Use CPU for reference model**: ```yaml ref_model: with_gpus: false # Run on CPU ``` 3. **Share resources between services**: ```yaml services: generator: procs: 1 num_replicas: 1 colocate_with: trainer # Share GPU with trainer ``` ### Issue: Minimum GPU Requirements **Reference**: - SFT: 2+ GPUs (trainer + generator) - GRPO: 3+ GPUs (trainer + generator + ref_model) - Large models: 8+ GPUs with tensor parallelism ## Memory Issues ### Issue: OOM During Generation **Symptoms**: CUDA OOM in vLLM **Solutions**: 1. **Reduce batch size**: ```yaml grpo: n_samples: 4 # Reduce from 8 ``` 2. **Reduce sequence length**: ```yaml training: seq_len: 2048 # Reduce from 4096 ``` 3. **Reduce vLLM memory**: ```yaml generator: gpu_memory_utilization: 0.7 # Reduce from 0.9 ``` ### Issue: OOM During Training **Symptoms**: CUDA OOM in backward pass **Solutions**: 1. **Enable gradient checkpointing**: ```yaml training: gradient_checkpointing: true ``` 2. **Increase gradient accumulation**: ```yaml training: gradient_accumulation_steps: 8 # Increase from 4 ``` 3. **Reduce batch size**: ```yaml training: batch_size: 2 # Reduce from 4 ``` ## Weight Synchronization Issues ### Issue: Slow Weight Sync **Symptoms**: Long pauses between training and generation **Solutions**: 1. **Enable RDMA** (if available): ```bash export TORCHSTORE_USE_RDMA=1 ``` 2. **Reduce sync frequency**: ```yaml training: sync_interval: 10 # Sync every 10 steps ``` 3. **Use colocated services**: ```yaml services: generator: colocate_with: trainer ``` ### Issue: Weight Sync Failures **Symptoms**: Errors in weight transfer, stale weights **Solutions**: 1. **Check network connectivity**: ```bash ping other_node ``` 2. **Increase timeout**: ```yaml services: weight_sync_timeout: 600 # 10 minutes ``` 3. **Enable sync verification**: ```yaml training: verify_weight_sync: true ``` ## Training Stability Issues ### Issue: Policy Collapse **Symptoms**: Entropy drops to zero, reward stops improving **Solutions**: 1. **Increase KL penalty**: ```yaml grpo: beta: 0.2 # Increase from 0.1 ``` 2. **Add entropy bonus**: ```yaml training: entropy_coef: 0.01 ``` 3. **Reduce learning rate**: ```yaml training: learning_rate: 5e-7 # Reduce from 1e-6 ``` ### Issue: Loss Spikes **Symptoms**: Sudden loss increases, training instability **Solutions**: 1. **Enable gradient clipping**: ```yaml training: max_grad_norm: 1.0 ``` 2. **Reduce clip range**: ```yaml grpo: clip_low: 0.1 # Reduce from 0.2 clip_high: 0.18 # Reduce from 0.28 ``` 3. **Use learning rate warmup**: ```yaml training: warmup_steps: 100 ``` ### Issue: Divergent Training **Symptoms**: Loss becomes NaN, model outputs garbage **Solutions**: 1. **Check for data issues**: ```python # Verify no empty sequences for batch in dataset: assert batch.input_ids.numel() > 0 ``` 2. **Use BF16 instead of FP16**: ```yaml training: dtype: bfloat16 ``` 3. **Reduce learning rate significantly**: ```yaml training: learning_rate: 1e-7 ``` ## Service Issues ### Issue: Service Startup Failures **Symptoms**: Services fail to initialize **Solutions**: 1. **Check resource availability**: ```bash nvidia-smi # Verify GPU availability ``` 2. **Increase startup timeout**: ```yaml services: startup_timeout: 600 ``` 3. **Check model path**: ```python from transformers import AutoModelForCausalLM model = AutoModelForCausalLM.from_pretrained("model_path") # Verify accessible ``` ### Issue: Generator Not Responding **Symptoms**: Generation hangs, timeouts **Solutions**: 1. **Check vLLM status**: ```python # Add health check await generator.health_check.route() ``` 2. **Restart service**: ```python await generator.restart.fanout() ``` 3. **Reduce concurrent requests**: ```yaml generator: max_concurrent_requests: 10 ``` ## Monarch Issues ### Issue: Monarch Actor Failures **Symptoms**: Actor crashes, communication errors **Solutions**: 1. **Enable fault tolerance**: ```yaml monarch: fault_tolerance: true max_restarts: 3 ``` 2. **Increase actor memory**: ```yaml services: actor_memory_mb: 4096 ``` 3. **Check Monarch logs**: ```bash export MONARCH_LOG_LEVEL=DEBUG ``` ### Issue: Deadlock in Distributed Communication **Symptoms**: Training hangs, no progress **Solutions**: 1. **Check for blocking calls**: ```python # Use async/await correctly result = await service.method.route(args) # Correct # result = service.method.route(args).wait() # May deadlock ``` 2. **Add timeouts**: ```python result = await asyncio.wait_for( service.method.route(args), timeout=60.0 ) ``` ## Installation Issues ### Issue: PyTorch Version Mismatch **Symptoms**: Import errors, CUDA errors **Solutions**: 1. **Use provided install script**: ```bash ./scripts/install.sh ``` 2. **Verify versions**: ```python import torch print(torch.__version__) # Should be 2.9.0+ ``` 3. **Clean reinstall**: ```bash pip uninstall torch torchvision torchaudio ./scripts/install.sh ``` ### Issue: Monarch Installation Fails **Symptoms**: Cannot import monarch **Solutions**: 1. **Install from source**: ```bash git clone https://github.com/meta-pytorch/monarch cd monarch && pip install -e . ``` 2. **Check CUDA compatibility**: ```bash nvcc --version # Should match PyTorch CUDA ``` ## Debugging Tips ### Enable Verbose Logging ```bash export FORGE_DEBUG=1 export MONARCH_LOG_LEVEL=DEBUG ``` ### Profile Services ```python # Add profiling with torch.profiler.profile() as prof: result = await trainer.train_step.route(batch) prof.export_chrome_trace("trace.json") ``` ### Monitor GPU Utilization ```bash watch -n 1 nvidia-smi ``` ### Test Services Individually ```python # Test generator completions = await generator.generate.route( prompts=["Hello"], max_tokens=10, ) print(completions[0].text) # Test trainer result = await trainer.train_step.route(dummy_batch) print(result.loss) ``` ## Experimental Warning Both Monarch and torchforge are experimental. Expect: - API changes between versions - Incomplete features - Bugs in edge cases Check Discord for latest updates and workarounds. ## Resources - GitHub Issues: https://github.com/meta-pytorch/torchforge/issues - Discord: https://discord.gg/YsTYBh6PD9 - Monarch Issues: https://github.com/meta-pytorch/monarch/issues