Skip to content

RD-Agent Tutorial - Chapter 1: Basic Introduction

1.1 Project Overview and Core Philosophy

RD-Agent Introduction and Background

RD-Agent (Research & Development Agent) is an open-source automated R&D framework developed by Microsoft Research, designed specifically for building data-driven AI solutions. The project achieved first place in the industry-authoritative machine learning engineering benchmark MLE-bench, making it the best-performing machine learning engineering agent to date.

Core Philosophy: The R&D Framework

RD-Agent's design philosophy is based on two core components:

  • R (Research): 🔬 The research component
  • Automatically proposes new hypotheses and ideas
  • Analyzes the limitations of existing solutions
  • Generates innovative solutions based on domain knowledge

  • D (Development): 🛠️ The development component

  • Translates research ideas into executable code
  • Implements and validates hypotheses
  • Continuously optimizes and improves solutions

MLE-bench Benchmark Performance

Agent Low (%) Medium (%) High (%) All (%)
R&D-Agent o1-preview 48.18 ± 2.49 8.95 ± 2.36 18.67 ± 2.98 22.4 ± 1.1
R&D-Agent o3(R)+GPT-4.1(D) 51.52 ± 6.21 7.89 ± 3.33 16.67 ± 3.65 22.45 ± 2.45
AIDE o1-preview 34.3 ± 2.4 8.8 ± 1.1 10.0 ± 1.9 16.9 ± 1.1

Core Advantages and Application Value

1. Automated Data-Driven R&D Workflow

  • End-to-end automation: The complete pipeline from data analysis to model deployment
  • Intelligent decision support: Automatically generates solutions based on data insights
  • Continuous learning capability: Learns from historical experience and keeps improving

2. Multi-Agent Collaborative Architecture

┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   研究代理       │    │   开发代理       │    │   评估代理       │
│  Research Agent │◄──►│ Development     │◄──►│ Evaluation      │
│                 │    │ Agent           │    │ Agent           │
├─────────────────┤    ├─────────────────┤    ├─────────────────┤
│• 假设生成        │    │• 代码实现        │    │• 性能评估        │
│• 策略规划        │    │• 测试验证        │    │• 反馈生成        │
│• 知识整合        │    │• 优化改进        │    │• 质量控制        │
└─────────────────┘    └─────────────────┘    └─────────────────┘

3. Continuous Learning and Evolution Capability

  • CoSTEER framework: A collaborative evolution strategy framework
  • Knowledge accumulation: Automatically builds and maintains domain knowledge bases
  • Adaptive optimization: Dynamically adjusts strategies based on feedback

Applicable Scenarios and Target Users

🏢 Quantitative Investment Teams

  • Automatic factor mining: Discover effective factors from massive datasets
  • Strategy development: Automatically generate and optimize trading strategies
  • Risk management: Intelligent risk control and portfolio optimization
  • Cost efficiency: Achieve 2x returns at a cost of less than $10 compared to traditional methods

🔬 Data Scientists

  • Feature engineering automation: Intelligent feature generation and selection
  • Model selection optimization: Automated model architecture search
  • Experiment management: Systematic experiment tracking and comparison
  • Pipeline construction: Automatic construction of end-to-end machine learning pipelines

🎓 Researchers and Academics

  • Paper reproduction: Automatically extract and implement models from papers
  • Research assistant: Intelligent literature analysis and knowledge extraction
  • Experiment design: Scientific experiment plan generation and execution
  • Results organization: Automated documentation of research results

🏆 Kaggle Competition Participants

  • End-to-end automation: From data exploration to model submission
  • Feature engineering: Intelligent feature creation and combination
  • Model ensembling: Multi-model fusion strategy optimization
  • Time management: Efficient competition time allocation

1.2 System Requirements and Environment Preparation

Hardware and Software Requirements

Operating System Support

  • Linux: ✅ Fully supported (Ubuntu 18.04+, CentOS 7+)
  • Windows: ❌ Not currently supported
  • macOS: ❌ Not currently supported

Important note: RD-Agent currently only supports Linux. If you use Windows or macOS, it is recommended to run a Linux environment using Docker, WSL2, or a virtual machine.

Python Version Requirements

  • Python 3.10: ✅ Fully supported (recommended)
  • Python 3.11: ✅ Fully supported
  • Python 3.9: ⚠️ Some features may be limited
  • Python 3.12+: ⚠️ Compatibility issues may exist

Memory and Compute Resource Recommendations

Usage Scenario Memory Requirement CPU Requirement Storage Space
Light usage 4GB+ 2 cores+ 10GB+
Medium scale 8GB+ 4 cores+ 50GB+
Large-scale applications 16GB+ 8 cores+ 100GB+
Production environment 32GB+ 16 cores+ 500GB+

Docker Environment Configuration

Docker Installation and Configuration

RD-Agent strongly depends on Docker to provide an isolated runtime environment, ensuring experiment reproducibility.

Installing Docker on Ubuntu/Debian:

# 更新包索引
sudo apt-get update

# 安装必要的包
sudo apt-get install \
    apt-transport-https \
    ca-certificates \
    curl \
    gnupg \
    lsb-release

# 添加 Docker 官方 GPG 密钥
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg

# 设置稳定版本仓库
echo \
  "deb [arch=amd64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu \
  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

# 安装 Docker Engine
sudo apt-get update
sudo apt-get install docker-ce docker-ce-cli containerd.io

Installing Docker on CentOS/RHEL:

# 安装必要的包
sudo yum install -y yum-utils

# 添加 Docker 仓库
sudo yum-config-manager \
    --add-repo \
    https://download.docker.com/linux/centos/docker-ce.repo

# 安装 Docker Engine
sudo yum install docker-ce docker-ce-cli containerd.io

# 启动 Docker 服务
sudo systemctl start docker
sudo systemctl enable docker

Running Without sudo Privileges

This is a required configuration step! RD-Agent requires the current user to be able to run Docker commands without using sudo.

# 创建 docker 用户组(如果不存在)
sudo groupadd docker

# 将当前用户添加到 docker 组
sudo usermod -aG docker $USER

# 重新加载用户组权限(或者重新登录)
newgrp docker

# 验证配置是否成功
docker run hello-world

Expected output:

Hello from Docker!
This message shows that your installation appears to be working correctly.
...

LLM API Preparation

Supported LLM Providers

RD-Agent supports multiple large language model providers through LiteLLM:

Provider Supported Models Recommended Use Case Cost Consideration
OpenAI GPT-4, GPT-4-turbo, GPT-3.5-turbo General scenarios, high-quality output $$$
Azure OpenAI GPT-4, GPT-3.5-turbo Enterprise applications, compliance requirements $$$
DeepSeek deepseek-chat, deepseek-coder Code generation, cost-effective $
Anthropic Claude-3.x Long-context processing, analysis tasks $$
Google Gemini Pro Multimodal needs $$

API Key Acquisition Guide

Obtaining an OpenAI API key: 1. Visit the OpenAI API website 2. Register/log in to your account 3. Go to the "API Keys" page 4. Click "Create new secret key" 5. Copy the API key and store it securely

Azure OpenAI configuration: 1. Log in to the Azure Portal 2. Create an Azure OpenAI resource 3. Obtain the API key, endpoint, and deployment name 4. Record the API version information

Obtaining a DeepSeek API key: 1. Visit the DeepSeek API platform 2. Register an account and complete verification 3. Go to the API key management page 4. Generate a new API key

Cost Budget Recommendations

Usage Intensity Monthly Cost Budget Applicable Scenarios
Learning/trial $10-50 Tutorial learning, small-scale experiments
Medium usage $50-200 Project development, medium-scale applications
Heavy usage $200-1000 Production environments, large-scale applications
Enterprise grade $1000+ Enterprise deployment, continuous operation

1.3 Installation and Basic Configuration

Choosing an Installation Method

For most users, installing directly from PyPI is the simplest approach:

# 创建 conda 环境(推荐)
conda create -n rdagent python=3.10
conda activate rdagent

# 或使用 venv
python3.10 -m venv rdagent_env
source rdagent_env/bin/activate

# 安装 RD-Agent
pip install rdagent

Developer Version Installation

If you want to try the latest features or participate in development, you can install from source:

# 克隆仓库
git clone https://github.com/microsoft/RD-Agent.git
cd RD-Agent

# 创建开发环境
conda create -n rdagent-dev python=3.10
conda activate rdagent-dev

# 安装开发依赖
make dev

# 或者手动安装
pip install -e .
pip install -r requirements/dev.txt

Dependency Management and Virtual Environments

Why do you need a virtual environment? - Avoids package version conflicts - Ensures environment consistency - Makes environment management and migration easier

Recommended environment management approach:

# 使用 conda(推荐)
conda create -n rdagent python=3.10
conda activate rdagent

# 或使用 pyenv + venv
pyenv install 3.10.12
pyenv local 3.10.12
python -m venv rdagent_env
source rdagent_env/bin/activate

Basic Environment Configuration

.env File Configuration Details

RD-Agent uses a .env file to manage environment variable configuration. Create a .env file in the project root directory:

# 在项目目录下创建 .env 文件
touch .env

LiteLLM Unified Backend Configuration

Option 1: Unified API base configuration (recommended for beginners)

# OpenAI 配置示例
CHAT_MODEL=gpt-4o
EMBEDDING_MODEL=text-embedding-3-small
OPENAI_API_KEY=sk-your-openai-api-key-here
OPENAI_API_BASE=https://api.openai.com/v1

Option 2: Split API configuration (advanced users)

# 聊天模型配置
CHAT_MODEL=gpt-4o
OPENAI_API_KEY=sk-your-openai-api-key-here
OPENAI_API_BASE=https://api.openai.com/v1

# 嵌入模型配置(使用不同提供商)
EMBEDDING_MODEL=litellm_proxy/BAAI/bge-large-en-v1.5
LITELLM_PROXY_API_KEY=your-siliconflow-api-key
LITELLM_PROXY_API_BASE=https://api.siliconflow.cn/v1

Configuration Examples for Multiple LLM Providers

Standard OpenAI configuration:

# 基础配置
CHAT_MODEL=gpt-4o
EMBEDDING_MODEL=text-embedding-3-small
OPENAI_API_KEY=sk-proj-xxxxxxxxxxxxxxxxxxxxxxxxxx
OPENAI_API_BASE=https://api.openai.com/v1

# 可选:推理模型特殊配置
# REASONING_THINK_RM=True  # 启用思维过程移除

Azure OpenAI configuration:

# Azure OpenAI 配置
CHAT_MODEL=azure/gpt-4o-deployment-name
EMBEDDING_MODEL=azure/embedding-deployment-name
AZURE_API_KEY=your-azure-api-key
AZURE_API_BASE=https://your-resource.openai.azure.com/
AZURE_API_VERSION=2024-02-15-preview

DeepSeek budget configuration:

# DeepSeek 聊天模型(经济实惠)
CHAT_MODEL=deepseek/deepseek-chat
DEEPSEEK_API_KEY=sk-your-deepseek-api-key

# SiliconFlow 嵌入模型(DeepSeek 无嵌入模型)
EMBEDDING_MODEL=litellm_proxy/BAAI/bge-m3
LITELLM_PROXY_API_KEY=sk-your-siliconflow-api-key
LITELLM_PROXY_API_BASE=https://api.siliconflow.cn/v1

# 推理模型配置
REASONING_THINK_RM=True

Hybrid configuration (recommended for advanced users):

# 高性能聊天模型
CHAT_MODEL=gpt-4o
OPENAI_API_KEY=sk-your-openai-api-key

# 经济型嵌入模型
EMBEDDING_MODEL=litellm_proxy/BAAI/bge-large-en-v1.5
LITELLM_PROXY_API_KEY=sk-your-siliconflow-api-key
LITELLM_PROXY_API_BASE=https://api.siliconflow.cn/v1

# 其他配置
REASONING_THINK_RM=False

Health Check and Verification

Using rdagent health_check

After installation is complete, first run a system health check:

# 完整健康检查
rdagent health_check

# 跳过环境变量检查
rdagent health_check --no-check-env

# 跳过 Docker 检查
rdagent health_check --no-check-docker

# 只检查端口占用
rdagent health_check --no-check-env --no-check-docker

Expected successful output:

✅ Docker is installed and working correctly
✅ Port 19899 is available
✅ Environment variables are configured correctly
✅ LLM API connection successful
✅ All systems check passed!

Troubleshooting Common Configuration Issues

Issue 1: Docker permission error

❌ Error: Got permission denied while trying to connect to the Docker daemon socket
Solution:
# 确保用户在 docker 组中
sudo usermod -aG docker $USER
newgrp docker
# 重新运行健康检查
rdagent health_check

Issue 2: Invalid API key

❌ Error: Invalid API key provided
Solution:
# 检查 .env 文件中的 API 密钥
cat .env | grep API_KEY
# 确保密钥正确且有效
# 重新配置 API 密钥

Issue 3: Port already in use

❌ Error: Port 19899 is already in use
Solution:
# 查看端口占用情况
netstat -tlnp | grep 19899
# 结束占用进程或更换端口
rdagent ui --port 19900

Issue 4: Network connection issues

❌ Error: Failed to connect to API endpoint
Solution:
# 检查网络连接
curl -I https://api.openai.com/v1/models
# 配置代理(如需要)
export https_proxy=http://proxy.company.com:8080
export http_proxy=http://proxy.company.com:8080

Environment Verification Checklist

Before starting to use RD-Agent, please confirm the following checklist items:

  • [ ] Operating system: Linux environment
  • [ ] Python version: 3.10 or 3.11
  • [ ] Virtual environment: A dedicated environment is activated
  • [ ] Docker: Installed and runnable without sudo
  • [ ] Network connection: Able to access LLM API endpoints
  • [ ] API key: Valid and with sufficient balance
  • [ ] Storage space: At least 10GB of available space
  • [ ] Port: Port 19899 available (or specify another port)
  • [ ] Permissions: The current user has read/write permissions on the project directory

At this point, you have completed the basic installation and configuration of RD-Agent. In the next chapter, we will introduce RD-Agent's core features and usage in detail.

Quick Installation Verification

After completing all configuration, let's run a quick functional verification:

# 激活环境
conda activate rdagent

# 验证安装
python -c "import rdagent; print('RD-Agent installed successfully!')"

# 运行健康检查
rdagent health_check

# 查看可用命令
rdagent --help

If all steps completed successfully, you are now ready to start using the powerful capabilities of RD-Agent!