Skip to content

title: Chapter 15: Data Management

Chapter 15: Data Management

This chapter covers the Lean data format, data directory structure, data downloading, and custom data import.

15.1 Data Directory Structure

15.1.1 Full Directory Tree

Data/
├── equity/                    # 股票数据:equity/{market}/{resolution}/{symbol}.zip
│   └── usa/
│       ├── daily/
│       │   └── spy.zip       # SPY 日线数据
│       ├── minute/            # 分钟数据
│       ├── hour/              # 小时数据
│       ├── tick/              # 逐笔数据
│       ├── map_files/         # 映射文件:data/equity/usa/map_files/{SYMBOL}.csv
│       │   └── spy.csv
│       └── factor_files/      # 复权因子:data/equity/usa/factor_files/{SYMBOL}.csv
│           └── spy.csv
├── forex/                    # 外汇数据:forex/{market}/{resolution}
│   └── fxcm/
│       ├── daily/
│       └── minute/
├── crypto/                    # 加密货币:crypto/{market}/{resolution}
│   └── binance/
│       ├── daily/
│       └── minute/
├── future/                   # 期货数据:future/{market}/{resolution}
│   ├── cme/                   # 芝加哥商品交易所
│   │   ├── daily/
│   │   └── minute/
│   ├── cbot/                  # 芝加哥期货交易所
│   │   └── daily/
│   └── ...
├── option/                   # 期权数据:option/{market}/{resolution}/{underlying}/
│   └── usa/
│       ├── daily/
│       │   └── spy/
│       │       └── ...
│       └── minute/
└── cfd/                      # CFD数据:cfd/{market}/{resolution}
    └── fxcm/
        └── daily/

15.1.2 Data File Format

Equity Data Format (zip + csv)

Inside the zip is a headerless CSV; each row has exactly 6 fields in the order [time, Open, High, Low, Close, Volume]:

20130102 00:00,1440000,1449000,1431000,1440000,128499600
...

Field descriptions:

  • Time: for daily/hour data, in yyyyMMdd HH:mm format (e.g., 20130102 00:00); for minute/tick data, an integer number of milliseconds since midnight (e.g., 57000000 means 15:50:00.000).
  • Open/High/Low/Close: stored as integers; the actual price = value / 10000. For example, 1440000 corresponds to 144.0000 USD.
  • Volume: number of shares traded (integer).
  • Note: there are no separate date/time columns and no floating-point prices; equity data has no openinterest column.

Map Files and Factor Files

Map files are located at data/equity/usa/map_files/{SYMBOL}.csv (not under the daily/ directory). Each row has the format yyyymmdd,mapped symbol,P, where the date is the effective date for that symbol; the last line of the file is usually 20501231:

# data/equity/usa/map_files/spy.csv
19980102,SPY,P
...
20501231,SPY,P

Factor files are located at data/equity/usa/factor_files/{SYMBOL}.csv, with columns date,priceFactor,splitFactor representing the price factor and split factor respectively; the engine uses them to restore historical prices to the prices under the selected data normalization mode.

15.2 Downloading Data with the Lean CLI

15.2.1 Downloading Datasets

Data files are placed in the data/ directory (from QuantConnect Datasets); during local backtests the engine reads them following the directory structure in Section 15.1.

# 下载特定数据集
lean data download --dataset "QuantConnect - USA Equity Price Data"

# 具体下载参数因数据集而异,围绕 --dataset 指定数据集 ID,
# 其余选项详见 lean data download --help
lean data download --dataset "<数据集ID>"

# 下载期权数据
lean data download --dataset "QuantConnect - USA Option Price Data"

15.2.2 Data Providers

Provider Type Description
QuantConnect Comprehensive data Cloud data, requires a subscription
Local Local files Read from local files
Interactive Brokers Real-time data Used for live trading
CoinAPI Crypto Real-time cryptocurrency data

15.3 Custom Data Import

15.3.1 CSV Data Import

class CustomCsvData(PythonData):
    """
    自定义CSV数据导入

    CSV格式要求:
    date,open,high,low,close,volume
    2023-01-01,100.00,102.00,99.50,101.00,1000000
    """

    def GetSource(self, config, date, isLiveMode):
        """定义数据源"""
        return SubscriptionDataSource(
            "data/my_custom_data.csv",
            SubscriptionTransportMedium.LocalFile,
            FileFormat.Csv
        )

    def Reader(self, config, line, date, isLiveMode):
        """解析CSV行"""
        if not line.strip():
            return None

        parts = line.split(',')
        if len(parts) < 7:
            return None

        # 解析数据
        data = CustomCsvData()
        data.Symbol = config.Symbol
        data.Time = datetime.strptime(parts[0], "%Y-%m-%d")
        data.EndTime = data.Time + timedelta(days=1)
        data.Value = float(parts[4])  # Close price

        return data

15.3.2 Adding Custom Data

def Initialize(self):
    # 注册自定义数据类型
    self.AddData(CustomCsvData, "CUSTOM", Resolution.Daily)

    # 或指定数据时区
    self.AddData(
        CustomCsvData,
        "CUSTOM",
        Resolution.Daily,
        TimeZones.NewYork
    )

15.3.3 URL Data Sources

class UrlDataSource(PythonData):
    """从URL获取数据"""

    def GetSource(self, config, date, isLiveMode):
        """定义URL数据源"""
        return SubscriptionDataSource(
            "https://api.example.com/data.csv",
            SubscriptionTransportMedium.Rest,
            FileFormat.Csv
        )

15.4 Data Conversion Tools

15.4.1 Using ToolBox

Lean provides the ToolBox project for data conversion.

# 运行 ToolBox
cd ToolBox
dotnet run

15.4.2 Data Converters

Lean includes several built-in data converters:

Converter Purpose
AlgoSeekFuturesConverter AlgoSeek futures data conversion
KaikoDataConverter Kaiko cryptocurrency data conversion
CoarseUniverseGenerator Generate coarse universes
RandomDataGenerator Generate random test data

15.5 Data Quality Checks

15.5.1 Checking Data Integrity

def CheckDataQuality(self):
    """检查数据质量"""
    # 获取历史数据
    history = self.History(self.spy, TimeSpan.FromDays(30), Resolution.Daily)

    if history.empty:
        self.Error("No data available!")
        return False

    # 检查缺失值
    missing = history.isnull().sum()
    if missing > 0:
        self.Log(f"Missing {missing} data points")

    # 检查异常值
    close_prices = history['close']
    extreme_changes = close_prices.pct_change().abs() > 0.5

    if extreme_changes.any():
        extreme_count = extreme_changes.sum()
        self.Log(f"Found {extreme_count} extreme price changes (>50%)")

    # 检查数据连续性
    expected_days = 30  # 预期30天
    actual_days = len(history)

    if actual_days < expected_days * 0.8:
        self.Log(f"Data gap: expected {expected_days} days, got {actual_days}")

    return True

15.5.2 Data Warm-Up Strategies

def Initialize(self):
    self.spy = self.AddEquity("SPY", Resolution.Daily).Symbol

    # 方式1:使用 SetWarmUp
    self.SetWarmUp(TimeSpan.FromDays(60))

    # 方式2:手动预热指标
    self.sma = self.SMA(self.spy, 50)

    # 预热特定指标
    self.WarmUpIndicator(
        self.spy,
        self.sma,
        TimeSpan.FromDays(50)
    )

def OnData(self, data):
    # 指标立即可用
    if self.sma.IsReady:
        # ...

15.6 Complete Example: Custom Data Strategy

from AlgorithmImports import *

class CustomDataStrategy(QCAlgorithm):
    """
    使用自定义数据的策略
    """

    def Initialize(self):
        self.SetStartDate(2020, 1, 1)
        self.SetEndDate(2023, 12, 31)
        self.SetCash(100000)

        # 添加标准数据
        self.spy = self.AddEquity("SPY", Resolution.Daily).Symbol

        # 添加自定义数据(假设已有文件)
        # self.AddData(CustomCsvData, "CUSTOM", Resolution.Daily)

        # 指标
        self.sma_fast = self.SMA(self.spy, 20)
        self.sma_slow = self.SMA(self.spy, 50)

        self.SetWarmUp(TimeSpan.FromDays(60))

    def OnData(self, data):
        # 数据质量检查
        if not self.ValidateDataQuality(data):
            return

        # 正常逻辑
        if self.sma_fast.IsReady:
            if self.sma_fast.Current.Value > self.sma_slow.Current.Value:
                if not self.Portfolio[self.spy].IsLong:
                    self.SetHoldings(self.spy, 1)
            else:
                if self.Portfolio[self.spy].IsLong:
                    self.Liquidate(self.spy)

    def ValidateDataQuality(self, data):
        """验证数据质量"""
        # 检查数据可用性
        if not data.ContainsKey(self.spy):
            return False

        # 检查价格合理性
        price = data[self.spy].Price

        if price <= 0:
            self.Log(f"Invalid price: ${price}")
            return False

        # 检查数据是否过时(self.Time 与 LocalTime 均为引擎 .NET DateTime,
        # 差值为 .NET TimeSpan,需使用 TotalSeconds)
        data_time = self.Securities[self.spy].LocalTime
        time_diff = (self.Time - data_time).TotalSeconds

        if time_diff > 86400:  # 超过1天
            self.Log(f"Data stale: {time_diff} seconds old")
            return False

        return True

15.7 Data Management Best Practices

  1. Update data regularly: keep your data up to date
  2. Validate data quality: check for missing values and outliers
  3. Use an appropriate resolution: avoid unnecessary high-frequency data
  4. Local caching: reduce repeated downloads
  5. Back up data: regularly back up important data
  6. Monitor data sources: ensure data sources remain stable and available