title: Chapter 15: Data Management¶
Chapter 15: Data Management¶
This chapter covers the Lean data format, data directory structure, data downloading, and custom data import.
15.1 Data Directory Structure¶
15.1.1 Full Directory Tree¶
Data/
├── equity/ # 股票数据:equity/{market}/{resolution}/{symbol}.zip
│ └── usa/
│ ├── daily/
│ │ └── spy.zip # SPY 日线数据
│ ├── minute/ # 分钟数据
│ ├── hour/ # 小时数据
│ ├── tick/ # 逐笔数据
│ ├── map_files/ # 映射文件:data/equity/usa/map_files/{SYMBOL}.csv
│ │ └── spy.csv
│ └── factor_files/ # 复权因子:data/equity/usa/factor_files/{SYMBOL}.csv
│ └── spy.csv
├── forex/ # 外汇数据:forex/{market}/{resolution}
│ └── fxcm/
│ ├── daily/
│ └── minute/
├── crypto/ # 加密货币:crypto/{market}/{resolution}
│ └── binance/
│ ├── daily/
│ └── minute/
├── future/ # 期货数据:future/{market}/{resolution}
│ ├── cme/ # 芝加哥商品交易所
│ │ ├── daily/
│ │ └── minute/
│ ├── cbot/ # 芝加哥期货交易所
│ │ └── daily/
│ └── ...
├── option/ # 期权数据:option/{market}/{resolution}/{underlying}/
│ └── usa/
│ ├── daily/
│ │ └── spy/
│ │ └── ...
│ └── minute/
└── cfd/ # CFD数据:cfd/{market}/{resolution}
└── fxcm/
└── daily/
15.1.2 Data File Format¶
Equity Data Format (zip + csv)¶
Inside the zip is a headerless CSV; each row has exactly 6 fields in the order [time, Open, High, Low, Close, Volume]:
20130102 00:00,1440000,1449000,1431000,1440000,128499600
...
Field descriptions:
- Time: for daily/hour data, in
yyyyMMdd HH:mmformat (e.g.,20130102 00:00); for minute/tick data, an integer number of milliseconds since midnight (e.g.,57000000means 15:50:00.000). - Open/High/Low/Close: stored as integers; the actual price = value / 10000. For example,
1440000corresponds to 144.0000 USD. - Volume: number of shares traded (integer).
- Note: there are no separate date/time columns and no floating-point prices; equity data has no openinterest column.
Map Files and Factor Files¶
Map files are located at data/equity/usa/map_files/{SYMBOL}.csv (not under the daily/ directory). Each row has the format yyyymmdd,mapped symbol,P, where the date is the effective date for that symbol; the last line of the file is usually 20501231:
# data/equity/usa/map_files/spy.csv
19980102,SPY,P
...
20501231,SPY,P
Factor files are located at data/equity/usa/factor_files/{SYMBOL}.csv, with columns date,priceFactor,splitFactor representing the price factor and split factor respectively; the engine uses them to restore historical prices to the prices under the selected data normalization mode.
15.2 Downloading Data with the Lean CLI¶
15.2.1 Downloading Datasets¶
Data files are placed in the data/ directory (from QuantConnect Datasets); during local backtests the engine reads them following the directory structure in Section 15.1.
# 下载特定数据集
lean data download --dataset "QuantConnect - USA Equity Price Data"
# 具体下载参数因数据集而异,围绕 --dataset 指定数据集 ID,
# 其余选项详见 lean data download --help
lean data download --dataset "<数据集ID>"
# 下载期权数据
lean data download --dataset "QuantConnect - USA Option Price Data"
15.2.2 Data Providers¶
| Provider | Type | Description |
|---|---|---|
| QuantConnect | Comprehensive data | Cloud data, requires a subscription |
| Local | Local files | Read from local files |
| Interactive Brokers | Real-time data | Used for live trading |
| CoinAPI | Crypto | Real-time cryptocurrency data |
15.3 Custom Data Import¶
15.3.1 CSV Data Import¶
class CustomCsvData(PythonData):
"""
自定义CSV数据导入
CSV格式要求:
date,open,high,low,close,volume
2023-01-01,100.00,102.00,99.50,101.00,1000000
"""
def GetSource(self, config, date, isLiveMode):
"""定义数据源"""
return SubscriptionDataSource(
"data/my_custom_data.csv",
SubscriptionTransportMedium.LocalFile,
FileFormat.Csv
)
def Reader(self, config, line, date, isLiveMode):
"""解析CSV行"""
if not line.strip():
return None
parts = line.split(',')
if len(parts) < 7:
return None
# 解析数据
data = CustomCsvData()
data.Symbol = config.Symbol
data.Time = datetime.strptime(parts[0], "%Y-%m-%d")
data.EndTime = data.Time + timedelta(days=1)
data.Value = float(parts[4]) # Close price
return data
15.3.2 Adding Custom Data¶
def Initialize(self):
# 注册自定义数据类型
self.AddData(CustomCsvData, "CUSTOM", Resolution.Daily)
# 或指定数据时区
self.AddData(
CustomCsvData,
"CUSTOM",
Resolution.Daily,
TimeZones.NewYork
)
15.3.3 URL Data Sources¶
class UrlDataSource(PythonData):
"""从URL获取数据"""
def GetSource(self, config, date, isLiveMode):
"""定义URL数据源"""
return SubscriptionDataSource(
"https://api.example.com/data.csv",
SubscriptionTransportMedium.Rest,
FileFormat.Csv
)
15.4 Data Conversion Tools¶
15.4.1 Using ToolBox¶
Lean provides the ToolBox project for data conversion.
# 运行 ToolBox
cd ToolBox
dotnet run
15.4.2 Data Converters¶
Lean includes several built-in data converters:
| Converter | Purpose |
|---|---|
| AlgoSeekFuturesConverter | AlgoSeek futures data conversion |
| KaikoDataConverter | Kaiko cryptocurrency data conversion |
| CoarseUniverseGenerator | Generate coarse universes |
| RandomDataGenerator | Generate random test data |
15.5 Data Quality Checks¶
15.5.1 Checking Data Integrity¶
def CheckDataQuality(self):
"""检查数据质量"""
# 获取历史数据
history = self.History(self.spy, TimeSpan.FromDays(30), Resolution.Daily)
if history.empty:
self.Error("No data available!")
return False
# 检查缺失值
missing = history.isnull().sum()
if missing > 0:
self.Log(f"Missing {missing} data points")
# 检查异常值
close_prices = history['close']
extreme_changes = close_prices.pct_change().abs() > 0.5
if extreme_changes.any():
extreme_count = extreme_changes.sum()
self.Log(f"Found {extreme_count} extreme price changes (>50%)")
# 检查数据连续性
expected_days = 30 # 预期30天
actual_days = len(history)
if actual_days < expected_days * 0.8:
self.Log(f"Data gap: expected {expected_days} days, got {actual_days}")
return True
15.5.2 Data Warm-Up Strategies¶
def Initialize(self):
self.spy = self.AddEquity("SPY", Resolution.Daily).Symbol
# 方式1:使用 SetWarmUp
self.SetWarmUp(TimeSpan.FromDays(60))
# 方式2:手动预热指标
self.sma = self.SMA(self.spy, 50)
# 预热特定指标
self.WarmUpIndicator(
self.spy,
self.sma,
TimeSpan.FromDays(50)
)
def OnData(self, data):
# 指标立即可用
if self.sma.IsReady:
# ...
15.6 Complete Example: Custom Data Strategy¶
from AlgorithmImports import *
class CustomDataStrategy(QCAlgorithm):
"""
使用自定义数据的策略
"""
def Initialize(self):
self.SetStartDate(2020, 1, 1)
self.SetEndDate(2023, 12, 31)
self.SetCash(100000)
# 添加标准数据
self.spy = self.AddEquity("SPY", Resolution.Daily).Symbol
# 添加自定义数据(假设已有文件)
# self.AddData(CustomCsvData, "CUSTOM", Resolution.Daily)
# 指标
self.sma_fast = self.SMA(self.spy, 20)
self.sma_slow = self.SMA(self.spy, 50)
self.SetWarmUp(TimeSpan.FromDays(60))
def OnData(self, data):
# 数据质量检查
if not self.ValidateDataQuality(data):
return
# 正常逻辑
if self.sma_fast.IsReady:
if self.sma_fast.Current.Value > self.sma_slow.Current.Value:
if not self.Portfolio[self.spy].IsLong:
self.SetHoldings(self.spy, 1)
else:
if self.Portfolio[self.spy].IsLong:
self.Liquidate(self.spy)
def ValidateDataQuality(self, data):
"""验证数据质量"""
# 检查数据可用性
if not data.ContainsKey(self.spy):
return False
# 检查价格合理性
price = data[self.spy].Price
if price <= 0:
self.Log(f"Invalid price: ${price}")
return False
# 检查数据是否过时(self.Time 与 LocalTime 均为引擎 .NET DateTime,
# 差值为 .NET TimeSpan,需使用 TotalSeconds)
data_time = self.Securities[self.spy].LocalTime
time_diff = (self.Time - data_time).TotalSeconds
if time_diff > 86400: # 超过1天
self.Log(f"Data stale: {time_diff} seconds old")
return False
return True
15.7 Data Management Best Practices¶
- Update data regularly: keep your data up to date
- Validate data quality: check for missing values and outliers
- Use an appropriate resolution: avoid unnecessary high-frequency data
- Local caching: reduce repeated downloads
- Back up data: regularly back up important data
- Monitor data sources: ensure data sources remain stable and available