Lesson 17: Online Learning and Strategy Evolution

Goal: Understand why static models inevitably decay, and master the core mechanisms for continuous strategy evolution.

A Typical Scenario (Illustrative)

Note: The following is a synthetic example to illustrate common phenomena; numbers are illustrative and don't correspond to any specific individual/account.

In 2018, a quant trader developed a Random Forest-based stock selection model for A-shares. Backtest showed 35% annual return, 1.8 Sharpe, excellent across all metrics.

For the first 6 months, the model performed stably, matching the backtest. Then, performance began to slowly decline. Not a sudden failure, but like a frog in slowly heating water - each month earning a bit less, drawing down a bit more.

12 months later, annualized returns dropped from 35% to 8%, barely beating the index.

He checked the code, no bugs. Checked the data, no errors. The model was running exactly as designed, just no longer making money.

What went wrong? The market changed, but the model didn't. In early 2018, retail investors dominated, momentum factors worked. By late 2018, institutional money entered, momentum factors were arbitraged away. The patterns the model learned were patterns of that specific period, not eternal truths.

This is concept drift - the data distribution the model learned from no longer matches the data distribution it's trading in. The model isn't wrong, the world changed.

17.1 Why Static Models Inevitably Decay

Market Non-Stationarity

Financial markets have a fundamental characteristic: non-stationarity. This means:

1. Statistical properties change: Mean, variance, correlations are not constants
2. Patterns expire: Effective alpha gets discovered, arbitraged, disappears
3. Participants adapt: Your strategy profits -> Others copy -> Alpha decays

Paper Exercise: Calculate Alpha Decay

Scenario: Your factor IC = 0.05, assume market efficiency improvement causes IC to decay 5% per month.

Calculation:

Month    IC        Expected Annual (IC x sqrt(252) x sigma, assuming sigma=20%)
0      0.050      15.8%
6      0.050 x 0.95^6 = 0.037      11.7%
12     0.050 x 0.95^12 = 0.027      8.5%
18     0.050 x 0.95^18 = 0.020      6.3%
24     0.050 x 0.95^24 = 0.015      4.7%

Conclusion: Even with only 5% monthly decay, after 2 years your strategy returns are only 30% of the original.

Self-check: Calculate your current strategy's IC, assume 3%, 5%, 10% monthly decay, how long until it drops to unacceptable levels?

Why "Periodic Retraining" Isn't Enough

Many people's solution is "retrain the model every month." This is better than static models, but has three problems:

Correct approach: Not "relearn," but "continuously adapt" - gradually adjust to new environments while retaining historical knowledge.

17.2 Core Mechanisms of Online Learning

Batch vs Online Learning

Three Modes of Online Learning

Paper Exercise: Choosing Forgetting Factor

Scenario: Your strategy is tested in two market environments.

Formula: Effective lookback days ~ ln(0.1) / ln(lambda)

Self-check: Based on your strategy's market environment, choose appropriate lambda value. What happens if you choose wrong?

• Lambda too large (forget too slowly): Adapt too late, lose in new environment
• Lambda too small (forget too fast): Overfit recent noise, poor stability

class ExponentialMovingModel:
    """Online learning model with exponential forgetting"""

    def __init__(self, decay_factor: float = 0.95):
        self.lambda_ = decay_factor
        self.weights = None
        self.cumulative_weight = 0

    def update(self, X: np.ndarray, y: float, learning_rate: float = 0.01):
        """Incremental update"""
        if self.weights is None:
            self.weights = np.zeros(X.shape[0])

        # Predict
        pred = np.dot(self.weights, X)
        error = y - pred

        # Update weights (gradient descent with forgetting)
        self.weights = self.lambda_ * self.weights + learning_rate * error * X
        self.cumulative_weight = self.lambda_ * self.cumulative_weight + 1

        return pred, error

    def get_effective_lookback(self, threshold: float = 0.1) -> int:
        """Calculate effective lookback days"""
        return int(np.log(threshold) / np.log(self.lambda_))

17.3 Four Levels of Strategy Evolution

Online learning is only part of strategy evolution. A complete evolution system has four levels:

Level 1: Signal Adaptation (Fastest)

Signal adaptation is the fastest evolution level, doesn't require retraining models.

Example: Dynamic Threshold Adjustment

Static threshold: Buy if signal > 0.5

Dynamic threshold:
- Calculate past N days signal distribution
- Threshold = Signal mean + k x Signal std
- High market volatility, threshold auto-increases, reduce trades
- Low market volatility, threshold auto-decreases, increase trades

Paper Exercise:

Level 2: Model Evolution (Medium Speed)

Model evolution involves parameter updates and feature engineering.

Key Mechanism: Feature Importance Monitoring

+-------------------------------------------------+
|           Feature Importance Change Monitoring   |
+----------------+-----------+-----------+---------+
| Feature        | 6 months ago | Now     | Trend  |
+----------------+-----------+-----------+---------+
| Momentum_20D   | 0.25      | 0.08      | Decaying |
| Volatility_30D | 0.15      | 0.22      | Strengthening |
| Volume_Ratio   | 0.18      | 0.05      | Failed |
| Earnings_Surprise | 0.12   | 0.18      | Effective |
+----------------+-----------+-----------+---------+

-> Decision: Lower momentum weight, remove volume ratio, increase volatility weight

Evolution Trigger Conditions:

Level 3: Strategy Evolution (Slower)

Strategy evolution involves reconfiguring the strategy portfolio.

Strategy Lifecycle Management:

                    Incubation      Validation      Maturity        Decay
Return/Risk    ................ ............ ............... .........
               Paper trading    Small live      Main strategy    Phase out
               3-6 months       3-6 months      Indefinite       Indefinite

Entry criteria:     Backtest pass    Sharpe>1.0      Sharpe>1.5       Sharpe<0.8
                   Quality Gate   Live DD<15%   6 months positive  3 months losing

Capital allocation:  0%           5-10%           20-40%          Phase to zero

Dynamic Weights for Strategy Portfolio:

Paper Exercise: Strategy Weight Allocation

Calculation process:

• Inverse volatility: 1/5 = 0.20, 1/8 = 0.125, 1/8 = 0.125, total = 0.45
  • A: 0.20/0.45 = 0.44, B: 0.125/0.45 = 0.28, C: 0.28
• Sharpe proportional: total = 1.6 + 0.75 + 1.25 = 3.6
  • A: 1.6/3.6 = 0.44, B: 0.75/3.6 = 0.21, C: 1.25/3.6 = 0.35

Level 4: Architecture Evolution (Slowest)

Architecture evolution is the most fundamental change, usually requires human intervention.

Typical Architecture Evolution Path:

Stage 1: Single Strategy
+-- One simple trend following strategy

Stage 2: Strategy Portfolio
+-- Trend + Mean Reversion + Momentum, fixed weights

Stage 3: Dynamic Portfolio
+-- Dynamically adjust strategy weights based on market state

Stage 4: Multi-Agent
+-- Expert Agents + Meta Agent + Risk Agent

Stage 5: Self-Evolving System
+-- System that can automatically discover, test, and deploy new strategies

Architecture Evolution Trigger Conditions:

17.4 Key Technologies for Implementing Online Learning

1. Concept Drift Detection

Before updating models, confirm whether drift has actually occurred.

Common Detection Methods:

Paper Exercise: Simple Drift Detection

Monitor metric: Model prediction sliding accuracy

Time    Accuracy    5-Day Mean    Alert? (Mean<50%)
D1      55%       55%           No
D2      52%       53.5%         No
D3      48%       51.7%         No
D4      45%       50.0%         No
D5      42%       48.4%         Yes <- Trigger drift detection
D6      40%       45.4%         Yes

Trigger model update when drift is detected, not blind periodic updates.

2. Preventing Catastrophic Forgetting

One risk of online learning is "catastrophic forgetting" - forgetting old knowledge when learning new.

Solutions:

Practical recommendations:

1. Retain 10-20% of historical representative samples
2. Each update, mix new data + historical samples
3. Monitor performance on historical test set to ensure no forgetting

3. Update Timing and Frequency

Too frequent updates:

• Overfit recent noise
• Waste compute resources
• Unstable models

Too infrequent updates:

• Adaptation lag
• Miss market changes
• Accumulated errors

Recommended Strategy:

class AdaptiveUpdateScheduler:
    """Adaptive update scheduler"""

    def __init__(self, min_interval: int = 5, max_interval: int = 20):
        self.min_interval = min_interval  # Minimum update interval (days)
        self.max_interval = max_interval  # Maximum update interval (days)
        self.last_update = 0
        self.drift_detector = DDM()  # Drift detector

    def should_update(self, day: int, recent_errors: list) -> tuple[bool, str]:
        """Decide whether model should be updated"""
        days_since_update = day - self.last_update

        # Check drift
        drift_detected = self.drift_detector.detect(recent_errors)

        # Decision logic
        if days_since_update >= self.max_interval:
            return True, "Reached max interval, force update"

        if drift_detected and days_since_update >= self.min_interval:
            return True, "Drift detected, trigger update"

        return False, "No update needed"

    def record_update(self, day: int):
        self.last_update = day
        self.drift_detector.reset()

17.5 Engineering Implementation of Online Learning

1. Sliding Window Retraining Implementation

from datetime import datetime, timedelta
from typing import Optional, Tuple
import numpy as np
import pandas as pd
from sklearn.ensemble import RandomForestClassifier

class SlidingWindowRetrainer:
    """Sliding window model retrainer"""

    def __init__(self,
                 window_size_days: int = 252,
                 min_samples: int = 200,
                 retrain_interval_days: int = 20):
        """
        window_size_days: Training window size (days)
        min_samples: Minimum training samples
        retrain_interval_days: Retrain interval (days)
        """
        self.window_size = window_size_days
        self.min_samples = min_samples
        self.retrain_interval = retrain_interval_days

        self.model = None
        self.last_train_date = None
        self.training_history = []

    def should_retrain(self, current_date: datetime) -> Tuple[bool, str]:
        """Decide whether retraining is needed"""
        if self.model is None:
            return True, "Initial training"

        days_since_train = (current_date - self.last_train_date).days

        if days_since_train >= self.retrain_interval:
            return True, f"{days_since_train} days since last training"

        return False, "No retrain needed"

    def train(self, data: pd.DataFrame,
              feature_cols: list,
              target_col: str,
              current_date: datetime) -> dict:
        """
        Train model using sliding window

        data: Must contain 'date' column and feature columns
        """
        # Calculate window boundaries
        window_end = current_date
        window_start = current_date - timedelta(days=self.window_size)

        # Filter data within window
        mask = (data['date'] >= window_start) & (data['date'] < window_end)
        train_data = data[mask]

        if len(train_data) < self.min_samples:
            return {
                'success': False,
                'reason': f'Insufficient samples: {len(train_data)} < {self.min_samples}'
            }

        X = train_data[feature_cols].values
        y = train_data[target_col].values

        # Train model
        self.model = RandomForestClassifier(
            n_estimators=100,
            max_depth=5,
            min_samples_leaf=20,
            random_state=42
        )
        self.model.fit(X, y)

        # Record training
        self.last_train_date = current_date

        train_info = {
            'success': True,
            'date': current_date,
            'window': (window_start, window_end),
            'n_samples': len(train_data),
            'train_accuracy': self.model.score(X, y)
        }
        self.training_history.append(train_info)

        return train_info

    def predict(self, X: np.ndarray) -> np.ndarray:
        """Predict"""
        if self.model is None:
            raise ValueError("Model not yet trained")
        return self.model.predict_proba(X)[:, 1]

2. Drift Detection Metrics Implementation

Population Stability Index (PSI)

PSI is a standard metric for detecting feature distribution drift.

def calculate_psi(expected: np.ndarray,
                  actual: np.ndarray,
                  n_bins: int = 10) -> float:
    """
    Calculate Population Stability Index

    PSI < 0.1: No significant drift
    0.1 <= PSI < 0.25: Moderate drift, needs attention
    PSI >= 0.25: Significant drift, needs action
    """
    # Create bins
    breakpoints = np.percentile(expected,
                                np.linspace(0, 100, n_bins + 1))
    breakpoints[0] = -np.inf
    breakpoints[-1] = np.inf

    # Calculate bin proportions
    expected_counts = np.histogram(expected, bins=breakpoints)[0]
    actual_counts = np.histogram(actual, bins=breakpoints)[0]

    # Convert to proportions (avoid division by zero)
    expected_pct = (expected_counts + 1) / (len(expected) + n_bins)
    actual_pct = (actual_counts + 1) / (len(actual) + n_bins)

    # Calculate PSI
    psi = np.sum((actual_pct - expected_pct) *
                 np.log(actual_pct / expected_pct))

    return psi


class FeatureDriftMonitor:
    """Feature drift monitor"""

    def __init__(self, psi_threshold: float = 0.2):
        self.baseline_distributions = {}
        self.psi_threshold = psi_threshold
        self.drift_history = []

    def set_baseline(self, feature_name: str, values: np.ndarray):
        """Set baseline distribution"""
        self.baseline_distributions[feature_name] = values.copy()

    def check_drift(self, feature_name: str,
                    current_values: np.ndarray,
                    date: datetime) -> dict:
        """Check single feature drift"""
        if feature_name not in self.baseline_distributions:
            raise ValueError(f"Baseline not set: {feature_name}")

        baseline = self.baseline_distributions[feature_name]
        psi = calculate_psi(baseline, current_values)

        result = {
            'feature': feature_name,
            'date': date,
            'psi': psi,
            'drift_detected': psi >= self.psi_threshold,
            'severity': 'high' if psi >= 0.25 else
                       ('medium' if psi >= 0.1 else 'low')
        }

        self.drift_history.append(result)
        return result

    def check_all_features(self, current_data: pd.DataFrame,
                           date: datetime) -> dict:
        """Check all feature drift"""
        results = {}
        drifted_features = []

        for feature in self.baseline_distributions:
            if feature in current_data.columns:
                result = self.check_drift(
                    feature,
                    current_data[feature].values,
                    date
                )
                results[feature] = result
                if result['drift_detected']:
                    drifted_features.append(feature)

        return {
            'date': date,
            'features_checked': len(results),
            'features_drifted': len(drifted_features),
            'drifted_features': drifted_features,
            'details': results,
            'action_required': len(drifted_features) > 0
        }

3. Retrain vs Pause Decision Framework

class RetrainDecisionEngine:
    """Retrain decision engine"""

    def __init__(self,
                 psi_threshold: float = 0.2,
                 performance_drop_threshold: float = 0.3,
                 min_confidence_for_retrain: float = 0.6):
        self.psi_threshold = psi_threshold
        self.perf_threshold = performance_drop_threshold
        self.min_confidence = min_confidence_for_retrain

    def decide(self,
               drift_report: dict,
               performance_report: dict,
               data_quality_report: dict) -> dict:
        """
        Decide next action

        Returns: 'retrain', 'pause', 'continue', 'investigate'
        """
        # Collect signals
        drift_detected = drift_report.get('action_required', False)
        perf_drop = performance_report.get('ic_change', 0)
        data_quality_ok = data_quality_report.get('quality_score', 1) > 0.8
        sample_size_ok = data_quality_report.get('n_samples', 0) > 200

        # Decision matrix
        #
        #                    Performance Drop?
        #                    Yes        No
        #   Drift?   Yes    [Retrain]  [Observe]
        #            No     [Pause]    [Continue]
        #

        if drift_detected:
            if perf_drop > self.perf_threshold:
                # Drift + Performance drop: Need retrain
                if data_quality_ok and sample_size_ok:
                    return {
                        'action': 'retrain',
                        'confidence': 0.9,
                        'reason': 'Drift detected and performance dropped, data quality sufficient',
                        'urgency': 'high'
                    }
                else:
                    return {
                        'action': 'investigate',
                        'confidence': 0.5,
                        'reason': 'Need retrain but data quality insufficient, human intervention needed',
                        'urgency': 'high'
                    }
            else:
                # Drift but performance OK: Continue observing
                return {
                    'action': 'continue',
                    'confidence': 0.6,
                    'reason': 'Drift detected but performance not yet affected, continue monitoring',
                    'urgency': 'low',
                    'next_check_days': 3
                }
        else:
            if perf_drop > self.perf_threshold:
                # No drift but performance drop: May be overfitting or market change
                return {
                    'action': 'pause',
                    'confidence': 0.7,
                    'reason': 'Performance dropped but no obvious drift, recommend pause and investigate',
                    'urgency': 'medium'
                }
            else:
                # No drift and performance normal: Continue running
                return {
                    'action': 'continue',
                    'confidence': 0.95,
                    'reason': 'All normal',
                    'urgency': 'none'
                }

4. Paper Exercise: When to Retrain vs Pause

Key principles:

1. Drift + Performance drop = Retrain (Market changed, model needs to adapt)
2. No drift + Performance drop = Pause (May be other issues, don't blindly retrain)
3. Drift + Performance normal = Observe (May be temporary fluctuation)
4. Poor data quality = Fix data first (Otherwise retraining is futile)

17.6 Evolution from Multi-Agent Perspective

In multi-agent architecture, evolution isn't just about updating individual models, but coordinating between Agents.

Evolution Architecture

Evolution Agent Responsibilities

Agent-Level A/B Testing

Coordinated Update Challenges

Acceptance Criteria

After completing this lesson, use these standards to verify learning:

Comprehensive Exercise

Scenario: You have an A-shares stock selection strategy with the following past performance:

Questions:

1. What type of decay is this? (Sudden/Gradual/Cyclical)
2. Calculate IC monthly decay rate
3. When should model update be triggered?
4. What update strategy do you recommend?

Lesson Deliverables

After completing this lesson, you will have:

1. Quantitative understanding of concept drift - Can calculate alpha decay, estimate strategy lifespan
2. Online learning implementation framework - Three modes (sliding window, incremental update, exponential forgetting) and their use cases
3. Four-level evolution system - Complete path from signal adaptation to architecture evolution
4. Multi-agent evolution mechanism - Evolution Agent design thinking

Lesson Summary

• Understand inevitability of static model decay: Concept drift is an inherent market characteristic
• Master three modes of online learning and their use cases
• Understand four levels of strategy evolution and their trigger conditions
• Understand coordinated update challenges in multi-agent architecture

Further Reading

• Lesson 11: Why Multi-Agent - Understand basics of multi-agent collaboration
• Lesson 14: LLM Applications in Quant - LLM as part of evolution system
• Background: Statistical Traps of Sharpe Ratio - Statistical pitfalls in model evaluation
• Lesson 21: Project Implementation - Put this lesson's theory into practice

Next Lesson Preview

Lesson 18: Trading Cost Modeling and Tradability

Theory ends here. Next lesson we enter the Production and Practice stage - first learning how to model trading costs and evaluate the conversion rate from paper returns to real returns. After all, a strategy with high Gross Alpha but negative Net Alpha will forever remain in backtests.