Lesson 12: Regime Detection

Goal: Enable the system to know "what market we're in now" so it can choose the right strategy.

A Typical Scenario (Illustrative)

Note: The following is a synthetic example to illustrate common phenomena; numbers are illustrative and don't correspond to any specific institution/product.

In early 2021, a quant fund simultaneously ran two strategies: trend following and mean reversion. Both strategies performed excellently in their respective favorable markets.

The problem was they used fixed weights - 50% each.

Result:

• In trending markets, trend strategy gained 20%, mean reversion lost 15%, net gain 2.5%
• In ranging markets, mean reversion gained 15%, trend strategy lost 20%, net loss 2.5%
• For the whole year, the two canceled each other out, returns close to zero

Their competitor? Same two strategies, but added a Regime Detection module:

• In trending markets, trend strategy weight 80%, mean reversion 20%
• In ranging markets, reversed

Result: 18% annual return, Sharpe ratio 1.6.

What's the difference? Not the strategies themselves, but knowing when to use which strategy. This is the value of regime detection - it's the "router" of the multi-agent system.

12.1 What Is Market Regime

Three Basic States

The market is always in some "state" at any moment. While boundaries between states aren't clear, they can be roughly categorized:

Why Is Regime Detection So Hard?

Paper Exercise: Identify Historical States

Look at this data and judge each period's market state:

12.2 Four Methods for Regime Detection

Method Comparison

Method 1: Rules-Based (Most Practical)

Use combinations of technical indicator thresholds to judge state:

Paper Exercise: Design Your Rules

Given the following market data, use the rules above to judge state:

Method 2: Hidden Markov Model (HMM)

HMM assumes the market switches between several hidden states - we can only observe prices, not the true state.

HMM Core Outputs:

• State probabilities: Probability of being in each state now (e.g., Trending 60%, Ranging 30%, Crisis 10%)
• Transition probabilities: Probability of switching from one state to another

Paper Exercise: Understanding State Probabilities

HMM outputs current state probabilities as: Trending 55%, Ranging 35%, Crisis 10%

Key insight: State probabilities shouldn't be used directly as strategy weights - they need adjustment based on risk preference.

# Note: This is illustrative code showing basic HMM usage
# Requires hmmlearn: pip install hmmlearn

import numpy as np
from hmmlearn import hmm

class RegimeDetector:
    """HMM-based market regime detector"""

    def __init__(self, n_states: int = 3):
        self.n_states = n_states
        self.model = hmm.GaussianHMM(
            n_components=n_states,
            covariance_type="full",
            n_iter=100
        )

    def fit(self, features: np.ndarray):
        """
        Train HMM model

        features: shape (n_samples, n_features)
                  typically includes: returns, volatility, ADX, etc.
        """
        self.model.fit(features)
        return self

    def predict_proba(self, features: np.ndarray) -> np.ndarray:
        """
        Predict current state probability

        Returns: shape (n_states,) probability distribution
        """
        # Get posterior probabilities
        posteriors = self.model.predict_proba(features)
        return posteriors[-1]  # Latest time point probability

    def get_regime(self, features: np.ndarray, threshold: float = 0.5) -> str:
        """
        Get current state (with confidence threshold)
        """
        probs = self.predict_proba(features)
        max_prob = np.max(probs)
        max_state = np.argmax(probs)

        if max_prob < threshold:
            return "uncertain"

        state_names = ["trending", "mean_reverting", "crisis"]
        return state_names[max_state]

Method 3: Volatility Clustering (Practical Variant)

A simplified but practical approach is to only look at volatility state:

Why Is Volatility So Important?

• Volatility is the market's "temperature," quickly reflects market health
• Volatility has clustering effect (high volatility tends to follow high volatility)
• Volatility is more predictable than returns

12.3 Evaluating Regime Detection

Regime detection isn't the goal - the goal is improving strategy returns. So the evaluation standard isn't "accuracy" but "did it help the strategy make more money."

Evaluation Framework

Paper Exercise: Calculate Regime Detection Value

Scenario: Your Regime Detection module's past year performance:

Strategy Performance Comparison:

Questions:

1. What is the net value-add of Regime Detection?
2. If false switch rate drops to 10%, how would expected return change?
3. What to do if switching cost is too high?

The Cost of Switching Lag

Conclusion: Better to have some false positives than too much lag. Especially crisis detection must be fast.

12.4 Common Misconceptions

Misconception 1: Higher Regime Detection accuracy is always better

Not entirely true. The evaluation standard isn't "accuracy" but "did it help the strategy make more money." 80% accuracy but switching costs eating returns is worse than 60% accuracy with fewer switches.

Misconception 2: Trend/Ranging is a binary judgment

In reality, most of the time the market is in "gray zones." Forcing binary judgment leads to frequent false switches. Correct approach: use probability weighting (Trend 60%/Ranging 40%) or acknowledge "Transition period."

Misconception 3: Switch strategy immediately when state change is detected

Dangerous approach. Switching itself has costs (trading costs, slippage), frequent switching can lose all gains. Correct approach: confirmation delay (state persists N days) + gradual switching (50%->70%->90%).

Misconception 4: Crisis detection can "wait for confirmation"

Crisis detection must be fast. By the time you confirm, you may have lost 20%. Better to false positive on crisis (earn less) than miss it (lose big).

12.5 Handling Transition Periods

The hardest thing to handle in reality isn't clear trends or ranging, but transition periods - state is changing, but not yet certain.

Characteristics of Transition Periods

Transition Period Strategies

Recommendation: For most investors, Strategy 3 (Risk Priority) is the safest choice. Transition periods are often the highest risk periods.

12.5 Multi-Agent Perspective

In multi-agent architecture, Regime Detection is Meta Agent's core capability.

Architecture Design

Regime Agent Responsibilities

Collaboration with Other Agents

Acceptance Criteria

After completing this lesson, use these standards to verify learning:

Comprehensive Exercise

Design Your Regime Detection System:

1. Choose detection method (Rules-based/HMM/Other) and explain why
2. Define input features and thresholds
3. Design transition period handling strategy
4. Set switching confirmation conditions (prevent over-switching)
5. Define evaluation metrics

Lesson Deliverables

After completing this lesson, you will have:

1. Regime Detection design framework - Complete thinking from method selection to evaluation metrics
2. Practical rules-based template - Directly usable ADX/volatility/correlation judgment rules
3. Method to evaluate Regime Detection value - Net value-add = Return improvement - Switching cost
4. Transition period handling strategies - Three approaches and their use cases

Lesson Summary

• Understand the concept of market regime and its impact on strategy selection
• Master four Regime Detection methods: rules-based, statistical, ML, HMM
• Understand Regime Detection evaluation isn't "accuracy" but "economic value"
• Master transition period handling strategies

Further Reading

• Lesson 05: Classic Strategy Paradigms - Strategies for different states
• Lesson 11: Why Multi-Agent - Role of Regime Detection in multi-agent
• Background: Famous Quant Disasters - Consequences of failing to detect crisis states

Next Lesson Preview

Lesson 13: Regime Misjudgment and Systemic Collapse

Regime Detection is the core of multi-agent systems, but it makes mistakes - misjudgment, lag, boundary oscillation. Next lesson we dive deep into five failure modes of Regime Detection, and how to design degradation strategies to prevent systemic collapse.