The first version of our data pipeline required managing seven separate WebSocket connections, four vendor SDKs, and two polling loops. At 9:30 AM ET on a typical trading day, we were burning CPU on connection management while missing half-second windows of opportunity.

The maintenance overhead was staggering. Every time a vendor changed their heartbeat interval, updated their authentication headers, or—worse—deprecated an endpoint, we spent sprint cycles patching adapters. The problem was not data acquisition. The problem was the infrastructure tax of connecting to heterogeneous markets.

This article dissects the architectural approach TickDB uses to consolidate multiple market feeds into a single WebSocket connection. We examine the protocol adaptation layer, the unified data model that normalizes disparate exchange formats, and the timezone standardization logic that makes cross-market analysis tractable.

The Fragmentation Problem

Each major market operates on fundamentally different protocols and data conventions.

Market Primary Protocol Auth Method Heartbeat Interval Timestamp Convention
US Equities Proprietary / WebSocket API key header 30s US Eastern Time
HK Equities HKEX gateway protocol Session token 15s Hong Kong Time
Crypto Binance/Kraken WebSocket API key URL param 3 min UTC
Futures CQG/Rithmic Certificate + user 20s Exchange local time

Building a unified gateway is not a matter of "translating JSON." It requires handling protocol handshakes, subscription models, authentication flows, and timestamp normalization across all of these environments—while maintaining sub-second latency.

Architecture Overview: Three-Layer Design

TickDB's unified gateway operates as a three-layer architecture:

┌─────────────────────────────────────────────────────────────┐
│                    Unified WebSocket Gateway                │
│                  (Single connection to client)              │
└────────────────────────────┬────────────────────────────────┘
                             │
┌────────────────────────────▼────────────────────────────────┐
│                  Protocol Adaptation Layer                   │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────┐    │
│  │ US Feed  │  │ HK Feed  │  │  Crypto  │  │ Futures  │    │
│  │ Adapter  │  │ Adapter  │  │ Adapter  │  │ Adapter  │    │
│  └──────────┘  └──────────┘  └──────────┘  └──────────┘    │
└────────────────────────────┬────────────────────────────────┘
                             │
┌────────────────────────────▼────────────────────────────────┐
│                    Exchange Connectors                      │
│    NYSE    │    HKEX    │   Binance   │   CME   │  ...     │
└─────────────────────────────────────────────────────────────┘

Layer 1: Exchange Connectors

Each exchange connector maintains a dedicated, optimized connection to its source. These connectors are built on exchange-specific SDKs where necessary, or raw WebSocket clients where the protocol is open.

Connectors are responsible for:

  • Raw message ingestion
  • Initial parsing and validation
  • Timestamp extraction (before normalization)
  • Error detection and reconnection to the source exchange

Layer 2: Protocol Adaptation Layer

This is the core innovation. Each adapter translates between the exchange-specific protocol and TickDB's internal message format. The adaptation layer handles:

  • Subscription model normalization: Exchanges use different subscription semantics (subscribe/unsubscribe frames, batch subscription formats, symbol formats like AAPL.US vs 00700.HK vs BTC-USD)
  • Authentication translation: Different auth methods are normalized to a single internal session model
  • Heartbeat protocol mapping: Ping/pong intervals are normalized to a unified keepalive schedule
  • Error code translation: Exchange-specific error codes map to TickDB's internal error taxonomy

Layer 3: Unified Gateway

The gateway exposes a single WebSocket interface to clients. It routes messages based on symbol routing rules, manages client sessions, and applies the unified data model to all outbound messages.

Protocol Adaptation: How Symbol Translation Works

The most visible challenge in multi-market data is symbol format divergence. Consider the same asset across three markets:

Asset US Symbol Format HK Symbol Format Crypto Symbol Format
Apple AAPL.US — —
Tencent — 0700.HK —
Bitcoin — — BTC-USD or BTCUSDT

TickDB's symbol registry normalizes all incoming symbols to a canonical format before they reach the gateway. The routing table below shows how subscriptions are resolved:

Canonical Symbol US Resolution HK Resolution Crypto Resolution
AAPL.US AAPL on NYSE feed Not applicable Not applicable
0700.HK Not applicable 0700 on HKEX feed Not applicable
BTC-USD Not applicable Not applicable BTCUSDT on Binance / BTC-USD on Kraken

When a client subscribes to AAPL.US, the gateway routes that subscription to the US equity adapter. When a client subscribes to BTC-USD, the gateway routes to the crypto adapter—and the adapter handles the translation to the specific exchange's expected symbol format.

This routing is transparent to the client. The client sends one subscription frame; the gateway handles distribution internally.

Unified Data Model

The unified data model is the contract between the gateway and clients. Regardless of source exchange, all order book, trade, and OHLCV data follows the same schema.

Kline (OHLCV) Normalization

For candle data, the unified model follows a consistent structure:

{
  "channel": "kline",
  "symbol": "AAPL.US",
  "interval": "1m",
  "data": {
    "open": 182.45,
    "high": 183.20,
    "low": 182.10,
    "close": 183.05,
    "volume": 45230000,
    "timestamp": 1747844400000
  }
}

The timestamp field is always epoch milliseconds in UTC. This is non-negotiable—the gateway never exposes exchange-native timestamps to the client.

Order Book Normalization

The depth (order book) channel follows a similar normalization:

{
  "channel": "depth",
  "symbol": "0700.HK",
  "data": {
    "bids": [[435.20, 120000], [435.00, 85000]],
    "asks": [[435.40, 95000], [435.60, 110000]],
    "timestamp": 1747844400500
  }
}

Each level is an [price, size] tuple. The gateway normalizes price precision and lot sizes across exchanges—US equities use two decimal places, while HK equities may use different tick sizes depending on price range.

Timezone Standardization: Why UTC Is the Only Viable Choice

Cross-market analysis is impossible without a unified time reference. Each market operates in its local timezone:

Market Exchange Timezone DST Behavior
US Equities America/New_York Observed
HK Equities Asia/Hong_Kong Not observed
Crypto UTC Not applicable

TickDB standardizes all timestamps to UTC epoch milliseconds at the adapter layer, before data reaches the unified gateway. This has three practical benefits:

  1. Cross-market alignment: A 15-minute candle for AAPL.US ending at 1747844400000 aligns precisely with a candle for 0700.HK ending at the same timestamp.
  2. No DST edge cases: UTC has no daylight saving transitions. A "9:30 AM ET" on March 10, 2025 is a different wall-clock time than on November 3, 2025 due to DST. The UTC timestamp 1747844400000 refers to exactly one instant, unambiguously.
  3. Simplified backtesting: When replaying historical data across multiple markets, UTC timestamps eliminate the need to track which markets were in DST on any given date.

The one caveat: for display purposes, clients are responsible for converting UTC to their target timezone. The gateway provides the raw epoch; display is the client's concern.

Implementation: Unified WebSocket Subscription

The following code demonstrates how a client connects to the unified gateway and subscribes to cross-market data. Note the production-grade implementation with heartbeat, reconnection, and rate-limit handling.

import os
import json
import time
import random
import threading
import websocket
from datetime import datetime, timezone


class UnifiedMarketDataClient:
    """
    Unified WebSocket client for TickDB multi-market data.
    Connects once, subscribes across US equities, HK equities, and crypto.
    """
    
    def __init__(self, api_key: str):
        self.api_key = api_key
        self.ws = None
        self.base_url = "wss://api.tickdb.ai/ws"
        self.running = False
        self.reconnect_delay = 1.0
        self.max_reconnect_delay = 32.0
        self.heartbeat_interval = 30
        
        # Subscription state
        self.subscribed_symbols = {
            "us": [],
            "hk": [],
            "crypto": []
        }
        
    def connect(self):
        """Establish WebSocket connection with API key in URL parameter."""
        # ⚠️ TickDB WebSocket auth: API key goes in URL parameter, not header
        url = f"{self.base_url}?api_key={self.api_key}"
        self.ws = websocket.WebSocketApp(
            url,
            on_open=self._on_open,
            on_message=self._on_message,
            on_error=self._on_error,
            on_close=self._on_close
        )
        
        self.running = True
        # Run in daemon thread; production use should use threading.Event for shutdown
        thread = threading.Thread(target=self.ws.run_forever)
        thread.daemon = True
        thread.start()
        
        print(f"[{self._timestamp()}] Connected to unified gateway")
        
    def subscribe(self, symbols: list[str]):
        """
        Subscribe to any combination of US, HK, and crypto symbols.
        Symbol format: AAPL.US, 0700.HK, BTC-USD, ETH-USDT
        """
        for symbol in symbols:
            subscribe_frame = {
                "cmd": "subscribe",
                "channel": self._detect_channel(symbol),
                "symbol": symbol
            }
            self.ws.send(json.dumps(subscribe_frame))
            market = self._market_from_symbol(symbol)
            self.subscribed_symbols[market].append(symbol)
            print(f"[{self._timestamp()}] Subscribed to {symbol} ({market})")
    
    def _detect_channel(self, symbol: str) -> str:
        """Determine the appropriate channel for a symbol."""
        if symbol.endswith(".US"):
            return "kline"
        elif symbol.endswith(".HK"):
            return "depth"
        elif "-" in symbol or symbol.endswith("USDT"):
            return "kline"
        return "kline"
    
    def _market_from_symbol(self, symbol: str) -> str:
        """Map symbol to market bucket."""
        if symbol.endswith(".US"):
            return "us"
        elif symbol.endswith(".HK"):
            return "hk"
        return "crypto"
    
    def _on_open(self, ws):
        """Called when WebSocket connection is established."""
        print(f"[{self._timestamp()}] WebSocket opened; starting heartbeat")
        self._start_heartbeat()
        
        # Resubscribe to previously subscribed symbols after reconnect
        for market, symbols in self.subscribed_symbols.items():
            for symbol in symbols:
                subscribe_frame = {
                    "cmd": "subscribe",
                    "channel": self._detect_channel(symbol),
                    "symbol": symbol
                }
                ws.send(json.dumps(subscribe_frame))
    
    def _start_heartbeat(self):
        """Send ping frames at regular intervals to maintain connection."""
        def heartbeat_loop():
            while self.running and self.ws:
                time.sleep(self.heartbeat_interval)
                if self.running and self.ws:
                    try:
                        # TickDB uses JSON-based ping/pong; check docs for your version
                        self.ws.send(json.dumps({"cmd": "ping"}))
                        print(f"[{self._timestamp()}] Heartbeat sent")
                    except Exception as e:
                        print(f"[{self._timestamp()}] Heartbeat error: {e}")
                        break
        
        thread = threading.Thread(target=heartbeat_loop, daemon=True)
        thread.start()
    
    def _on_message(self, ws, message: str):
        """Handle incoming market data messages."""
        try:
            data = json.loads(message)
            
            # Handle pong responses
            if data.get("cmd") == "pong":
                return
            
            # Handle error responses (including rate limits)
            if "code" in data:
                code = data["code"]
                if code == 0:
                    # Success, no action needed
                    return
                elif code == 3001:
                    # Rate limit exceeded — extract Retry-After
                    retry_after = int(data.get("headers", {}).get("Retry-After", 5))
                    print(f"[{self._timestamp()}] Rate limited; waiting {retry_after}s")
                    time.sleep(retry_after)
                    return
                else:
                    print(f"[{self._timestamp()}] Gateway error {code}: {data.get('message')}")
                    return
            
            # Handle market data messages
            channel = data.get("channel")
            symbol = data.get("symbol")
            
            if channel == "kline":
                self._process_kline(data)
            elif channel == "depth":
                self._process_depth(data)
            elif channel == "trades":
                self._process_trade(data)
            else:
                print(f"[{self._timestamp()}] Unknown channel: {channel}")
                
        except json.JSONDecodeError as e:
            print(f"[{self._timestamp()}] JSON decode error: {e}")
        except Exception as e:
            print(f"[{self._timestamp()}] Message handling error: {e}")
    
    def _process_kline(self, data: dict):
        """Process OHLCV candle data."""
        kline = data.get("data", {})
        ts = datetime.fromtimestamp(kline.get("timestamp", 0) / 1000, tz=timezone.utc)
        print(
            f"[{ts.isoformat()}] {data.get('symbol')} | "
            f"O:{kline['open']} H:{kline['high']} L:{kline['low']} "
            f"C:{kline['close']} V:{kline['volume']}"
        )
    
    def _process_depth(self, data: dict):
        """Process order book depth snapshot."""
        depth = data.get("data", {})
        ts = datetime.fromtimestamp(depth.get("timestamp", 0) / 1000, tz=timezone.utc)
        best_bid = depth["bids"][0] if depth.get("bids") else None
        best_ask = depth["asks"][0] if depth.get("asks") else None
        
        if best_bid and best_ask:
            spread = best_ask[0] - best_bid[0]
            print(
                f"[{ts.isoformat()}] {data.get('symbol')} | "
                f"Bid:{best_bid[0]}({best_bid[1]}) Ask:{best_ask[0]}({best_ask[1]}) "
                f"Spread:{spread:.4f}"
            )
    
    def _process_trade(self, data: dict):
        """Process trade tick data."""
        trade = data.get("data", {})
        ts = datetime.fromtimestamp(trade.get("timestamp", 0) / 1000, tz=timezone.utc)
        print(
            f"[{ts.isoformat()}] TRADE {data.get('symbol')} | "
            f"Price:{trade['price']} Size:{trade['size']} Side:{trade.get('side', 'N/A')}"
        )
    
    def _on_error(self, ws, error):
        """Handle WebSocket errors and trigger reconnection."""
        print(f"[{self._timestamp()}] WebSocket error: {error}")
        self._schedule_reconnect()
    
    def _on_close(self, ws, close_status_code, close_msg):
        """Handle connection close and schedule reconnection."""
        print(f"[{self._timestamp()}] Connection closed ({close_status_code}): {close_msg}")
        self.running = False
        self._schedule_reconnect()
    
    def _schedule_reconnect(self):
        """Reconnect with exponential backoff and jitter."""
        self.running = False
        
        # Exponential backoff
        delay = min(self.reconnect_delay * 2, self.max_reconnect_delay)
        # Add jitter to prevent thundering herd
        jitter = random.uniform(0, delay * 0.1)
        total_delay = delay + jitter
        
        print(f"[{self._timestamp()}] Reconnecting in {total_delay:.2f}s")
        time.sleep(total_delay)
        self.reconnect_delay = delay
        self.connect()
    
    @staticmethod
    def _timestamp() -> str:
        """Return current UTC timestamp for logging."""
        return datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M:%S.%f")[:-3]


# Usage example
if __name__ == "__main__":
    # ⚠️ Load API key from environment variable — never hardcode credentials
    api_key = os.environ.get("TICKDB_API_KEY")
    if not api_key:
        raise ValueError("TICKDB_API_KEY environment variable not set")
    
    client = UnifiedMarketDataClient(api_key)
    client.connect()
    
    # Subscribe across markets with a single connection
    symbols = [
        "AAPL.US",      # US equity
        "0700.HK",      # HK equity
        "BTC-USD"       # Crypto
    ]
    client.subscribe(symbols)
    
    # Keep main thread alive; production use should implement graceful shutdown
    try:
        while True:
            time.sleep(1)
    except KeyboardInterrupt:
        print("Shutting down...")

Key architectural points in this implementation:

  1. Single connection, multiple markets: The client establishes one WebSocket connection. The gateway handles routing to individual exchange adapters internally.
  2. Symbol-based routing: The _market_from_symbol() method routes subscriptions to the correct adapter based on symbol suffix (.US, .HK, or crypto format).
  3. Heartbeat management: The client maintains its own heartbeat loop, which is independent of exchange-specific heartbeat intervals. This decouples client-side keepalive from the underlying adapters.
  4. Exponential backoff with jitter: Reconnection uses delay = min(base * (2 ** retry), max_delay) plus random jitter, preventing thundering herd on gateway restart.
  5. Rate-limit handling: The _on_message() method checks for code: 3001 and respects the Retry-After header.

Data Coverage by Market

The unified gateway exposes different channels depending on market capability. The table below summarizes supported channels:

Channel US Equities HK Equities Crypto
kline (OHLCV) ✅ 10+ years ✅ ✅
depth (order book) ✅ L1 ✅ L1–L10 ✅ L1–L10
trades (tick data) ❌ Not supported ✅ ✅

Important: TickDB's trades endpoint does not support US equities or A-shares. For US equity tick data, you must use a specialized tick data provider. For cross-market OHLCV backtesting, the kline endpoint is fully supported across all three markets.

Unified Gateway vs. Multi-Connection Approach

For teams evaluating whether to build their own multi-vendor integration or adopt a unified gateway, the comparison below quantifies the tradeoffs:

Dimension Build Your Own TickDB Unified Gateway
Connections managed N (one per vendor) 1 (to TickDB)
Protocol maintenance N integrations 0 (managed by TickDB)
Auth management N credential sets 1 API key
Timezone normalization DIY Built-in (UTC)
Symbol format translation DIY Automatic
Latency overhead None (direct to exchange) ~50–100 ms added
Historical data Requires separate backfill Unified via /v1/market/kline
Cost model Per-vendor pricing Single TickDB plan
Multi-market candle alignment DIY Automatic

The latency overhead is the primary tradeoff. Direct exchange connections offer the lowest possible latency, but at the cost of operational complexity. The unified gateway adds approximately 50–100 ms of latency—acceptable for most quantitative strategies, but potentially disqualifying for latency-sensitive high-frequency strategies.

Deployment Recommendations

The unified gateway is suitable for the following scenarios:

Use Case Recommendation
Cross-market strategy backtesting Use /v1/market/kline REST endpoint for historical OHLCV; single API key, unified timestamps
Multi-monitor dashboards Use the WebSocket gateway; subscribe to 20+ symbols across markets on one connection
Pairs trading (US + HK) Use WebSocket with depth channel for real-time order book; kline for signal generation
Crypto-only strategies Valid but consider direct exchange WebSocket if latency is critical
HFT / market making Not recommended; direct exchange connectivity required for sub-millisecond latency

Conclusion

The unified gateway solves a real operational problem: the infrastructure tax of managing heterogeneous market data sources. By centralizing protocol adaptation, symbol translation, and timezone normalization into a single layer, TickDB allows quantitative teams to focus on strategy development rather than connector maintenance.

The architecture is straightforward in concept—three-layer design with an adapter per market—but the implementation details matter. Heartbeat protocol mapping, error code translation, reconnection logic, and UTC normalization are the details that determine whether the unified gateway actually works in production.

For cross-market strategies, the single WebSocket connection approach reduces connection management overhead significantly. For latency-critical applications, the tradeoff is explicit: simplicity versus speed.

Next Steps

If you're building a cross-market quantitative strategy, start with the free TickDB API tier to test multi-market OHLCV data alignment across US equities, HK equities, and crypto.

If you need historical data for backtesting:

  1. Sign up at tickdb.ai (free, no credit card required)
  2. Generate an API key in the dashboard
  3. Set the TICKDB_API_KEY environment variable
  4. Use the /v1/market/kline endpoint with cross-market symbol parameters

If you're evaluating enterprise data volumes, contact enterprise@tickdb.ai for plans covering extended historical depth, real-time depth for US equities, and dedicated support.

If you use AI coding assistants, search for the tickdb-market-data SKILL in your AI tool's marketplace for assisted integration.

This article does not constitute investment advice. Markets involve risk; past performance does not guarantee future results. Data accuracy depends on exchange-provided feeds; TickDB serves as a data aggregation layer and does not guarantee the completeness or accuracy of exchange-sourced information.