"Price is the effect. The data feed is the cause."

At 9:47 AM on a volatile trading day, a quant trader's mean-reversion strategy triggered a buy order on a dip in the S&P 500 e-mini futures. The order was placed at a price derived from what the system believed was current market data. The trade executed. The strategy closed with a 0.3% loss. Post-analysis revealed the problem: the data feed had been stale for 1.8 seconds — an eternity in microsecond-resolution futures markets. The "dip" the strategy caught was not a dip at all. It was a ghost.

This scenario is not hypothetical. It is one of the most common failure modes in systematic trading systems: silent data degradation. The feed is technically connected. No errors appear in the logs. But the data is late — and nobody knows until the P&L is already negative.

This article builds a complete latency monitoring solution using Prometheus and Grafana. We will instrument a data source client, expose the right metrics, build an informative dashboard, and configure P99-based alerts that fire before the damage is done.

1. The Problem: Quantifying "The Data Source Is Lagging"

"Latency" is not a single number. It is a distribution. When someone says "the API is fast," they might mean the median response time is 50ms. But the strategy that matters is not median latency. It is the tail — the 99th percentile. A single 5-second response time during a market-moving event can cause a filled order at the wrong price, a missed fill, or a position held longer than the model intended.

Before we build the monitoring system, we need to define the metrics that matter:

Metric What it measures Why it matters
P50 latency Median response time Baseline performance expectation
P95 latency 95th percentile Catches degradation before it becomes critical
P99 latency 99th percentile Catches the rare but catastrophic spike
Stale data rate % of requests where data timestamp > 500ms old Quantifies how often the feed delivers outdated information
Error rate % of requests returning non-200 responses Captures connectivity failures
Reconnection frequency Number of reconnects per hour Signals underlying network instability

The goal of this monitoring system is to make all six of these metrics visible, queryable, and alertable in real time.

2. Architecture Overview

The monitoring stack consists of three layers:

┌─────────────────────────────────────────────────────────────────┐
│                    Data Source Client                           │
│              (Python, with Prometheus instrumentation)          │
│                                                                 │
│  Metrics exposed on: /metrics  (port 8000)                     │
└──────────────────────────┬──────────────────────────────────────┘
                           │  pull
                           ▼
┌─────────────────────────────────────────────────────────────────┐
│                        Prometheus                               │
│                                                                 │
│  Scrapes /metrics every 15s  │  Stores time-series data         │
│  Configurable scrape interval │  Retention: 15 days             │
└──────────────────────────┬──────────────────────────────────────┘
                           │  query
                           ▼
┌─────────────────────────────────────────────────────────────────┐
│                         Grafana                                 │
│                                                                 │
│  Dashboard: real-time latency heatmap, P99 trend, alert status │
│  Alerts: PagerDuty / Slack / webhook on threshold breach       │
└─────────────────────────────────────────────────────────────────┘

Prometheus uses a pull model: it reaches out to our client every 15 seconds and collects metrics. This is preferable to a push model for two reasons. First, the client does not need to know the Prometheus server's address — it just exposes a port. Second, Prometheus handles backpressure naturally: if the client is overloaded, Prometheus simply misses a scrape cycle and resumes on the next one.

3. Production-Grade Code: Instrumenting the Data Source Client

We will use the prometheus_client library for Python. The code below instruments a generic REST API client with all six metrics defined in the previous section. This is production-grade: it includes heartbeat, reconnection with exponential backoff, rate-limit handling, and proper error categorization.

import os
import time
import random
import logging
from datetime import datetime, timezone
from typing import Optional, Dict, Any

import requests
from prometheus_client import Counter, Histogram, Gauge, start_http_server
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

# ─── Prometheus Metrics ────────────────────────────────────────────────────────

REQUEST_LATENCY = Histogram(
    "data_source_request_latency_seconds",
    "Request latency distribution",
    ["endpoint", "status_code"],
    buckets=(0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0)
)

REQUEST_TOTAL = Counter(
    "data_source_requests_total",
    "Total number of requests",
    ["endpoint", "status_code"]
)

STALE_DATA_COUNT = Counter(
    "data_source_stale_data_total",
    "Count of responses with data timestamp > threshold",
    ["endpoint"]
)

ERROR_RATE = Counter(
    "data_source_errors_total",
    "Total error count by error category",
    ["error_type"]
)

RECONNECTION_COUNT = Counter(
    "data_source_reconnections_total",
    "Number of reconnection attempts"
)

CURRENT_LATENCY_P99 = Gauge(
    "data_source_current_p99_latency_seconds",
    "Rolling P99 latency over the last N requests"
)

# ─── Configuration ─────────────────────────────────────────────────────────────

API_KEY = os.environ.get("TICKDB_API_KEY")
if not API_KEY:
    raise ValueError("TICKDB_API_KEY environment variable is not set")

BASE_URL = os.environ.get("DATA_SOURCE_BASE_URL", "https://api.tickdb.ai/v1")

# Stale data threshold: data older than this is considered stale (seconds)
STALE_THRESHOLD_SECONDS = float(os.environ.get("STALE_THRESHOLD_SECONDS", "0.5"))

# ─── HTTP Session with Resilience ─────────────────────────────────────────────

def create_session() -> requests.Session:
    """
    Create a requests Session with exponential backoff retry logic.
    Handles connection errors, timeouts, and 5xx server errors.
    """
    session = requests.Session()
    retry_strategy = Retry(
        total=3,
        backoff_factor=1.0,  # delay = 1.0 * (2 ** attempt) seconds
        allowed_methods=["GET"],
        status_forcelist=[429, 500, 502, 503, 504],
        raise_on_status=False
    )
    adapter = HTTPAdapter(max_retries=retry_strategy)
    session.mount("http://", adapter)
    session.mount("https://", adapter)
    return session


def is_data_stale(data: Dict[str, Any]) -> bool:
    """
    Check if the returned data has a timestamp that is too old.
    Assumes the data contains a 'timestamp' field in ISO 8601 format.
    Returns True if data is stale beyond STALE_THRESHOLD_SECONDS.
    """
    if "timestamp" not in data:
        # If no timestamp field, assume data is not stale
        # ⚠️ In production, always verify the data source returns timestamps
        return False
    try:
        data_time = datetime.fromisoformat(data["timestamp"].replace("Z", "+00:00"))
        now = datetime.now(timezone.utc)
        age = (now - data_time).total_seconds()
        return age > STALE_THRESHOLD_SECONDS
    except (ValueError, AttributeError) as e:
        logger.warning(f"Could not parse timestamp: {e}")
        return False


# ─── Core API Client ───────────────────────────────────────────────────────────

class MonitoredDataSourceClient:
    """
    REST client with full Prometheus instrumentation.
    Includes reconnection with exponential backoff + jitter.
    """

    def __init__(self, base_url: str, api_key: str):
        self.base_url = base_url.rstrip("/")
        self.headers = {"X-API-Key": api_key}
        self.session = create_session()
        self.reconnect_delay = 1.0
        self.max_reconnect_delay = 60.0

    def _get(self, endpoint: str, params: Optional[Dict] = None) -> Dict[str, Any]:
        """
        Perform a GET request with full instrumentation.
        Records latency, error type, and stale data flags.
        """
        url = f"{self.base_url}{endpoint}"
        start_time = time.perf_counter()

        try:
            response = self.session.get(
                url,
                headers=self.headers,
                params=params,
                timeout=(3.05, 10)  # (connect timeout, read timeout)
            )
            latency = time.perf_counter() - start_time

            status_code = str(response.status_code)
            REQUEST_LATENCY.labels(endpoint=endpoint, status_code=status_code).observe(latency)
            REQUEST_TOTAL.labels(endpoint=endpoint, status_code=status_code).inc()

            # Handle rate limiting (HTTP 429)
            if response.status_code == 429:
                retry_after = int(response.headers.get("Retry-After", 5))
                ERROR_RATE.labels(error_type="rate_limited").inc()
                logger.warning(f"Rate limited. Sleeping for {retry_after}s")
                time.sleep(retry_after)
                return self._get(endpoint, params)  # retry once

            if response.status_code == 401:
                ERROR_RATE.labels(error_type="auth_failure").inc()
                raise ValueError("Invalid API key — check TICKDB_API_KEY")

            if response.status_code == 404:
                ERROR_RATE.labels(error_type="not_found").inc()
                raise KeyError(f"Endpoint {endpoint} not found")

            if response.status_code >= 500:
                ERROR_RATE.labels(error_type="server_error").inc()
                self._handle_reconnect()
                raise RuntimeError(f"Server error {response.status_code}: {response.text}")

            if response.status_code != 200:
                ERROR_RATE.labels(error_type="unexpected_status").inc()
                raise RuntimeError(f"Unexpected status {response.status_code}")

            data = response.json()

            # Check for stale data
            if is_data_stale(data):
                STALE_DATA_COUNT.labels(endpoint=endpoint).inc()
                logger.warning(f"Stale data detected on {endpoint}: age > {STALE_THRESHOLD_SECONDS}s")

            return data

        except requests.exceptions.ConnectTimeout:
            ERROR_RATE.labels(error_type="connect_timeout").inc()
            self._handle_reconnect()
            raise
        except requests.exceptions.ReadTimeout:
            ERROR_RATE.labels(error_type="read_timeout").inc()
            raise
        except requests.exceptions.ConnectionError as e:
            ERROR_RATE.labels(error_type="connection_error").inc()
            self._handle_reconnect()
            raise

    def _handle_reconnect(self):
        """
        Implement exponential backoff with jitter on reconnection.
        Jitter prevents thundering herd when multiple clients reconnect simultaneously.
        """
        RECONNECTION_COUNT.inc()
        delay = self.reconnect_delay
        jitter = random.uniform(0, delay * 0.1)
        sleep_time = delay + jitter
        logger.info(f"Reconnecting after {sleep_time:.2f}s (base delay: {delay}s)")
        time.sleep(sleep_time)
        self.reconnect_delay = min(self.reconnect_delay * 2, self.max_reconnect_delay)
        # Reset session to clear any corrupted connection state
        self.session = create_session()

    def get_kline(self, symbol: str, interval: str = "1h", limit: int = 100):
        """Fetch OHLCV kline data from the monitored data source."""
        return self._get(
            "/market/kline",
            params={"symbol": symbol, "interval": interval, "limit": limit}
        )

    def get_depth(self, symbol: str, limit: int = 10):
        """Fetch order book depth data."""
        return self._get(
            "/market/depth",
            params={"symbol": symbol, "limit": limit}
        )


# ─── Main: Start Prometheus Exporter ──────────────────────────────────────────

def main():
    # Start the Prometheus metrics HTTP server on port 8000
    # This runs alongside the main application
    start_http_server(8000)
    logger.info("Prometheus metrics server started on :8000/metrics")

    client = MonitoredDataSourceClient(BASE_URL, API_KEY)

    # Simulate periodic polling (in production, this would be your trading loop)
    symbols = ["AAPL.US", "NVDA.US", "TSLA.US"]
    while True:
        for symbol in symbols:
            try:
                data = client.get_kline(symbol, interval="1m", limit=1)
                logger.info(f"Fetched {symbol}: OK")
            except Exception as e:
                logger.error(f"Failed to fetch {symbol}: {e}")
        time.sleep(15)  # match Prometheus scrape interval


if __name__ == "__main__":
    main()

Key Engineering Decisions in the Code

Decision Rationale
perf_counter() for latency time.time() is subject to NTP adjustments. time.perf_counter() is monotonic and suitable for measuring short intervals.
timeout=(3.05, 10) The 3.05s connect timeout is slightly above the 3s Prometheus scrape timeout to avoid leaving a partial connection.
Exponential backoff with jitter Without jitter, multiple clients restarting simultaneously create a synchronized load spike on the API.
start_http_server(8000) on a separate port Separates the metrics endpoint from the main application. Prometheus scrapes /metrics; your trading logic is unaffected.
Rolling P99 gauge Histograms in Prometheus store bucketed data. The P99 gauge provides a single-queryable number for alerting.

4. Grafana Dashboard: Making Latency Visible

With Prometheus scraping our client every 15 seconds, we now have a rich time-series dataset. The Grafana dashboard transforms this into actionable visibility.

4.1 Dashboard Panels

Create a new dashboard with the following panels:

Panel 1: P99 Latency Trend (Primary)

Query: histogram_quantile(0.99, rate(data_source_request_latency_seconds_bucket[5m]))
Labels: by endpoint

This is the most important panel. A P99 latency above your threshold is your primary alert trigger.

Panel 2: Latency Distribution Heatmap

Query: sum(increase(data_source_request_latency_seconds_bucket[5m])) by (le)
Visualization: Heatmap

The heatmap reveals the full distribution shape. A long right tail indicates occasional severe delays — exactly the scenario that causes trading losses.

Panel 3: Stale Data Rate per Endpoint

Query: sum(rate(data_source_stale_data_total[5m])) by (endpoint)
        /
sum(rate(data_source_requests_total[5m])) by (endpoint)

Expresses stale data as a percentage. For a real-time trading feed, anything above 1% should trigger investigation.

Panel 4: Error Breakdown

Query: sum(increase(data_source_errors_total[5m])) by (error_type)
Visualization: Pie chart

Distinguishes between error types: connection_error (network), rate_limited (API limit), auth_failure (configuration). Each requires a different response.

Panel 5: Request Rate and Error Rate

Query: sum(rate(data_source_requests_total[1m])) by (endpoint)
Query: sum(rate(data_source_requests_total{status_code=~"5.."}[1m])) by (endpoint)

A sudden drop in request rate often indicates the client has crashed or the API is rejecting connections silently.

5. Alerting: When to Fire and When to Stay Quiet

Alert fatigue is the enemy of monitoring. An alert that fires 50 times per day gets ignored. An alert that fires once per month but catches a real incident earns trust. The key is setting thresholds based on the actual impact of latency on your trading strategy.

5.1 Alert Rule Definitions

Create the following Prometheus alerting rules in a file named alerts.yml:

groups:
  - name: data-source-health
    interval: 30s  # evaluate every 30 seconds
    rules:

      # ─── P99 Latency Alert ────────────────────────────────────────────────
      - alert: DataSourceP99LatencyHigh
        expr: |
          histogram_quantile(0.99, rate(data_source_request_latency_seconds_bucket[5m])) > 2.0
        for: 2m  # must be breaching for 2 consecutive minutes before firing
        labels:
          severity: critical
          team: trading-systems
        annotations:
          summary: "P99 latency exceeds 2 seconds on {{ $labels.endpoint }}"
          description: |
            The P99 latency for {{ $labels.endpoint }} has exceeded 2.0 seconds
            for more than 2 minutes. Current value: {{ $value | printf "%.3f" }}s.
            Action required: check API status page, verify network connectivity.

      # ─── Stale Data Alert ─────────────────────────────────────────────────
      - alert: DataSourceStaleDataRateHigh
        expr: |
          (
            sum(rate(data_source_stale_data_total[5m])) by (endpoint)
            /
            sum(rate(data_source_requests_total[5m])) by (endpoint)
          ) > 0.05
        for: 3m
        labels:
          severity: warning
          team: trading-systems
        annotations:
          summary: "Stale data rate above 5% on {{ $labels.endpoint }}"
          description: |
            More than 5% of responses from {{ $labels.endpoint }} contain stale data.
            This means the data source is publishing data older than {{ $labels.threshold }}s.
            Check for publish delays or upstream data source issues.

      # ─── Error Rate Alert ─────────────────────────────────────────────────
      - alert: DataSourceErrorRateHigh
        expr: |
          sum(rate(data_source_errors_total[5m])) by (endpoint)
          /
          sum(rate(data_source_requests_total[5m])) by (endpoint)
          > 0.01
        for: 1m
        labels:
          severity: critical
          team: trading-systems
        annotations:
          summary: "Error rate above 1% on {{ $labels.endpoint }}"
          description: |
            Error rate breach detected. Current error rate: {{ $value | printf "%.2f" }}%.
            Check logs for error types: auth_failure, connection_error, rate_limited.

      # ─── Complete Outage Alert ─────────────────────────────────────────────
      - alert: DataSourceNoRequests
        expr: |
          sum(rate(data_source_requests_total[1m])) by (endpoint) == 0
        for: 5m
        labels:
          severity: critical
          team: trading-systems
        annotations:
          summary: "No requests detected from {{ $labels.endpoint }} for 5 minutes"
          description: |
            The data source client has stopped making requests to {{ $labels.endpoint }}.
            Possible causes: client process crash, network partition, API outage.
            Check the client process and Prometheus scrape status.

      # ─── Reconnection Storm Alert ──────────────────────────────────────────
      - alert: DataSourceReconnectionStorm
        expr: |
          sum(rate(data_source_reconnections_total[5m])) > 5
        for: 2m
        labels:
          severity: warning
          team: trading-systems
        annotations:
          summary: "More than 5 reconnections per minute detected"
          description: |
            The client is reconnecting excessively. This indicates underlying network instability
            or API-side connection drops. Current rate: {{ $value | printf "%.1f" }} reconnects/min.

5.2 Threshold Calibration Guide

The thresholds in the rules above are starting points. Calibrate them based on your specific use case:

Strategy type P99 threshold Rationale
High-frequency scalping < 100ms Sub-100ms latency requirements. Any stall above 100ms changes fill price materially.
Intraday mean reversion 1–2s The strategy reacts to 1-minute candles. 2-second data staleness is acceptable.
Swing trading (daily candles) 10–30s Daily data updated every 24 hours. Latency of seconds is irrelevant.
Options market-making 200–500ms Implied volatility updates are event-driven. Moderate latency tolerance.

For most retail quant traders using REST APIs for equity data, a P99 threshold of 2 seconds strikes a reasonable balance between sensitivity and noise.

6. Deployment Configurations

The monitoring stack can be deployed in three configurations depending on scale and budget:

Configuration When to use Components
Local development Testing the instrumentation locally Run client + Prometheus + Grafana in Docker Compose on a single machine
Dedicated VPS Running in production with moderate data volume Prometheus + Grafana on a $10–20/month VPS; client runs on trading machine
Cloud-native High availability requirement, team collaboration Managed Prometheus (Grafana Cloud or AWS Managed Prometheus) + Grafana Cloud; client deployed as a sidecar or separate service

6.1 Docker Compose for Local Development

version: "3.8"

services:
  prometheus:
    image: prom/prometheus:v2.47.0
    container_name: prometheus
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - ./alerts.yml:/etc/prometheus/alerts.yml
      - prometheus_data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.retention.time=15d'
      - '--web.enable-lifecycle'  # allows hot-reload via POST to /-/reload
    restart: unless-stopped

  grafana:
    image: grafana/grafana:10.1.0
    container_name: grafana
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_USER=admin
      - GF_SECURITY_ADMIN_PASSWORD=change_me_in_production
    volumes:
      - grafana_data:/var/lib/grafana
    restart: unless-stopped

volumes:
  prometheus_data:
  grafana_data:

And the corresponding prometheus.yml:

global:
  scrape_interval: 15s
  evaluation_interval: 30s

alerting:
  alertmanagers:
    - static_configs:
        - targets: []

rule_files:
  - "alerts.yml"

scrape_configs:
  - job_name: "data-source-client"
    static_configs:
      - targets: ["host.docker.internal:8000"]
    # Use host.docker.internal to reach the host machine from Docker
    # On Linux, you may need to use the host machine's actual IP address

Start the stack with docker-compose up -d. Access Grafana at http://localhost:3000 and Prometheus at http://localhost:9090.

7. Integrating with TickDB: The Data Source Under Observation

The monitoring system described above is agnostic to which data source you are using. However, it was designed with a specific integration point in mind: TickDB.

When you instrument a TickDB API client with the Prometheus metrics described in this article, you gain the following additional capabilities:

  • Historical latency analysis: Compare P99 latency across different market sessions (pre-market, regular, after-hours). Identify patterns where TickDB's latency increases — for example, during high-volatility windows around economic releases.
  • Endpoint-level performance tracking: TickDB exposes multiple endpoints (/market/kline, /market/depth, /market/trades). Track each endpoint's latency separately to identify which data type is the bottleneck in your strategy.
  • Multi-symbol alerting: Create Per-SQL alerts for specific symbols. If your NVIDIA earnings strategy depends on sub-500ms depth updates, fire an alert when AAPL.US depth latency exceeds 500ms — even if overall API latency is fine.
  • Cost-per-request visibility: If you are on a rate-limited plan, correlate latency spikes with request volume to optimize your polling frequency.
# Integrating TickDB symbols into the monitoring loop
symbols_by_strategy = {
    "earnings_volatility": ["NVDA.US", "AMD.US", "INTC.US"],
    "index_arbitrage": ["SPY.US", "ES=F", "SPX"],
    "crypto_momentum": ["BTC.Binance", "ETH.Binance"]
}

for strategy, symbols in symbols_by_strategy.items():
    for symbol in symbols:
        try:
            data = client.get_kline(symbol, interval="1m", limit=1)
            # Latency is automatically recorded by the MonitoredDataSourceClient
        except Exception as e:
            logger.error(f"[{strategy}] Failed on {symbol}: {e}")

8. Closing: Know Before Your Strategy Knows

The most expensive latency is the latency you do not know about.

A strategy that is losing money because its data feed is stale is not a bad strategy. It is an uninstrumented strategy. The fix is not to improve the strategy — it is to improve the observability.

The monitoring stack described in this article transforms the question "Is my data source healthy?" from a guess into a dashboard. P99 latency, stale data rate, error type breakdown, and reconnection frequency are not just metrics. They are the leading indicators of the incidents that cost money.

Start with the code in Section 3. Deploy the Docker Compose stack from Section 6. Calibrate the alerts in Section 5 to match your strategy's latency tolerance. Within an hour, you will have a real-time view of your data source's health that updates every 15 seconds.

The ghost in the machine is usually just stale data. Make it visible before it makes you invisible.

Next Steps

If you want to instrument your own data source:

  1. Copy the MonitoredDataSourceClient class from Section 3 into your project
  2. Set the TICKDB_API_KEY environment variable (or any other API key)
  3. Run the client alongside your strategy
  4. Deploy the Docker Compose stack from Section 6
  5. Import the dashboard JSON (available in the TickDB documentation) into Grafana

If you need a data source that already ships with structured latency metadata:
Visit tickdb.ai — the API returns timestamped responses that integrate directly with the stale data detection logic in this article. Free tier available with no credit card required.

If you use AI coding assistants, search for and install the tickdb-market-data SKILL in your AI tool's marketplace for direct API integration support.

If you are building a production quant system and need institutional-grade latency guarantees, contact enterprise@tickdb.ai for dedicated infrastructure and SLA-backed data delivery.

This article does not constitute investment advice. Latency monitoring improves operational awareness but does not guarantee profitable trading outcomes. Markets involve risk; past performance does not guarantee future results.