Building Production-Grade Data Infrastructure for Small Quantitative Teams

One data outage costs more than six months of cloud fees.

At 9:32 AM on a Tuesday, the senior researcher on a three-person quant team discovered that his backtest had been running on stale data for three days. The cause: his teammate had pushed a local configuration change that redirected the data pipeline to a deprecated endpoint. Neither had a clear picture of which data source was authoritative, because there was no shared registry. The team lost four hours of compute time and, worse, lost confidence in their entire data pipeline.

This is not a hypothetical. Small quantitative teams face infrastructure challenges that scale poorly from solo work to collaborative environments. A solo researcher owns every variable. A three-person team must negotiate data ownership, coordinate code changes, and protect sensitive credentials across machines with different security postures. The technical complexity does not grow linearly — it compounds.

This article provides a production-grade blueprint for equipping a three-person quantitative team with shared data infrastructure. It covers three operational layers: data sharing architecture, Git-based code collaboration, and API key management with permission controls. Every recommendation is designed for a team that needs to move fast without breaking things.

---

The Three-Layer Collaboration Problem

When a quantitative team scales from one to three, three distinct failure modes emerge simultaneously.

Data fragmentation occurs when each researcher maintains a local copy of market data with no synchronization mechanism. Researcher A pulls daily OHLCV from one vendor. Researcher B pulls from another. Researcher C combines both with his own proprietary additions. When the strategy goes live, no one knows which dataset is correct. Reconciliation becomes a manual, error-prone process that consumes hours before every important deadline.

Code drift happens when team members modify shared scripts independently. Without version control discipline, the codebase accumulates divergent branches that no one fully understands. Merging becomes painful enough that teams avoid it — and then two researchers are effectively maintaining separate codebases while believing they share one.

Credential sprawl is the most dangerous failure mode. API keys are copied into Slack messages, pasted into shared spreadsheets, or stored in .env files that get committed to repositories. The attack surface is wide, the audit trail is nonexistent, and a single leaked key can compromise the entire team's data access — or worse, create financial liability.

The solution requires addressing all three layers in coordination. Partial implementations create false confidence.

---

Layer One: Shared Data Infrastructure

Architecture Overview

The goal is a single authoritative data source that all team members query through a shared access layer. The architecture consists of three components: a TickDB account as the primary data source, a local PostgreSQL instance for derived datasets, and a lightweight metadata registry that tracks what data exists and where.

The key principle is query-through, not copy-through. Rather than downloading data to each researcher's machine and hoping the copies stay synchronized, every team member queries the same TickDB API. This eliminates copy divergence entirely. The only data that lives locally is computed derivatives — alpha signals, feature matrices, and backtest results — which are stored in the shared PostgreSQL instance.

┌─────────────────────────────────────────────────────────────┐
│                    Team Members (×3)                        │
│   Researcher A          Researcher B          Researcher C │
└────────────────────┬────────────────────────────────────────┘
                     │         TickDB API          │
                     │◄────────────────────────────│
                     │                               │
          ┌──────────▼──────────┐                    │
          │   Local TickDB     │◄───────── (Optional Cache)
          │   Python Client    │                    │
          └────────────────────┘                    │
                                                      │
┌─────────────────────────────────────────────────────▼───────┐
│                   Shared PostgreSQL                        │
│  • Derived alpha signals  • Feature matrices  • Backtests  │
└─────────────────────────────────────────────────────────────┘

Shared Data Registry

Every team needs a lightweight registry that answers one question: "What data do we have, and where?" Without this, researchers spend time discovering data rather than analyzing it.

Create a simple data_registry.csv in the project repository:

dataset_name,source,last_updated,owner,description
us_equity_ohlcv_1d,TickDB /v1/market/kline,2026-04-15,alice,US stocks daily OHLCV, 10+ year history
hk_equity_depth,TickDB /v1/market/depth,2026-04-15,bob,HK stocks L1-L10 order book depth
crypto_spot_trades,TickDB /v1/market/trades,2026-04-15,carol,Binance BTC/ETH spot trades
alpha_signals_local,Local computation,2026-04-14,alice,Factor signals derived from us_equity_ohlcv_1d

The registry lives in the Git repository and follows the same review process as code. When a researcher adds a new dataset, they open a pull request. Teammates can review the source, verify the update frequency, and flag conflicts before the data enters the pipeline.

Querying TickDB as a Team

When multiple researchers query TickDB simultaneously, the team's combined request volume counts toward the same rate limits. Coordinate usage by implementing a simple request scheduler that distributes queries over time.

import os
import time
import requests
from datetime import datetime, timedelta
import logging

logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(message)s')
logger = logging.getLogger(__name__)

TICKDB_API_KEY = os.environ.get("TICKDB_API_KEY")
BASE_URL = "https://api.tickdb.ai/v1"
REQUEST_DELAY = 0.5  # seconds between requests to avoid rate limits

class SharedTickDBClient:
    """
    Team-shared TickDB client with rate limiting and retry logic.
    All team members use this client to ensure coordinated API usage.
    """
    
    def __init__(self, api_key: str = None):
        self.api_key = api_key or os.environ.get("TICKDB_API_KEY")
        if not self.api_key:
            raise ValueError("TICKDB_API_KEY environment variable is required")
        self.headers = {"X-API-Key": self.api_key}
    
    def _request_with_backoff(self, method: str, endpoint: str, **kwargs) -> dict:
        """Make HTTP request with exponential backoff and rate-limit handling."""
        url = f"{BASE_URL}/{endpoint}"
        max_retries = 5
        base_delay = 1
        
        for attempt in range(max_retries):
            try:
                response = requests.request(
                    method=method,
                    url=url,
                    headers=self.headers,
                    timeout=(3.05, 10),
                    **kwargs
                )
                
                if response.status_code == 429:
                    retry_after = int(response.headers.get("Retry-After", 60))
                    logger.warning(f"Rate limited. Waiting {retry_after}s before retry.")
                    time.sleep(retry_after)
                    continue
                
                response.raise_for_status()
                return response.json()
                
            except requests.exceptions.RequestException as e:
                if attempt == max_retries - 1:
                    raise
                delay = min(base_delay * (2 ** attempt), 30)
                jitter = time.uniform(0, delay * 0.1)
                logger.warning(f"Request failed (attempt {attempt + 1}): {e}. Retrying in {delay:.1f}s")
                time.sleep(delay + jitter)
        
        raise RuntimeError(f"Request failed after {max_retries} attempts")
    
    def get_kline(self, symbol: str, interval: str, start_time: datetime, 
                  end_time: datetime, limit: int = 1000) -> list:
        """
        Fetch historical OHLCV klines for a symbol.
        
        Args:
            symbol: Trading pair (e.g., "AAPL.US", "NVDA.US")
            interval: Candle interval (e.g., "1h", "1d")
            start_time: Start of the period
            end_time: End of the period
            limit: Maximum records per request (max 1000)
        """
        params = {
            "symbol": symbol,
            "interval": interval,
            "start_time": int(start_time.timestamp()),
            "end_time": int(end_time.timestamp()),
            "limit": limit
        }
        
        all_data = []
        current_start = start_time
        
        while current_start < end_time:
            params["start_time"] = int(current_start.timestamp())
            result = self._request_with_backoff("GET", "market/kline", params=params)
            data = result.get("data", {}).get("klines", [])
            
            if not data:
                break
                
            all_data.extend(data)
            
            # ⚠️ Respect rate limits — adjust REQUEST_DELAY based on your plan tier
            time.sleep(REQUEST_DELAY)
            
            # Move to next batch
            last_candle_time = datetime.fromtimestamp(data[-1]["t"] / 1000)
            current_start = last_candle_time + timedelta(minutes=1)
        
        logger.info(f"Fetched {len(all_data)} candles for {symbol} from {start_time.date()} to {end_time.date()}")
        return all_data


# Team usage: share the initialized client across research scripts
# from team_infrastructure import SharedTickDBClient
# client = SharedTickDBClient()  # reads TICKDB_API_KEY from shared vault

The client enforces a 0.5-second delay between requests. For teams on paid plans with higher rate limits, reduce REQUEST_DELAY to 0.1 seconds. Document any changes to shared infrastructure parameters in the registry.

PostgreSQL Setup for Derived Data

Store computed derivatives — alpha signals, feature matrices, and backtest results — in a shared PostgreSQL database. Each table includes metadata columns for provenance tracking.

-- Shared feature store schema
CREATE TABLE alpha_signals (
    id SERIAL PRIMARY KEY,
    signal_date DATE NOT NULL,
    symbol VARCHAR(20) NOT NULL,
    signal_type VARCHAR(50) NOT NULL,
    signal_value DECIMAL(10, 6),
    computed_by VARCHAR(100) NOT NULL,  -- team member identifier
    computed_at TIMESTAMP DEFAULT NOW(),
    data_source_id VARCHAR(100),         -- references data_registry.dataset_name
    UNIQUE(signal_date, symbol, signal_type)
);

CREATE INDEX idx_signals_symbol_date ON alpha_signals(symbol, signal_date);
CREATE INDEX idx_signals_computed_by ON alpha_signals(computed_by);

-- Backtest results table
CREATE TABLE backtest_results (
    id SERIAL PRIMARY KEY,
    strategy_name VARCHAR(100) NOT NULL,
    run_date DATE NOT NULL,
    total_return DECIMAL(8, 4),
    sharpe_ratio DECIMAL(6, 3),
    max_drawdown DECIMAL(8, 4),
    win_rate DECIMAL(5, 3),
    sample_size INTEGER,
    period_start DATE,
    period_end DATE,
    git_commit_hash VARCHAR(40),         -- links to exact code version
    computed_by VARCHAR(100) NOT NULL,
    created_at TIMESTAMP DEFAULT NOW(),
    notes TEXT
);

CREATE INDEX idx_backtest_strategy ON backtest_results(strategy_name, run_date);

Every row includes computed_by and git_commit_hash columns. This creates a complete audit trail connecting any signal or result to the exact data and code that produced it.

---

Layer Two: Git-Based Code Collaboration

Repository Structure

Organize the repository into four directories with clear ownership:

quant-team-project/
├── data/                  # Not data files — only data registry and documentation
│   ├── data_registry.csv  # Authoritative source of truth for available datasets
│   └── schemas/           # PostgreSQL schema files
├── src/                   # Research and strategy code
│   ├── features/          # Feature engineering modules
│   ├── strategies/        # Strategy implementations
│   └── backtests/         # Backtest framework and results
├── infrastructure/         # Shared tooling (team-accessible)
│   ├── client.py          # Shared TickDB client
│   ├── db_connection.py   # PostgreSQL connection utilities
│   └── schedule.py        # Shared scheduling utilities
├── configs/               # Configuration files (secrets excluded)
│   └── config_template.yaml
├── .env.example           # Template for environment variables
├── .gitignore
└── README.md

The .gitignore must explicitly exclude .env, *.pyc, __pycache__/, data/*.csv, and any file containing the string api_key.

Branching Strategy

For a three-person team, a simplified GitHub Flow works better than Git Flow.

• main: Stable, deployable code. Protected. Requires at least one review.
• feature/*: Individual work branches. Short-lived (1–3 days maximum).
• hotfix/*: Emergency fixes for production issues.

main ──────────────────────────────────────────────────────────
        │                    │                    │
    feature/data-reg    feature/momentum-alpha    feature/deploy-automation
        │                    │                    │
        └────────────────────┴────────────────────┘
                            │
                         (merge)

Commit messages must follow the Conventional Commits specification. This makes git log readable and enables automated changelog generation.

feat: add momentum alpha factor for US equities
fix: correct TickDB timestamp parsing in client
docs: update data registry with new HK depth data source
refactor: extract shared database connection pool

Code Review Process

Code review is not optional — it is the mechanism that prevents credential sprawl and data fragmentation from reappearing. Three rules:

1. No self-merge. Every branch must be reviewed by at least one other team member before merging to main.
2. Review the data registry changes. When a teammate adds a new dataset, verify the source URL, update frequency, and ownership assignment.
3. Review credential handling. Any pull request that touches .env, config, or *secret* files must be flagged for security review.

Use branch protection rules on GitHub or GitLab:

# GitHub branch protection (CLI)
gh api repos/{owner}/{repo}/branches/main/protection \
  --method PUT \
  --field required_pull_request_reviews="{\"required_approving_review_count\": 1}" \
  --field enforce_admins="true" \
  --field require_linear_history="true"

---

Layer Three: API Key Management and Permission Control

The Credential Problem

In an informal three-person team, credential management often looks like this: one researcher shares the API key in a Slack direct message, another writes it in a Google Doc titled "Important Stuff," and the third hardcodes it in a Jupyter notebook that eventually gets committed to the repository.

This is not a hypothetical risk. Leaked credentials are actively exploited within hours of exposure on GitHub. A leaked TickDB API key could allow an attacker to consume the team's rate limit quota, corrupt data access, or generate charges on a paid plan.

Secrets Management with HashiCorp Vault (Simplified)

For a three-person team, a full Vault deployment is overkill. Use a simpler but still secure approach: a team secrets manager with per-key audit logging.

Option A: 1Password Teams (Recommended for Small Teams)

1. Create a shared 1Password vault named "Quant Infrastructure."
2. Store each credential as a separate item with the following fields: service name, key/value, owner, rotation date, and notes.
3. Use the 1Password CLI to inject secrets into the environment:

# Install 1Password CLI
brew install 1password-cli

# Sign in (one-time setup per machine)
op signin

# Inject TickDB API key into environment for a script
eval $(op run --env-file=".env" -- python src/strategies/run_backtest.py)

Create a .env.template file with placeholder values:

# .env.template
# Copy this file to .env and fill in values from 1Password vault "Quant Infrastructure"
TICKDB_API_KEY=your-key-from-1password
POSTGRES_HOST=your-shared-db-host
POSTGRES_DB=quant_team_db
POSTGRES_USER=your-db-username
POSTGRES_PASSWORD=your-db-password-from-1password

Option B: AWS Secrets Manager (For Teams Already on AWS)

If the team uses AWS EC2 or Lambda for deployment, AWS Secrets Manager provides native integration:

import boto3
import json

def get_shared_secret(secret_name: str) -> dict:
    """Retrieve shared credentials from AWS Secrets Manager."""
    client = boto3.client('secretsmanager')
    
    try:
        response = client.get_secret_value(SecretId=secret_name)
        return json.loads(response['SecretString'])
    except client.exceptions.ResourceNotFoundException:
        raise ValueError(f"Secret '{secret_name}' not found. Verify name in AWS console.")
    except client.exceptions.DecryptionFailure:
        raise RuntimeError(f"Failed to decrypt secret '{secret_name}'. Check KMS permissions.")

Permission Controls for TickDB

TickDB plans support different access levels. Assign keys based on role necessity:

Role	Recommended Plan	Key Permissions	Use Case
Researcher A	Free	Read-only kline, depth	Historical analysis, backtesting
Researcher B	Free	Read-only kline, depth, trades	Real-time feature extraction
Infrastructure (CI/CD)	Professional	Read-only + higher rate limits	Automated data pipelines
Admin	Enterprise (if needed)	Full read/write access	Account management, billing

Never assign more permissions than a role requires. If a researcher only reads kline data, their key should not have trades access. This limits blast radius if a key is compromised.

Audit Trail

Every TickDB API key should have an associated log entry in the team's secrets manager:

service,key_id,owner,created_date,rotation_date,last_used,purpose,permissions
TickDB,key_01,alice@team.com,2026-01-15,2026-07-15,2026-04-14,Primary research data access,Read kline,depth;no trades
TickDB,key_02,bob@team.com,2026-02-20,2026-08-20,2026-04-14,Backtesting pipeline,Read kline only
TickDB,key_03,carol@team.com,2026-03-10,2026-09-10,2026-04-13,Real-time monitoring,Read depth;no kline

Rotate keys every six months or immediately after a potential exposure. Document every rotation in the audit log.

---

Deployment Recommendations by Team Stage

Team stage	Priority 1	Priority 2	Priority 3
Just starting	Set up shared data registry	Configure 1Password vault	Initialize Git repo with branch protection
Running first backtests	Implement shared PostgreSQL	Standardize TickDB client	Add git_commit_hash to backtest results
Scaling to production	Implement CI/CD pipeline	Add rate-limit monitoring	Explore enterprise TickDB plan for higher quotas

Minimum Viable Stack (First 30 Days)

A three-person team can implement the core infrastructure within a month with the following time allocation:

• Week 1: Set up shared Git repository with branch protection, create data registry, provision shared PostgreSQL instance.
• Week 2: Deploy shared TickDB client to all team machines, configure 1Password vault, rotate all initial credentials.
• Week 3: Migrate existing scripts to repository, enforce code review process, document data ownership in registry.
• Week 4: Audit all existing credentials, establish rotation schedule, run first coordinated backtest using shared pipeline.

---

Closing: From Chaos to Coordination

The three failure modes — data fragmentation, code drift, and credential sprawl — are not technical problems that require sophisticated tools. They are coordination problems that require discipline and shared conventions.

The good news: a three-person team has a structural advantage over larger organizations. Fewer people means faster consensus, shorter review cycles, and fewer moving parts to maintain. The challenge is not scale — it is establishing the conventions early enough that they become invisible habits.

The architecture described in this article — a shared data registry, a standardized TickDB client, a Git-based collaboration workflow with review gates, and a secrets manager with audit trails — transforms an informal collaboration into a resilient shared infrastructure.

The four-hour backtest outage described at the opening would not happen in a team with this architecture. The stale data would be flagged by the registry, the configuration drift would be caught in code review, and the credential would be protected by a secrets manager with rotation policies.

Infrastructure is not overhead. It is the foundation that lets researchers focus on research instead of debugging each other's environments.

---

Next Steps

If you are an individual quant researcher transitioning to a team, start with the shared data registry and the Git repository. These two components cost nothing and prevent the most common collaboration failures.

If your team is already sharing scripts informally, audit the current state: count how many copies of data exist across machines, identify hardcoded credentials, and map every backtest to its data source and code version. The audit itself will reveal where the pain points are.

If you need enterprise-grade rate limits and historical data coverage for a team backtesting infrastructure, reach out to enterprise@tickdb.ai for plans designed for collaborative quantitative research environments.

If you use AI coding assistants, search for and install the tickdb-market-data SKILL in your AI tool's marketplace to streamline data pipeline generation directly from natural-language prompts.

---

This article does not constitute investment advice. Markets involve risk; past performance does not guarantee future results.