Page Two

Processing a major exchange feed as a consistency problem.

This page treats a major exchange market data feed as an admissibility pipeline: bootstrap the book, apply incrementals, measure sequence and lifecycle defects, then decompose the residual into structural causes instead of a single black-box alert. The worked lab stays closest to a T7 or EOBI-style tape, but the framing is intended to carry across other major feed families as well.

Grounding

What A Major Venue Feed Looks Like Here

The concrete event tape on this page is closest to Deutsche Börse T7 and EOBI, but the consistency framing is meant to generalize across major venue protocols rather than only one exchange implementation.

Recovery + Live Flow

Major venues expose some combination of snapshot, replay, or recovery channels plus a live incremental stream. The exact mechanism differs, but the reconstruction problem is the same.

Depth Model Differences

Some feeds are order-by-order, others are price-level oriented, and some split top-of-book, full depth, and trade channels. The consistency object has to respect whichever state model the venue publishes.

Sequencing And Lifecycle Semantics

Add, modify, delete, execute, state-change, and sequencing semantics recur across the major feed families even though field names and wire formats differ.

Deutsche Börse T7 / EOBI

A strong order-by-order reference model for visible depth, sequencing, and stateful book maintenance. This remains the closest concrete anchor for the interactive tape.

Nasdaq TotalView-ITCH

A major equity depth feed family centered on order-level updates, executions, and recovery logic that makes it a natural comparison point for consistency processing.

NYSE Pillar

Pillar integrates depth and exchange-state semantics in a way that fits the same residual view: sequencing, instrument state, and book-validity checks remain central.

Cboe PITCH / Depth

Cboe market data products differ in detail, but the same admissibility questions appear around ordering, lifecycle transitions, and depth reconstruction.

CME MDP 3.0

Futures and options feeds bring a different venue context, but sequence, state, and book-maintenance defect still decompose naturally into the same structural categories.

ICE iMpact

ICE-style feeds add another major venue family where feed quality, recovery, and order-book validity matter operationally rather than only analytically.

Why Here

Why The Use Case Is Strong

Order-book processing is unusually well suited to a consistency object because the feed is dense with structural rules, recovery logic, and states that are mechanically meaningful rather than merely statistical.

There Is A Real Admissible State

A major-venue book is not arbitrary data. It has visible depth, queue or level structure, product state, instrument state, and sequencing rules that define what a coherent maintained book can be.

Failures Have Distinct Shapes

A packet gap, a bad delete, an impossible execution, and a crossed book are all different failures. A decomposed residual can keep them separate.

The Same Layer Serves Multiple Jobs

One consistency layer can support feed monitoring, book reconstruction, simulator QA, and surveillance instead of forcing a different logic stack for each.

Standard Feed Handler
  • Receive UDP.
  • Parse bytes into messages.
  • Update maps and price levels.
  • Recover only when something obviously breaks.
With Consistency Layer
  • Receive and parse as usual.
  • Score sequencing, state, identity, and execution defect continuously.
  • Explain why the maintained book is drifting from validity.
  • Trigger recovery or alerting based on structural failure, not only crashes.
Pipeline

Book Reconstruction As Consistency Processing

The feed handler does not just decode messages. It constructs a state that should stay near an admissible manifold defined by market mechanics and feed semantics, whether the concrete venue is T7, ITCH, Pillar, PITCH, MDP, or a similar book feed.

1

Bootstrap

Load reference data and an initial snapshot. Set the last processed message sequence.

2

Decode

Read incremental messages: add, modify, delete, partial execution, full execution.

3

Apply

Mutate the maintained book and active-order index using feed semantics.

4

Score

Measure sequence, lifecycle, quantity, and price-order defects after each step.

5

Explain

Expose a residual vector, not just an alert, so a human can see which mechanism broke.

Operational Uses

What This Could Be Used For In Practice

The use case is broader than “detect spoofing.” A processor for a major venue feed, with a consistency layer attached, becomes a general-purpose validity engine for the market data plant.

Feed Quality Monitoring

Track packet gaps, per-product message sequence gaps, malformed transitions, and bad recovery states before they propagate into downstream analytics.

Snapshot Recovery

Use the residual to decide when the maintained book has drifted too far from admissibility and a snapshot rebuild should be triggered.

Market Surveillance

Turn suspicious behavior into defect signatures: repeated add-delete patterns, non-contributory size, or unusual mismatch between displayed liquidity and execution.

Strategy Guardrails

Reject internal signals that are being computed from an inconsistent maintained book rather than from structurally trustworthy market state.

Simulation And Replay QA

Verify whether replay engines, simulators, and historical reconstructions preserve the same consistency relations as the live system.

Research Feature Extraction

Use total residual and per-defect components as features for downstream models without giving up structural interpretability.

Stage C Result

The First Useful Difference Is Already Visible

On the current 23-event normalized replay, the consistency layer now does more than rename the same checks. Under proxy snapshot rebuild simulation it changes when to rebuild and reduces the amount of time spent in inconsistent state.

Residual Exposure
29.1

Consistency policy cumulative residual, versus 35.1 for the naive rebuild rule.

Inconsistent Events
6 vs 7

Same rebuild count, but one fewer inconsistent event across the replay.

Late False Rebuild
0 vs 1

The consistency policy avoids the late rebuild on the tiny crossed-book event at 21.

Naive Rebuild Path
  • Triggers at events 10, 12, and 21.
  • Ignores the repeated auction-state drift at events 15 and 16.
  • Rebuilds on the tiny crossed ask, then turns the next delete into a missing-order residual.
Consistency Rebuild Path
  • Triggers at events 10, 12, and 16.
  • Uses persistent moderate residual to detect drift before the late crossed-book cleanup.
  • Clears the next event twice and avoids the unnecessary final rebuild.

Residual Trace Driving The Decisions

Stage B residual trace showing defect spikes and rebuild trigger markers

The figure shows the base residual trace. The rebuild comparison uses this trace plus a proxy authoritative snapshot model to test what each policy would do after each trigger.

Why This Matters

  • It is the first result where the consistency layer changes operational behavior, not just labeling.
  • It shows why persistent moderate drift deserves its own trigger logic.
  • It suggests rebuild policy can be driven by structural exposure, not only by crashes.
  • It gives a concrete path from theory to feed-quality monitoring and recovery policy.
The rebuild step is still a proxy model, not real exchange snapshot recovery. The point of this stage is comparative: does the policy make a better decision on the same tape?
Application Equations

Consistency Structure For The Order Book

One possible state and residual design for an exchange feed is shown below. The exact weighting is illustrative; in real work it would be calibrated, normalized, and tied to the exact venue semantics. The interactive tape still uses a T7-anchored event model.

Xt = (Ot, Lt, sfeed,t, smarket,t) active orders, price levels, feed sequencing state, product/instrument state C(Xt) = (cpkt, cmsg, cid, cqty, cpx, cstate) Rt = 3cpkt + 4cmsg + 6cid + 5cqty + 30cpx + 4cstate
  • cpkt: packet-sequence defect on the multicast channel.
  • cmsg: per-product message-sequence defect.
  • cid: modify/delete/execute references a missing visible order key.
  • cqty: execution exceeds remaining displayed quantity.
  • cpx: best bid rises above best ask after processing.
  • cstate: message is inconsistent with product or instrument trading state.
In a real venue implementation, the identity key, state model, and recovery semantics depend on the feed family. In the T7-anchored example here, order identity is tied to instrument, side, and priority timestamp, and snapshot synchronization uses LastMsgSeqNumProcessed.
Protocol Fidelity

What A Real Venue Integration Requires

The lab is intentionally lightweight, but the real use case becomes stronger when the consistency structure is attached to actual venue semantics rather than to an abstract event list. T7 is the concrete example, not the only target.

Venue-Native Parsing

Real feeds are carried in venue-specific binary or structured wire formats and must be decoded with exact packet and message semantics before consistency logic can help.

Dual Sequencing

Packet sequence numbers track the channel; message sequence numbers track the product. Both can contribute separate residual components.

Snapshot / Incremental Sync

The maintained book should be rebuilt from the venue's recovery path and then reconciled against the live stream using the appropriate last-processed sequence semantics.

Identity Model

Some feeds give direct order identity, some emphasize price levels, and some blend state channels. The maintained key structure has to match the venue model exactly.

Execution Correctness

Partial and full execution messages are the authoritative book-maintenance path for visible passive orders and should dominate quantity defect logic.

State-Aware Validity

During auction or other trading states, depth publication rules change, so admissibility has to be conditioned on product and instrument state as well.

Interactive Lab

Step Through A T7-Anchored Event Tape

The tape starts from a clean snapshot and then introduces realistic T7-like feed events plus a few deliberately broken ones: a delete on a missing order, a sequence gap, and an over-execution. The point is to show the consistency logic on one concrete feed family while keeping the framing broader.

Total Residual
0.00
Best Bid / Ask
99.70 / 100.20

Updated from the maintained book.

Last Sequence
snapshot: 1000

Sequence gaps contribute to Rseq.

Residual Breakdown

Latest Diagnostics

Aggregated Book

Bid Qty Bid Px Ask Px Ask Qty

Active Orders

Order Side Px Qty
Decoder Sketch

What A Real Feed Handler Could Look Like

The toy lab above uses JSON-like events. A production pipeline would decode the venue-native feed, synchronize recovery and live updates, maintain the exact venue identity model, and persist defect traces for recovery and surveillance. The pseudocode below stays closest to T7 and EOBI so one concrete wire model is visible.

def order_key(msg):
    return (msg.security_id, msg.side, msg.priority_ts)

def process_datagram(raw, state, structure):
    packet = parse_packet_header(raw)      # little-endian EOBI packet header
    defect = {"pkt": 0, "msg": 0, "id": 0, "qty": 0, "px": 0, "state": 0}

    if packet.seq_num != state.next_packet_seq:
        defect["pkt"] += abs(packet.seq_num - state.next_packet_seq)

    offset = packet.header_len
    while offset < packet.packet_len:
        header = parse_message_header(raw, offset)
        body = raw[offset:offset + header.body_len]
        msg = decode_message(header.template_id, body)
        apply_eobi_message(state, msg, defect)
        state.next_msg_seq[msg.product_id] = header.msg_seq_num + 1
        offset += header.body_len

    defect["px"] += max(0.0, best_bid(state) - best_ask(state))
    return structure.residual_from_defect(defect)
def apply_eobi_message(state, msg, defect):
    key = order_key(msg) if hasattr(msg, "priority_ts") else None

    if msg.template_id == 13100:  # Order Add
        state.orders[key] = msg
    elif msg.template_id in (13101, 13106):  # Modify
        prev_key = (msg.security_id, msg.side, msg.prev_priority_ts)
        if prev_key not in state.orders:
            defect["id"] += 1
            return
        state.orders.pop(prev_key)
        state.orders[key] = msg
    elif msg.template_id == 13102:  # Delete
        if state.orders.pop(key, None) is None:
            defect["id"] += 1
    elif msg.template_id == 13105:  # Partial execution
        order = state.orders.get(key)
        if order is None:
            defect["id"] += 1
            return
        if msg.last_qty > order.display_qty:
            defect["qty"] += msg.last_qty - order.display_qty
        order.display_qty = max(0, order.display_qty - msg.last_qty)
    elif msg.template_id == 13104:  # Full execution
        if state.orders.pop(key, None) is None:
            defect["id"] += 1