Recovery + Live Flow
Major venues expose some combination of snapshot, replay, or recovery channels plus a live incremental stream. The exact mechanism differs, but the reconstruction problem is the same.
This page treats a major exchange market data feed as an admissibility pipeline: bootstrap the book, apply incrementals, measure sequence and lifecycle defects, then decompose the residual into structural causes instead of a single black-box alert. The worked lab stays closest to a T7 or EOBI-style tape, but the framing is intended to carry across other major feed families as well.
The concrete event tape on this page is closest to Deutsche Börse T7 and EOBI, but the consistency framing is meant to generalize across major venue protocols rather than only one exchange implementation.
Major venues expose some combination of snapshot, replay, or recovery channels plus a live incremental stream. The exact mechanism differs, but the reconstruction problem is the same.
Some feeds are order-by-order, others are price-level oriented, and some split top-of-book, full depth, and trade channels. The consistency object has to respect whichever state model the venue publishes.
Add, modify, delete, execute, state-change, and sequencing semantics recur across the major feed families even though field names and wire formats differ.
A strong order-by-order reference model for visible depth, sequencing, and stateful book maintenance. This remains the closest concrete anchor for the interactive tape.
A major equity depth feed family centered on order-level updates, executions, and recovery logic that makes it a natural comparison point for consistency processing.
Pillar integrates depth and exchange-state semantics in a way that fits the same residual view: sequencing, instrument state, and book-validity checks remain central.
Cboe market data products differ in detail, but the same admissibility questions appear around ordering, lifecycle transitions, and depth reconstruction.
Futures and options feeds bring a different venue context, but sequence, state, and book-maintenance defect still decompose naturally into the same structural categories.
ICE-style feeds add another major venue family where feed quality, recovery, and order-book validity matter operationally rather than only analytically.
Order-book processing is unusually well suited to a consistency object because the feed is dense with structural rules, recovery logic, and states that are mechanically meaningful rather than merely statistical.
A major-venue book is not arbitrary data. It has visible depth, queue or level structure, product state, instrument state, and sequencing rules that define what a coherent maintained book can be.
A packet gap, a bad delete, an impossible execution, and a crossed book are all different failures. A decomposed residual can keep them separate.
One consistency layer can support feed monitoring, book reconstruction, simulator QA, and surveillance instead of forcing a different logic stack for each.
The feed handler does not just decode messages. It constructs a state that should stay near an admissible manifold defined by market mechanics and feed semantics, whether the concrete venue is T7, ITCH, Pillar, PITCH, MDP, or a similar book feed.
Load reference data and an initial snapshot. Set the last processed message sequence.
Read incremental messages: add, modify, delete, partial execution, full execution.
Mutate the maintained book and active-order index using feed semantics.
Measure sequence, lifecycle, quantity, and price-order defects after each step.
Expose a residual vector, not just an alert, so a human can see which mechanism broke.
The use case is broader than “detect spoofing.” A processor for a major venue feed, with a consistency layer attached, becomes a general-purpose validity engine for the market data plant.
Track packet gaps, per-product message sequence gaps, malformed transitions, and bad recovery states before they propagate into downstream analytics.
Use the residual to decide when the maintained book has drifted too far from admissibility and a snapshot rebuild should be triggered.
Turn suspicious behavior into defect signatures: repeated add-delete patterns, non-contributory size, or unusual mismatch between displayed liquidity and execution.
Reject internal signals that are being computed from an inconsistent maintained book rather than from structurally trustworthy market state.
Verify whether replay engines, simulators, and historical reconstructions preserve the same consistency relations as the live system.
Use total residual and per-defect components as features for downstream models without giving up structural interpretability.
On the current 23-event normalized replay, the consistency layer now does more than rename the same checks. Under proxy snapshot rebuild simulation it changes when to rebuild and reduces the amount of time spent in inconsistent state.
Consistency policy cumulative residual, versus 35.1 for the naive
rebuild rule.
Same rebuild count, but one fewer inconsistent event across the replay.
The consistency policy avoids the late rebuild on the tiny crossed-book event at
21.
10, 12, and 21.15 and 16.10, 12, and 16.The figure shows the base residual trace. The rebuild comparison uses this trace plus a proxy authoritative snapshot model to test what each policy would do after each trigger.
One possible state and residual design for an exchange feed is shown below. The exact weighting is illustrative; in real work it would be calibrated, normalized, and tied to the exact venue semantics. The interactive tape still uses a T7-anchored event model.
The lab is intentionally lightweight, but the real use case becomes stronger when the consistency structure is attached to actual venue semantics rather than to an abstract event list. T7 is the concrete example, not the only target.
Real feeds are carried in venue-specific binary or structured wire formats and must be decoded with exact packet and message semantics before consistency logic can help.
Packet sequence numbers track the channel; message sequence numbers track the product. Both can contribute separate residual components.
The maintained book should be rebuilt from the venue's recovery path and then reconciled against the live stream using the appropriate last-processed sequence semantics.
Some feeds give direct order identity, some emphasize price levels, and some blend state channels. The maintained key structure has to match the venue model exactly.
Partial and full execution messages are the authoritative book-maintenance path for visible passive orders and should dominate quantity defect logic.
During auction or other trading states, depth publication rules change, so admissibility has to be conditioned on product and instrument state as well.
The tape starts from a clean snapshot and then introduces realistic T7-like feed events plus a few deliberately broken ones: a delete on a missing order, a sequence gap, and an over-execution. The point is to show the consistency logic on one concrete feed family while keeping the framing broader.
Updated from the maintained book.
Sequence gaps contribute to Rseq.
| Bid Qty | Bid Px | Ask Px | Ask Qty |
|---|
| Order | Side | Px | Qty |
|---|
The toy lab above uses JSON-like events. A production pipeline would decode the venue-native feed, synchronize recovery and live updates, maintain the exact venue identity model, and persist defect traces for recovery and surveillance. The pseudocode below stays closest to T7 and EOBI so one concrete wire model is visible.
def order_key(msg):
return (msg.security_id, msg.side, msg.priority_ts)
def process_datagram(raw, state, structure):
packet = parse_packet_header(raw) # little-endian EOBI packet header
defect = {"pkt": 0, "msg": 0, "id": 0, "qty": 0, "px": 0, "state": 0}
if packet.seq_num != state.next_packet_seq:
defect["pkt"] += abs(packet.seq_num - state.next_packet_seq)
offset = packet.header_len
while offset < packet.packet_len:
header = parse_message_header(raw, offset)
body = raw[offset:offset + header.body_len]
msg = decode_message(header.template_id, body)
apply_eobi_message(state, msg, defect)
state.next_msg_seq[msg.product_id] = header.msg_seq_num + 1
offset += header.body_len
defect["px"] += max(0.0, best_bid(state) - best_ask(state))
return structure.residual_from_defect(defect)
def apply_eobi_message(state, msg, defect):
key = order_key(msg) if hasattr(msg, "priority_ts") else None
if msg.template_id == 13100: # Order Add
state.orders[key] = msg
elif msg.template_id in (13101, 13106): # Modify
prev_key = (msg.security_id, msg.side, msg.prev_priority_ts)
if prev_key not in state.orders:
defect["id"] += 1
return
state.orders.pop(prev_key)
state.orders[key] = msg
elif msg.template_id == 13102: # Delete
if state.orders.pop(key, None) is None:
defect["id"] += 1
elif msg.template_id == 13105: # Partial execution
order = state.orders.get(key)
if order is None:
defect["id"] += 1
return
if msg.last_qty > order.display_qty:
defect["qty"] += msg.last_qty - order.display_qty
order.display_qty = max(0, order.display_qty - msg.last_qty)
elif msg.template_id == 13104: # Full execution
if state.orders.pop(key, None) is None:
defect["id"] += 1