Modeling Market Microstructure

There's a well-known paper called DeepLOB — Deep Convolutional Neural Networks for Limit Order Books (Zhang, Zohren & Roberts, 2019, arXiv:1808.03668). It puts a CNN-LSTM on top of a raw limit order book, classifies which way price moves next, and reports about 78% accuracy on an academic equities dataset.

Reproducing that number wasn't the point. I had a more selfish question: does any of this work for a retail trader? Take the same architecture, feed it an order book I scraped myself off a retail futures feed, ask it for an up / flat / down call I could put a bracket order on, and see whether the answer still means anything after you pay a spread and a commission.

So I changed almost everything around the model.

What I changed

Two of those rows do most of the damage.

The paper predicts a smoothed mid-price: it averages the next k ticks and compares that against the previous k. Smoothing makes the target well-behaved and heavily autocorrelated, which makes it much easier to hit. I used the raw mid-price change against a threshold, because you can't trade a smoothed mid. You get filled at the raw price. That swap makes the problem harder and drags the headline accuracy down, and it's most of why my numbers look worse than the paper's.

The other row is where the finish line sits. The paper stops at classification accuracy. Mine had to clear about three ticks of spread plus commission before it meant anything, which is a far less forgiving bar.

Getting the data

Nobody publishes an FI-2010 for NQ futures, so I had to go get the data myself. A collector on Fly.io held a WebSocket open to Tradovate through US market hours, pulled Level-2 DOM, batched the writes, flushed every 500 records or five seconds, and backed off whenever the feed rate-limited me (which it did, a lot). Everything landed in a TimescaleDB hypertable. Final haul: about 6.45M book snapshots over 26 trading days, late January to mid-February 2026, around 28 snapshots a second.

Two things about that data I shrugged off at the time and shouldn't have. The feed sends conflated snapshots, meaning aggregated size per level on a refresh cadence, so you never see individual orders or where you sit in the queue. And 26 patchy days is nothing next to the year the paper trained on.

The model

You hand the network the book as a picture: 100 timesteps × 40 features, where the 40 is 10 price levels times (ask price, ask size, bid price, bid size). Convolutions crawl over it. The first block uses a (1×2) stride-2 kernel to squash each price against its own volume, then (4×1) kernels run along the time axis. Two more blocks narrow things down to a single column of 32 channels. An Inception module fans out into three parallel convolutions (1×1, 3×1, and 5×1 with a pooling path) and glues them back together into 192 channels, which lets it look at a few time scales at once. That feeds an LSTM (192 → 64), and the last timestep goes through a small fully-connected head into three logits: down, flat, up. PyTorch, Adam, class-weighted cross-entropy so it doesn't just ignore the rare classes.

Defining the target

The three classes come from how far mid-price moves over some horizon, compared against a threshold I called epsilon. Move more than epsilon up, it's up. More than epsilon down, down. Anything in the middle is flat. Epsilon is a percentage so it scales with price instead of being stuck to a fixed number of ticks.

Picking epsilon is fiddly. Set it too small and you're labelling noise. Set it too big and everything turns into flat. I ran eighteen configurations hunting for an even three-way split, and 0.002% — roughly 0.42 NQ points at a 0.36-second horizon — got closest. Later I switched the horizon from "n snapshots ahead" to 60 seconds of wall-clock time, since what matters to a trader is how long they're sitting in the position, and book updates arrive at whatever rate they feel like.

What came back

Two experiments I'd keep:

• A horizon sweep. Predicting further out pushed accuracy from 39% to 42%, but macro-F1 didn't move. The model was quietly giving up on the flat class, its recall sliding toward 9%. Accuracy went up while the model got worse.
• A market-hours filter that turned out to be a null result. My collector had only ever run during US hours, so filtering to US hours removed exactly nothing, and the +0.9% was noise between runs. I'm keeping it here because I spent real time on it before I noticed.

Most runs came in between 33% and 44% out-of-sample accuracy, macro-F1 around 0.35, which is barely clear of just guessing the most common class. The loudest signal was the gap between training and testing: one run scored 27 points higher on data it had already seen. It had memorized a stretch of market.

What the results tell us

Getting to a number I trusted took a while. The feature set went from 188 engineered columns down to the paper's 40 raw ones. Epsilon got swept across eighteen configurations. Class weights were retuned more times than I want to admit, layers got reshuffled, and the labels moved from a snapshot count to wall-clock seconds. Two problems surfaced in the middle of all that: normalizing every snapshot on its own was wiping out the absolute price level and the trend — the exact thing I was asking the model to predict — and the feature layout was pairing ask prices with other ask prices instead of pairing each price with its own volume.

Once the data was finally going in the right way, the real question was still sitting there, so I built a clean experiment to answer it head-on: is there any signal in this order book at all?

I ran that one on a gradient-boosted baseline instead of the CNN-LSTM, on purpose. Separating "is there signal in the data" from "is my network any good" is much easier with a simple model, and if a boosted tree on correct features can't beat the base rate, a deep net isn't going to save it. Features were computed properly and causally this time — order-flow imbalance, book imbalance at depth 1, 5 and 10, micro-price deviation from mid — standardized on the training split only, with a temporal split and an embargo so nothing leaked across. The label was a triple-barrier one I could actually trade, and I swept round-trip costs across the whole thing.

There was something there. Direction is predictable at short horizons at roughly 55%, decaying to a coinflip by the 60-second mark. Statistically that's a real edge. Practically it's useless, because three ticks of spread plus commission eats it, and whatever survives that needs colocated HFT-grade execution in the microseconds. I have a laptop and a retail broker.

Then there's the number that got me excited for about a day. An early run hit 62% accuracy out-of-sample at a 0.36-second horizon — the best figure the project ever produced, at exactly the short timeframe where you'd hope to find something. It was hollow. The model had folded onto flat, which scored an F1 of 0.77 while up and down limped in at 0.16 and 0.13. On a three-class problem where one class dominates, you can score well by predicting the same thing forever. If I'd only looked at accuracy I'd have called it a win and gone shopping for a broker.

Two bigger walls sat behind all of it:

• A data ceiling. Twenty-six patchy days versus the paper's year, and conflated snapshots instead of real order-by-order data. The bigger problem: retail CME depth is licensed for display, not recording. There was no amount of money I could pay to legally grow that dataset. I shut the collector and the database down.
• The benchmark answers a different question. The paper's ~78% is real. It just answers "can you classify a smoothed mid-price," and that target is autocorrelated enough that guessing the previous label gets you most of the way there. It doesn't tell you whether the thing makes money on live NQ after costs. Both are fine questions. Only one of them pays rent.

So I shelved it. The architecture is fine. The question got answered, and the answer was no — you can run the exact network from the paper and still have no way to act on what it tells you, because the signal is thinner than the spread and the machine is too slow. Which is roughly where chapter two starts.

Chapter one settled one thing for me: I'm not going to out-predict the market's next move from a laptop. So this time I left price alone and modeled the two things I could actually get a handle on — the spread of outcomes a strategy produces, and whether the economics of the game are any good.

It's a local-first research platform for short-horizon order-flow strategies on CME futures, running on Databento's 10-deep (MBP-10) book data for NQ. Execution is manual on purpose. On a prop-firm evaluation account the broker API is locked down, so the system fires an alert and I place the order by hand. Every design decision downstream bends around that.

Strategies are just functions

A strategy is one method: on_book_update(snapshot) → Signal. No clocks, no IO, no randomness. Keeping it pure means the backtest and the live alert run literally the same code, so parity comes free instead of being something I have to babysit. There are eight signals in there, all microstructure rather than chart indicators: top-of-book size imbalance, distance-weighted depth imbalance, the Stoikov microprice, iceberg/absorption detection, queue depletion, liquidity vacuums, and cumulative-delta divergence.

The backtester assumes I'm wrong

The engine walks the book event by event. The part I care about is the fill model. Latency pushes every fill a few snapshots into the future, and slippage always goes against me. Limit orders only fill when price genuinely trades through them, and — this is the useful bit — the ones that miss get logged with the PnL they would have made. That gives me a number for adverse selection: how often I fill my losers and miss my winners.

Under all that, a columnar "DayBook" read path made loading a day of 16–31M book updates about 35× faster and let it run out-of-core. A 551-test suite checks that the streaming and parallel paths return byte-identical results to the simple one. Walk-forward validation reports how much worse things get out-of-sample, so curve-fitting shows up as a number instead of a feeling.

The statistics

This is where most of the project lives.

Monte Carlo. One backtest is a single sample. It tells you what happened, not what could have happened. So I bootstrap-resample the trade returns with replacement into 10,000 equity paths, then read percentile confidence intervals off them for both return and drawdown, plus a probability of ruin — the share of paths that punch through a drawdown floor. There's also a parametric mode that builds paths straight from a risk geometry (win rate, reward-to-risk) using a Bernoulli model and a moment-matched lognormal one. All hand-rolled NumPy. I wanted to write the statistics myself rather than import them.

Optimization. An exhaustive grid across parameters × exit policies × position sizing, hard-capped so it can't explode into a week-long job. Each cell gets scored on pass-rate and net expected value. Two guards watch for overfitting: walk-forward degradation, and a "knife-edge" check that flags a winning cell when it's a lonely island — a big drop to its neighbours, or almost nothing else within 10% of its score.

My favourite part: the account is a call option

The prop-firm model is the piece I like most. A funded-trading challenge is a convex, capped-loss payoff. Blow the evaluation and you're out the fee, not the drawdown — that money was never yours. Pass it and the upside comes back as payouts. Which leads somewhere counterintuitive: a strategy with roughly zero per-trade edge can still be net EV positive across the whole challenge-to-funded lifecycle. Judging it on alpha alone misses the point.

I modeled the real rule sets by name, including Apex-style floors that lock at the starting balance and MyFundedFutures-style floors that lock on your first payout. Passing is a barrier problem: get to the profit target before you touch the trailing drawdown floor. In that framing, low trade-to-trade variance does more for you than a big expectancy does. A structured-product simulator puts it together — expected attempts at 1/P(pass), expected cost to get funded, net EV, ROI, and the full payout distribution — driven by the real L2 trade distribution and haircut for execution realism.

Size of the thing

About 23,000 lines of typed Python for the core, another 13,000 for a Next.js dashboard with Monte Carlo fan charts, funded-account curves with payout markers, and optimizer heatmaps. Around 3.9 GB of cached NQ order-book Parquet, and those 551 tests. No trading library anywhere in it — the engine, the statistics and the payoff model are all built from scratch, which was most of the fun.

Tying it back to chapter one: I stopped guessing where price goes, and started paying attention to risk asymmetry, being picky about setups, and understanding the shape of the payoff I'm being handed.