Testing whether a known market signal actually makes money

There's a well-known pattern in financial markets. If there are more buy orders than sell orders sitting on an exchange at a given moment, the price tends to tick up over the next few seconds. That much is settled and has been for years.

Whether it predicts well enough to be worth trading on is a different question, because trading isn't free. Every trade pays a fee and gives up a little on the gap between what buyers offer and what sellers ask. So the real question isn't "does this predict", it's "does it predict by more than it costs to act on".

I tested it on ten days of order book data from one cryptocurrency market. The answer is no.

Financial data will hand you a profitable-looking result if you keep asking. Try enough time horizons, enough ways of slicing the market, enough assumptions about cost, and something eventually clears. Every choice you make after seeing the data is a chance to nudge things toward what you were hoping for, usually without noticing you're doing it.

So the first thing I wrote wasn't code. It was the method: which horizons I'd test, how I'd define different market conditions, how I'd calculate costs, what would count as the signal being worth trading, and how I'd test whether any result was real rather than noise. I committed that as the very first thing in the project, before any analysis existed, so the order is checkable by anyone who looks.

I also wrote down all four possible outcomes in advance and said each was a valid finding, including "no effect" and "not enough data to tell". That's the part that matters. Committing to a method only counts if you've committed to publishing whatever it produces.

The exchange doesn't publish what its order book looked like at any given moment. It publishes one snapshot and then a continuous stream of changes, so the actual state at any point in time exists nowhere in the file. You have to rebuild it by replaying every update in order.

One misapplied update silently corrupts everything after it, and there's no correct answer anywhere to compare against. So I wrote a second, completely separate version of the rebuild and checked the two against each other at sampled points. Writing the same thing twice feels like wasted effort right up until it's the only way to know you got it right.

That produced 8.5 million rows from about seven gigabytes of raw feed.

I gave myself no favourable treatment anywhere. I assumed I'd always pay the full gap between buy and sell prices, on both entry and exit, and get no priority in the queue. Then I ran the entire analysis across four different fee levels rather than picking one, because picking one makes the answer a consequence of that choice.

Partway through, I found exactly what I'd set out to look for. In about a third of the cases I tested, the signal appeared to be profitable in some market conditions and not others. That was the whole point of the study, sitting right there.

Then I looked at why.

The measure I was ranking things by is roughly average profit, minus cost, divided by how volatile the market was. At realistic fees the cost is far larger than the profit, and the cost barely changes between one market condition and another. So the top of that fraction was essentially the same number everywhere, and what I was actually ranking was volatility. Nothing to do with profitability at all.

The giveaway is that the whole effect disappears if you set fees to zero, where the cost term nearly vanishes.

I wrote it up as an artefact rather than a finding. It would have been the most impressive-looking thing in the study.

The signal predicts. That part holds up cleanly, and its power fades the further ahead you look, which is what the theory says should happen.

It isn't tradable. The largest edge the model predicts anywhere in the held-back data is smaller than the cheapest realistic cost of making the trade. The rule I'd committed to in advance, trade only when expected profit exceeds cost, never fired once at any real fee level. That isn't a near miss where a better model closes the gap. The two distributions don't overlap.

The economics also came out backwards. The standard theory says this signal should work best in thin, jumpy markets where individual orders move the price more. Measured, it works worst there. Two separate parts of the analysis found that independently, which is the main reason I believe it.

I also tested a more sophisticated version using more layers of the order book. It lost to the simple one at every horizon, so it stayed out.

Ten days, one asset, in a fairly calm stretch of market. Nothing here says anything about stressed conditions or other instruments.

The data is snapshots roughly ten times a second rather than every individual event, so this is a medium-speed study and I've drawn no high-frequency conclusions from it.

And the market conditions I defined turned out to flip every few seconds, which makes them a short-lived state rather than a regime in the sense my own framing implied. That gap is real and I haven't resolved it.

A negative result, a contradicted theory, and the most interesting thing I found turning out to be an artefact of my own arithmetic. None of that is what I was hoping for.

The useful part is the order I did things in. The method existed before the data, so when the exciting result appeared I had no room to talk myself into it. I don't think I'd have caught it otherwise.

• GitHub

All projects