Compare

How Compare is doing

The pick, graded

Reference backtest, not live results yet

Reference backtest: 2013 departures from JFK, LGA and EWR (public nycflights13 data), July to December, same-day same-route pairs.

2013, NYC departures only, no rotation features, 60-day history window. Order-of-magnitude anchors, not results for current flights.

838,961

same-day, same-route pairs

68% had enough history

30%

of pairs where the pick mattered

61% both fine · 10% both bad

Accuracy by picker

Was the pick right?

Among pairs where exactly one flight went bad, how often each picker chose the good one. A coin should land near 50%.

  • Coin flip50.0%

    Bad trips 25% when following it vs 25% for the other flight

  • Earlier departure67.1%

    Bad trips 20% when following it vs 30% for the other flight

  • 60-day on-time history68.8%

    Bad trips 19% when following it vs 30% for the other flight · +1.7 pts over earlier departure

  • History, abstaining under a 10-point gap75.8%

    Bad trips 19% when following it vs 30% for the other flight · picks on 56% of pairs · +2.6 pts over earlier on the same pairs

Why “landed closer to schedule” isn’t the headline

Counting ties as wins, the deck’s metric gives the coin flip 70.5%, earlier departure 77.1% and history 78.2%. Most pairs land fine either way, so ties flatter everyone. The bar is lift over “take the earlier flight,” not 65%.

Abstaining helps

Raise the gap a picker needs before it commits, and the picks it still makes get more accurate. Compare calls everything under 10 points “too close to call.”

History picker Earlier departure Coin flip
50%60%70%80%90%14%29%56%100%share of pairs where a pick is made≥30 pts≥20 pts≥10 ptsany gap
Show the numbers
Abstain underCoverageHistoryEarlier
30 pts14%84.6%83.7%
20 pts29%80.9%79.1%
10 pts56%75.8%73.2%
never100%68.8%67.1%

Beyond time of day

22%

of pairs where history and “take the earlier flight” disagree

54.2%

history’s accuracy on those pairs

barely above a coin

History agrees with “earlier is better” on most pairs (68.8% vs 67.1% overall). The real test for Compare is whether rotation and congestion features beat the coin on the disagreements. That is what the live grading will measure.

Self-grading

The guardrail

  • It grades itself. After both flights land, Compare checks whether its pick had the better trip, every time. Most features can’t tell if they worked; this one can.
  • If it’s wrong too often, it stops picking. Under 65% right on the pairs that mattered, Compare shows both sets of odds and no verdict.
  • Today’s rule. Within 10 points of each other: too close to call. Under 20 recent flights: no pick. Model history-2026-07.

Live grading begins when tracked compares land. Until then this page shows the reference backtest. How Compare works ›