Hindsight

What it is

A simulator that takes a proposed automation, replays a company's own history through it, and reports what it would have done differently, where it would have been wrong, and what being wrong would have cost. Before a line of the real thing exists.

Why we built it

This is our own argument taken to its end. We will not build where the evidence does not support it, and the honest way to know that in advance is to run the thing against reality first.

What we learned

1. 01

   History is not a clean test set. The record shows what people did, not what they should have done, and separating a decision from an uncaught mistake is most of the work.

2. 02

   The counterfactual is the hard part. Once the automation would have acted differently, the rest of the recorded history stops being valid.

3. 03

   The cost of an error dominates the rate of them. Wrong twice a year can be far worse than wrong twice a week, and almost nothing is evaluated that way.