Abnormal states are difficult to judge. I didn't have a good model of what is abnormal.

I would have liked to add a process that lurks, looks at the situation and yells murder if something weird happens. But what is weird? Post-mortems of error/recovery incidents always left me with yet another weird event for our log book.

I did collect statistics on performance to model throughput. Later production logs validated throughput estimates.

And speaking of weird