The biggest issue in my mind was how to test recovery code. It is difficult to fake network issues! We were lucky that the network group experience a bad week. This gave us hundreds of test points. It took a few days to document all the cases as an outcome of that week. As part of the documentation , we have those examples to train new support people in what network failures look like, and how the system recovers from them.