Understand, I used to work in financial markets. I was part of the team that built up Morgan Stanley's technology platform for international trading in the 80s and early 90s. I know the level of caution and care that is essential in order to avoid losing critical systems during the week.

Every change was well planned out. For the most part they happened on Friday night so we could have the whole weekend for validation and backout if necessary. We always had backout plans (in fact, backouts were automated if needed). Short of an emergency, we never did anything on a weekday morning during a volatile week.

I recall one big change we did to update how we managed some of our key data. We ran the "new" system in parallel for weeks. We shut down all other changes to any of our systems for the week of the system upgrade. We had all hands on deck all weekend to validate everything. We did our Friday night production runs TWICE, once under the old system, a second time using the new, then we compared everything.

After over 24 hours of going non-stop, with many people sleeping in the office, a discrepancy was found. The VP in charge called for us to begin preparation for a backout.

Fortunately, somebody decided to take a closer look at the program that was at the source of the discrepancy and found that despite the best precautions, a programmer has managed to get a change through by using somebody else's password and despite the absolute prohibition on any such changes for that week.

So, with the deadline for starting the backout process looming, we backed out his change, re-ran everything and it worked. Disaster averted.

I don't recall what happened to that programmer or to the person who had given him a password he shouldn't have had, apparently allowing him to regularly bypass all controls. I seriously doubt he was employed for long though. Screwing up something like that is just not acceptable in those situations.

Yet today, Schwab did just that. They decided that they needed to upgrade systems on a Thursday morning, with apparently no real backout plan in place, in order to handle the recent levels of volume and volatility. Sorry, but that should have been last weekend's business. Or if they couldn't get the plans, hardware and people together for last weekend, then they could have waited another 36 hours. And if thing were truly so critical that it couldn't wait, then somebody needs to answer for why that happened.

In the meantime, Schwab just lost a nice big chunk of my change. Over the coming weeks they'll lose more. (I don't want to have too much of my money in transit all at once.) They showed me how they manage operational risk and they lost my trust. It'll take a long, long time to regain it.

Sorry Chuck.

-btc