The most common cause of failure is an un-handled exception caused by unexpected format of data. When such a thing happens, human intervention is required, which involves restarting the application. What if we can get the application to notify us when this happens and then restart itself?

Notifications when there is an un-handled exception is a common practice. If you are not already doing it, you must be. All it takes is a few lines of code. In the event handler for un-handled exception, log the exception details and send an email out to the dev team (ensure through TDD that there can be no failures in the email sending part!!!). Going on to the next step, if the application restarts itself we can have a self healing application.

Since it is not possible to restart itself, we will use a helper. A second application that wraps around the main application. The helper wraps the main application in an App Domain and monitors for exceptions. All un-handled exceptions from the App Domain are caught, notified, logged and the application restarted. Here we can setup a new App Domain. So the application keeps running even during failures.

There will be situations when the problematic data keeps tripping the application and may keep failing over and again. Having a threshold of the number of consecutive fails helps escalating the issue. It can follow escalation by sending an email on the first fail, email with high priority for the second fail and an SMS for the third failure. At this point the wrapper application can wait for human intervention and not restart the application.

 

Namaste

Nrupal