Monday, June 7, 2010

How To Trash An Essbase Database (and how to fix it)

I was stress testing some new Essbase clustering functionality that we put into Dodeca this weekend and decided to let it run on my laptop for a couple of hours while I relaxed at home. However, I had forgotten that the new battery on my laptop seems to act strangely and not alert my system that it is running out of juice and, as a result, it just immediately turns off.. This is definitely *not* what you want to do to Essbase, especially when you are changing the configuration.

In my testing, I wrote a little C# routine that retrieved data into a grid every 1/2 second. The application also had a background thread that was running a maxl script every 3 seconds to turn off one active database in my cluster and start the other database in the cluster. I doubt anyone would need to swap clusters this fast, but I wanted to make sure it worked, so I wrote the test routine.

It just so happened that my machine quit in the middle of one of these maxl runs. I didn't think anything of it and put my laptop away until this morning when I noticed my Essbase service wasn't running. When I tried to start it, it immediately shut down. I started Essbase in a command window and it looked like it was working, but it didn't behave properly when I tried to connect to it.

Apparently, several files got trashed when the laptop ran out of power. To get everything back to normal, I basically had to create a new application and copy the files from the old application over to the new application. Not very pretty but it worked. Thank goodness it was only my laptop system and not a production system.

Lesson to be learned.. Make sure your production systems have good UPS's or other backup power behind them.

No comments: