TechNotes :: Graceful Network Shutdown During a Power Outage — Automated.

Background:
Ideally, we all have a generator supporting our network and the UPS's role is to simply hold us over until the generator comes up to speed. It is not so ideal when the generator fails to start or runs out of fuel. In that case, our UPS is left to hold the load for just as long as it can. The procedure described in this document is our power failure backup plan and is what we want to occur when a UPS failure is inevitable. Our process will be to use the "Low Battery" alarm from the UPS to trigger an email to the administrator. A few moments later, the "Critical Battery" alarm will kick off our script to shutdown any set of servers that we choose.

No special software required! If you are working in a Windows Server 2003/2008 environment and your UPS has a communications connection to a server, then you have everything that you need. If you find that PowerChute doesn't work, then here's your no-cost workaround.

Required components:

• The appropriate Power Restoration setting in BIOS.
• A service account in Active Directory
• Windows Server 2003 or 2008 (includes all versions such as Small Business Server)
• Some very simple DOS Batch Scripts (provided later in the document)
• Blat email program (download here)

The appropriate Power Restoration setting in BIOS

Any plan to automatically shutdown a network had better give consideration for how the system will auto-start or your solution can get ugly very quickly. For this very reason we will want to visit the BIOS settings on every server that we'll be shutting down with our process. The power options, in most cases are ON, OFF, or LAST STATE. Selecting "ON" tells the device to go ahead and power up once it sees that power had been restored. You'll find that this works very well if you have your servers plugged into a power sequencing device that staggers the timing of when available power is allowed to return to each server.

A Service Account in Active Directory

In Active Directory, create a new User account and give it a meaningful name like "upsAgent". This account will be used to shutdown our servers, so it will be necessary to make this user a Domain Admin. I never use a service account for anything other than the single service that it is meant to control AND I give it a very complex password that does NOT expire. Keep this user name and password handy because we'll need them later.

Setting Power Options in Windows Server 2003 or 2008 (2003 shown)

If your UPS is not automatically recognized when you connect the USB or serial cable to your server, you may have to install the drivers that came with your UPS. As you'll see in the figure below, our APC units models connected in with very little fuss.

Go to Control Panel > Power Options and click on the Power Meter tab. Click on the icon. If you can see your UPS's information, your communications are functioning and you are ready to proceed.

You may close out the Power Options dialog box for now. We'll return once we have the scripts in place.

Some Very Simple DOS Batch Scripts

Our scripts are well documented so that even a beginner can find their way through them. We'll cover the parts of the scripts that are specific to your location. The remainder of the script will run as is.

You can install the files to any directory you want. For our purposes, we created a folder at:

     C:\Program Files\arzyNet\Utilities\

Right-click and save the downloads to the above folder.

• upsLow
• upsCritical

Once downloaded, right-click each ZIP file and select "Extract All...". Allow the extract to create a subfolder in our Utilities folder and expand the files into each. Our utilities folder should now have a upsLow and a upsCritical subfolder.

        utilities _
                    |_upsCritical
                    |_upsLow
        

If you haven't already downloaded and extracted Blat, you'll find that we have already included all of the Blat files that you'll need in the upsLow folder. When the UPS reports a "Low Battery" condition, we are going to call upsLow\upsBlat.bat to do our email notification for us. These are the four lines that need to be changed to match your environment. Right-click on the upsBlat.bat file and choose "Edit".

• set eTo=upsNotifications@yourDomain.com ← your user or group that should receive the alert
• set eFrm=upsAgent@yourDomain.com ← I like to use my service accounts ID
• set eSub=-s "yourDomain UPS Alert: Low Battery" ← keep your Subject inside the quotes
• set eSvr=-server "123.123.123.123" ← the IP address of your email server, use quotes

When finished, double-click your upsBlat.bat file to test. Use the blatLog.txt file to troubleshoot.

CAUTION: Be extremely careful when working with the upsNetShutdown.bat file. If you double-click on a completed or partially completed file, it will begin to shutdown servers!

When the UPS reports a "Critical Battery" condition, we are going to call \upsCritical\upsNetShutdown.bat to start shuting down servers. Let's look at the script and see how to manage our changes.

	:	Variables:
	:
	:	Set the path of the folder containing these scripts
	set upsPath=C:\Program Files\arzyNet\Utilities\upsCritical

	:	List all the servers to be shutdown (do not use quotes)
	:	Do not list THIS server as it will be done last
	set svr1=myFirstServer
	set svr2=mySecondServer
	REM out any un-needed variables
	REM set svr3=
	REM set svr4=
	REM set svr5=
	REM set svr6=
	REM ...and so on.		
		

As mentioned earlier, you can place your scripts anywhere on the server. We've set our path to match this documentation. Make adjustments to your upsPath, if necessary.

In my case, the first server that I want to shutdown is "rziis01", so I'll change the svr1 line to read:

set svr1=rziis01

I have two more servers to add, so I update those and remove the "REM" from the svr3 line. The final results look like the following code block. Remember to add in REM comments if a specific line is not used.

	:	Variables:
	:
	:	Set the path of the folder containing these scripts
	set upsPath=C:\Program Files\arzyNet\Utilities\upsCritical

	:	List all the servers to be shutdown (do not use quotes)
	:	Do not list THIS server as it will be done last
	set svr1=rziis01
	set svr2=rzsql01
	REM out any un-needed variables
	set svr3=rzarc01
	REM set svr4=
	REM set svr5=
	REM set svr6=
	REM ...and so on.		
			

Now that the servers have been listed. We have to adjust the "Calls" to perform the shutdown. If I have three servers listed, then I ought to have three "calls" to go with them. To do so, just add or remove the REM at the beginning of the line. You can add more lines to shutdown more servers, if you'd like. In my case, I have three servers, so my script shows three calls.

	call %upsPath%\upsSvrShutdown.bat %svr1%
	call %upsPath%\upsSvrShutdown.bat %svr2%
	REM out any un-needed server calls
	call %upsPath%\upsSvrShutdown.bat %svr3%
	REM call %upsPath%\upsSvrShutdown.bat %svr4%
	REM call %upsPath%\upsSvrShutdown.bat %svr5%
	REM call %upsPath%\upsSvrShutdown.bat %svr6%
	REM ...and so on.
			

Notice the last line in the script, "call %upsPath%\ upsSvrShutdown.bat %COMPUTERNAME%". This shuts down the server that's running the script. Appropriately, it's last.

No script is complete until fully tested. This will be a tricky one because testing WILL impact users if you start dropping servers in the middle of the day. I highly recommend testing after-hours. Likewise, you may want to adjust your scripts so that they only "call" one or two servers.

Going Live: the Final Setting

This final step brings together all of the pieces that we've built so far.

Go to Control Panel > Power Options and click on the Alarms tab.

Under the Low Battery Alarm section, click on the "Alarm Actions..." button.



Check "Run a program" and click on the "Configure program" button.

(1)Browse to locate your upsBlat.bat file, (2)Enter your UPS service account, and (3)Set your password.



Click OK without setting any other tabs.

Repeat these steps under the Critical Battery Alarm section, except you'll want to browse to and locate the upsCritical\upsNetShutdown.bat file. When you return to the Power Options settings, your Alarms folder should look like the figure below. Your power threshold sliders should be customized to meet your needs. Setting your Low Battery threshold higher might give you more time to react to a problem. Your Critical Battery slider may be set higher if you have a heavy load on the UPS or an Exchange server that takes a long time to shutdown.



Furthermore, you'll find some new events in your Scheduled Tasks.

One known minor risk:

This process assumes that a total power outage is inevitable and that each server will eventually see 0 VAC. Under these conditions, they will automatically start back up once power is reapplied. If full power were to be restored between the time that the servers have received their call to shutdown and the moment when zero power SHOULD have happened, then the servers may not start up automatically. This risk is minimal in comparison to the data that could have been completely lost without a graceful shutdown.

Cisco, Dell, Microsoft, and Symantec are registered trademarks.
© arzynet, 2010. All rights reserved.