Archive | SCOM

Russ Slaten is an MP authoring expert and that created a custom URL monitoring solution.  This is my favorite solution as it addresses one of the most common issues that I have often heard from customers.  “Don’t alert me if the website is down for only one second.  Alert me if the website had be down for say five minutes”.  The solution is a little more difficult to setup initially but once you have it working adding and removing website is as easy as updating an excel spreadsheet.  You can even have the spreadsheet sit on a common file server so that people who have no clue how SCOM works can setup URL monitors.

I recommend installing this in your test lab first to get familiar with how it works.

So lets get started.  First I download the Management Pack.

Custom.Example.WebsiteWatcher.zip

**Note an updated the solution since this post to target Resource Pools in SCOM 2012**   Most of the configuration steps documented here still apply. 

Download Updated MP Supporting Resouce pools:  URLMonitoring.zip

I extract the files to see what’s in the zip file

There are two files in the zip package.  The first one is the management pack.  The second one is the CSV file for listing website to be monitored.

Now lets take a look at the management pack in MP Viewer

There are two classes.  The Website class and the Website Watcher Class

There are two discoveries.  One discovers which servers will be watcher nodes.  The other discovers what website to monitor.

There are three monitors. One “monitors whether a website is up or down”.  The second one “checks for a string in the response from an http request”.  The third one “Monitors the Performance of a Website”.  Seems easy enough

Also there is one relationship that ties the websites to the watcher nodes.

Let get this puppy installed.

I open the administration console and go Management Packs.  Right click and select Import Management Packs.

Then I go to Add and select Add from disk.  Click no to search for dependencies.

Now I browse to the folder that I extracted the files to and select the “Custom.Example.WebsiteWatcher.xml” file

I click Install,  and when its finished click Close

I have the Management Pack installed. Now I need to configure it for my environment. To keep it simple I am going to configure all my websites to be monitored from my RMS server.  This way I can avoid messing with run as accounts.

First I am going to copy the CSV file to a simple location on the RMS.  I created a folder on the RMS called c:\websitemonitoring and copied the “URLMonitoringList.csv” file there.

I open up the Operations Manager console and go to the Authoring Console and select Object Discoveries

Now I need to change my scope

I select Clear All at the bottom and then select View all targets.

Now I type in Website Watcher and click Select All then ok

As I seen before with the MP Viewer tool there are two discoveries in this management pack

I double click on the Discover Websites Discovery

I go to the configuration tab and I can see the discovery contains an interval on how often websites are discovered and a path to the file.

I click the edit button and copy and paste the location of my URLMonitoringList.csv which is “C:\websitemonitoring\URLMonitoringList.csv”

I am also going to change the interval to every 300 seconds for testing purposes.  Once I have my everything working I will change it back to one day or 86400 seconds

I click Ok and then Ok again.

Now I need to configure what server I will be monitoring the websites from.  In this case I will just be using the RMS.

I remote desktop into into the RMS and create a registry Key (not value) in HKLM\Software called “WebsiteWatcher”

I go the registry by typing regedit

Navigate to HKEY_LOCALMACHINE\Software

I right click and select New, Key

Then type in “WebsiteWatcher”

The discovery for this is set at once a day, while on the RMS I also cycled the System Center Management Services to speed up the discovery process. (I don’t recommend this in production)

Now I need to open the URLMonitoringList.csv and edit with excel. I set second column to my RMS server and save it.

Now I go to the Monitoring console and go to the Discovered Inventory view and select View All target and type in website.

I can see all the website have been discovered and are now being monitored.

The Bulk URL Editor was introduced in SCOM 2007 R2.  I don’t often see this tool used as most customers don’t even know it exists or don’t understand the benefits of it.  The first benefit of the Bulk URL editor is that it scales to thousand of URLs.  If you were to try to create hundreds of URLs with the Web Application Templates it won’t work.  I have tried this in the past and there are so many workflows running at the same time that the agent fails and you end of not monitoring anything.   The second benefit of the tool is that you can add a bunch of websites in a few minutes. 

The Bulk URL Editor is not very intuitive, but once you understand how to use it the process is pretty easy.  If you haven’t used the tool I highly recommend giving it a try.

TechNet has some good documentation here. http://technet.microsoft.com/en-us/library/dd788987.aspx

To use the Bulk URL Editor I copy the tool from the installation media.  The file is stored in the “SupportTools\AMD64” directory

On my computer that has the SCOM console installed, I copy the “BulkUrlManager.exe” file to “C:\Program Files\System Center Operations Manager 2007” (If you copy it anywhere else it won’t work)

I double click on the “BulkUrlManager.exe” file.

On the Connect to Server dialog box I type in the name of my Root Management Server and click connect

I click the New Icon

I then type in a name of my website template.  I choose “Standard URL Monitoring”

Now I click Create a new Management Pack,

I then give the management pack the name. I choose “BUE Website Monitoring” and click OK

I click OK on the Add New Template Screen

On the next dialog box I click Yes, then OK

Under Templates I click the template I created called “Standard URL Monitoring”

Now I click Add

Now I simply add the URLs that I want to monitor (Note: You need to add http:// or https:// or it will fail)

I click OK and I see all of my URLs are attached to my Standard URL Monitoring Template

Now I simply hit save. I click yes to save the changes to the selected web template.

I am done in the Bulk URL Editor for now. But I am not finished setting up my URL monitoring.  I need to select where I want the URLs to be monitored from.

I launch the Operations Manager Console and go to the Authoring screen.

I expand out Web Application and right click to refresh the screen.

Now I see the website I created using the Bulk URL editor

Under the Actions pane, Custom Actions I click Edit web application settings

My website opens and all I see is a string of text that looks like a ugly variable.  (Don’t panic this is how it is supposed to look)

Now under the Actions Pane, under Web Application I click Configure settings

I click the Watcher Node tab and select the server I want to run the website monitoring.  I choose my RMS server.

I click OK and then click Apply at the bottom of the Web Application Editor screen

I then close the screen with the red X at the top right

Now I go back into the Bulk URL Editor.

I select the Template I have been working with and hit Synchronize

I click Yes,

I close the Bulk URL Editor as I am done with it.

Now I open the SCOM Monitoring Console and look for our Web Application, Standard URL Monitoring Instances.

I can see all my websites are now being monitored.

As you can see each website is its own object.  This I nice for putting them into maintenance mode putting them into groups.

I go to groups I can see that the Bulk URL editor also created a group.

This is the most common way to monitor URLs.  If you only have a few websites to monitor then this solution works fine. But most organizations have a ton of internal and external websites. I would say if you are monitoring over 25 websites I would look at using the Bulk URL Editor or the custom solution by Russ Slaten

TechNet has some good documentation here. http://technet.microsoft.com/en-us/library/bb381370.aspx

Lets setup a basic website monitor for my blog using the Web Application Management Pack Template.

In the Authoring console I expand Management Pack Templates. I right click on Web Application and click Add Monitoring Wizard

I select Web Application and then click next

For the name I like to give it a descriptive name like “Website – scom2k7.com”.  I like to put the “Website – “ in front of all my websites as it makes it easier later on.

Also I like to have at least one custom management pack to store all of my URL monitors in. If you are monitoring a lot of applications it will make sense to break them up into a few different management packs.

I click Next,

Now I type in the url of the website I want to monitor, in this case I type in “www.scom2k7.com.com” I click test to make sure that SCOM can access the URL I want to monitor. (Often times the test will timeout don’t worry if it does, just continue on as the test functionality doesn’t always seem to work.)

Now I need to pick my watcher node. This is where I want my website to be monitored from. If you are going to be monitoring a large number of websites, it is recommended to run these from a dedicated management server. The reason for this is that management servers write directly to the Ops DB. If you run all of your website monitoring from an agent, the agent has to monitor the website and then send the status to a management servers. Running the website monitoring directly from a management server avoids the middle man.

I also change the “Run this query” option to ten minutes. I think two minutes is too aggressive and causes excessive alerts. Most other URL monitoring tools I have seen use ten minutes as their default.

I click next and then click Create

I open the monitoring console and wait for it to show up under Web Application, Web Applications State

If I right click on my website and go to health explorer I can see that I am monitoring a lot more then just the website status.

Also if I go to groups I can see that the wizard automatically created a group called “Website – scom2k7.com watcher computers group”

There are three different ways to monitor a website with operations manager in SCOM.  Each of the different ways has its own pros and cons.

1. Web Application Management Pack Template
2. Bulk URL Editor or BUE
3. Custom URL management pack by Russ Slaten

Web Application Management Pack Template

Pros:

• Ease of Use
• Integrated into the Console
• Supported by MS Support

Cons:

• Produces a lot of alerts
• Not very scalable
• Alerts are not very descriptive to what the actual issue is
• Creates too many groups
• Time consuming to setup a lot of websites

Bulk URL Editor

Pros:

• Scales to 3,000 websites per management servers and 12,000 per management group
• Saves time by importing a list of of multiple websites from a text file
• Each URL creates its own instance so putting individual URLs in maintenance mode is easier
• Less groups are created
• Supported by MS Support

Cons:

• Not Intuitive
• Not integrated and requires an extra console to be copied for tool to be run
• Produces a lot of alerts
• Alerts are not very descriptive to what the actual issue is

Custom URL management pack by Russ Slaten

Pros:

• Easy to add and remove websites
• Reduces number of alerts by consolidating them and only alerting if website is really down.
• Alerts are descriptive telling you exactly why the alert was created
• Doesn’t create any extra groups
• Doesn’t require extra console as everything is added and removed from an csv (Excel) document
• Seems to scale well
• Reduced administration as csv document can be modified by anyone with no knowledge of SCOM

Cons:

• Not integrated in console
• Initial setup of MP is more difficult
• Logging of performance data is not possible but could be added
• No MS Support if you have issues

In all actuality there is probably a ton of custom ways you can monitor a website with SCOM.  For simplicity I am just going to cover the three most common.

I recently ran into this error. “The report server has encountered a configuration error. Logon failed for the unattended execution account.” 

A couple weeks ago I changed the passwords on all my SCOM Service Accounts.  I didn’t run any reports since then, but when I finally ran one I got the above error.

So to fix it I logged into my SQL Server and looked at Reporting Services Configuration.  I initially thought it was the service account.

I changed the password and tried to run the reports but got the same error.

The looking a little further I realized the it was obviously the Execution Account as that is what was specified in the error.  

I then ran reports without any issues.

Here is the full text of the error

Date: 7/13/2011 10:10:35 AM
Application: System Center Operations Manager 2007 R2
Application Version: 6.1.7221.61
Severity: Error
Message: Cannot initialize report.

Microsoft.Reporting.WinForms.ReportServerException: The report server has encountered a configuration error. Logon failed for the unattended execution account. (rsServerConfigurationError) —> Microsoft.Reporting.WinForms.ReportServerException: Log on failed. Ensure the user name and password are correct. (rsLogonFailed) —> Microsoft.Reporting.WinForms.ReportServerException: For more information about this error navigate to the report server on the local server machine, or enable remote errors
   — End of inner exception stack trace —
   — End of inner exception stack trace —
   at Microsoft.Reporting.WinForms.ServerReport.GetExecutionInfo()
   at Microsoft.Reporting.WinForms.ServerReport.GetParameters()
   at Microsoft.EnterpriseManagement.Mom.Internal.UI.Reporting.Parameters.ReportParameterBlock.Initialize(ServerReport serverReport)
   at Microsoft.EnterpriseManagement.Mom.Internal.UI.Console.ReportForm.SetReportJob(Object sender, ConsoleJobEventArgs args)
Microsoft.Reporting.WinForms.ReportServerException: Log on failed. Ensure the user name and password are correct. (rsLogonFailed) —> Microsoft.Reporting.WinForms.ReportServerException: For more information about this error navigate to the report server on the local server machine, or enable remote errors
   — End of inner exception stack trace —
Microsoft.Reporting.WinForms.ReportServerException: For more information about this error navigate to the report server on the local server machine, or enable remote errors

Download: Link

The System Center Operation Manager 2007 Administration Resource Kit provides the following features to aid in management group administration:

• Scheduled Maintenance Mode – Ability to schedule and manage maintenance mode in the management group.
• Clean Mom – Helps remove all installed R2 components.
• MP Event Analyzer – MP Event Analyzer tool is designed to help a user with functional and exploratory testing and debugging of event based management pack workflows like rules and monitors.

Feature Summary:

• Ability to schedule Maintenance mode for “run once”, daily, weekly, and monthly schedules
• Ability to removal all OpsMgr 2007 R2 files from a system
• Ability to debug event based management pack workflows

Download: SubscriptionCopier.zip

Guide: SubscriptionCopierGuide.pdf

Subscriptions copier is a simple tool for copying subscriptions and helping with alert escalation. I created this tool after multiple customers have asked me why they couldn’t copy an existing subscription.

I have found that customers are not using subscriptions to their full potential; most customers only setup one subscription. While this works for some, I think that all alerts should be actionable. If an alert has not had some type of action taken in a given time, the alert e-mail should be re-sent or escalated to someone who can take action on it. A ticketing system such as SCSM would also fulfill this function, but few customers have their SCOM system linked to a ticketing system.

You want to make a copy of the Windows Server Team subscription and send alerts to the Windows Server Team’s manager, if they not closed within one hour. To do this today is a very manual process. You would have to open up the subscription and take a screenshot of the configuration. Then create a new subscription with the same configuration and add the Windows Team Manager to the recipients. While this works, there is a possibility that you don’t select that exact same criteria and accidently send the windows team manager an informational alert in the middle of the night that the windows team never even saw.

So let’s show how this is done with the subscription copier tool:

As you can see we have 3 subscriptions

Step 1: Launch the Subscription Copier Tool

Step 2: Type in the name of your root management server

Step 3: The tool will then open

Step 4: Select the Subscription you would like to copy from the drop down.

Step 5: Select how may copies you would like to make. (In this scenario we will only create one.)

Step 6: Select how many minutes in between re-alerts. In this scenario we will leave it at the default of 0 (This will be used in scenario 2)

Step 7: Check this box if you want the copied subscription enabled. (In this scenario we are leaving it unchecked because we need to make some changes before it is enabled.)

Step 8: Click Create

Step 9: You should see a message telling you that the subscription was create successfully.

Step 10: Open the SCOM console and refresh the subscriptions screen. You should see the copied alert under the subscriptions screen.

Now back to our scenario we want to alert the windows team manager if an alert has stayed open for longer than one hour.

We go to the properties of the copied alert.

Change the name to “Windows Server Team (Escalated to manager after one hour)” and click next

As you can see all of our criteria has been copied from the original subscription.

Click Next

Remove the Windows Team E-Mail and add Windows Server Team Manager to the subscribers

On the channels screen set the alert aging to 60 minutes. This will only send alerts that have been open for longer than 60 minutes to the Windows Server Team Manager

On the Summary Page click “Enable this notification subscription.”

Then click Finish

You want a subscription to re-alert on open alerts every 10 minutes for one hour. To do this today you would have to open up the subscription and take a screenshot of the configuration. Then create six new subscriptions with the same configuration and set the delay by adding an extra ten minutes to every subscription. This could take up to 10 minutes manually. If you were to do this for every subscription in your environment it could take many hours.

So let’s see how this is done with the subscription copier tool

Step 1: Launch the Subscription Copier Tool

Step 2: Type in the name of your Root Management server

Step 4: Select the Subscription you would like to copy from the drop down.

Step 5: Select how may copies you would like to make. In this scenario we will create six.

Step 6: Select how many minutes between each re-alert. In this scenario we will set it to 10 minutes.

Step 7: Check the enabled box to enable the 6 subscriptions we are going to create.

Step 8: Click Create

Step 9: It will take a short time as the copies are created. After it completes you should see a message telling you that 6 subscription copies were created at a 10 minute interval.

Step 10: Open the SCOM Console and refresh the subscriptions screen.

You should now see the copied subscriptions created by the tool.

• When creating multiple copies with a re-alert interval. Set your subscription criteria to only include only

Reason: When your subscription criteria includes closed alerts, the closed alerts get delayed and you will get some a bunch of extra closed alerts that you won’t want.

• If you have subscriptions with duplicate names you will get an error of “An entry with the same key already exists”.

Workaround: Delete subscription(s) with duplicate name

• Copied subscriptions with a delay set. The delay doesn’t interval doesn’t get copied.

Workaround: Manually set the delay on copied subscriptions

SCOM 2007 R2 CU4 has been released to the web.  You can download it here

CU 4 contains a fix that I have been waiting for since SCOM 2007 RTM.  It has the ability for the RMS to recover from a SQL disconnect, restart, or failure.  This is obviously super important for a monitoring system.  

Automatic recovery of Health Service in a SQL Server failure scenario
After SQL Server goes offline, Health Service of Root Management Server (RMS) stops responding. For example, Health Service stops responding after SQL Server disconnects, restarts, or fails. To recover from this issue when SQL Server is available again, you must restart Health Service. 

By default, this automatic recovery feature is disabled. The DALInitiateClearPoolSeconds setting controls when RMS drops the current connection pool and when RMS tries to reestablish an SQL connection. We recommend that you set this setting to 60 seconds or more to avoid performance issues. To configure this recommended setting, set the following values:

• SOFTWARE\Microsoft\Microsoft Operations Manager\3.0\DAL\DALInitiateClearPool = true
• SOFTWARE\Microsoft\Microsoft Operations Manager\3.0\DAL\DALInitiateClearPoolSeconds = 60

*Note* To apply the DALInitiateClearPoolSeconds setting, restart Health Service of RMS.

Kevin Holman has done a nice job providing step-by-step instructions on CU4.  Link

When I wrote SCOM VBScripts in the past, it was easy to test the script and see if I was getting the right output. 

I would simply run the cscript command from the command prompt to get the output.

C:\temp2\cscript count.vbs "c:\temp2" 4

But now with a similar SCOM PowerShell script I get useless output.

PS C:\temp2\ .\filecount.ps1 -Folder ‘C:\temp2’ -FCount 5

To fix this I have to add a simple line of code to the end of my PowerShell script

$api.Return($bag)

Now I can see the same output that my VBScripts gave me in the past.

Don’t forget to comment the line out after you are done testing.

I decided I needed to re-install my lab environment.  I wanted to keep all of the firewalls on during the install process and only open the ports that are actually needed. I installed SQL using a named instance as many customers use a SQL 2008 cluster.

After I installed the SCOM database on the SQL 2008 server with all firewalls on.  I created a firewall rule to let port 1433 allow connections.  As specified in the Supported Configurations doc

Root management server 1433 —> OperationsManager database

I also setup a firewall rule to allow port 1434 back to the RMS server from the SQL Instance Server. (Also in the guide)

Root management server 1434 UDP < — OperationsManager database

I start the install of SCOM to the RMS server.  I unchecked Database as my database is already install the on the SQL instance.

I typed in my SC Database Instance Name and clicked Next

But I got this error “Setup cannot location the SC database”

So I enabled firewall logging to see what was getting dropped blocked by setting the firewall to log dropped packets.

In the SCOM setup I clicked back and then next.

I checked the firewall logs in %systemroot%\system32\Logfiles\Firewall\pfirewall.log  and it looks like UDP port 1434 is being dropped

I create another rule on the SQL server to enable UDP port 1434

In the SCOM setup I click back and next again.

Once again same failure.  “Setup cannot location the SC database”

Back to the firewall logs.  It now needs TCP port 62756 (Not in the guide)

I create another rule on the SQL server to enable TCP port 62756

After that rule is enabled I am able to continue on a install SCOM successfully with all of the windows firewalls still on.