Archive | SCOM

Users closing monitor based alerts creates chaos in a SCOM environment.  The SCOM Monitor Reset app keeps your alerts in sync.  Now users can close any alert and know that if the problem still exists they will get re-alerted automatically.

The SCOM Monitor Reset App is a scalable windows service that runs and resets any monitor based alerts closed by an end user.  You just install it and forget it.   The service takes care of all the work.

Now as an SCOM administrator you no longer have to explain the difference between monitors and rules.  End users can close any and all alerts.  Closed alerts that are still a problem will re-open a new alert.

Usage:

A disk space alert comes in.

The user closes the alert without fixing the issue.

Normally the alert would stay closed and the monitor would remain in a critical state.

But with the monitor reset app.  After 5 minutes the alert is detected closed.  The monitor state is reset.

If disk space issue still exists a new alert will be created.

Any problems, bugs, or issues please e-mail:  support@scom2k7.com.com

One of my customers was experiencing lots of growth in their OperationsManager DB.

They monitor hundreds of SQL servers. I had a look into their Top Tables using Kevin Holman’s Large table query. http://bit.ly/1REx9Os

Things looked pretty normal where Performance tables are the top tables

I drilled down a little further in the performance data and see this.

Focusing on the two top counters that are 4x larger then the next few.

I took a look at the counters for one of my SQL servers and realize that we are collecting the same counter over and over for each database instance on a SQL server.

This is crazy.  It might make some sense if a customer ran every database on a individual drive, but that is not the norm.  Most SQL servers have one or a few drives for their DB files and Logs.

To double check I looked at the OperationsManager database performance tables.

Sure enough we are collecting the same data 13 times in my case.

So how do we fix this?  Disable the rule that collects this data using an override for SQL 2005, 2008, 2012, and 2014.

Rule for SQL 2012 is called

• MSSQL 2012: Collect DB Disk Write Latency (ms)
• MSSQL 2012: Collect DB Disk Read Latency (ms)

I then recommend collecting this data if you need it once per disk.

To enable it, create an override for Windows Server 2008 and 2012 Logical Disk

• Collection Rule for Average Disk Seconds Per Write Windows Server 2012
• Collection Rule for Average Disk Seconds Per Read Windows Server 2012

New Features in Version 7.2

• Intelligent Maintenance Mode Bug Fix – Fixed bug where total minutes was being incorrectly calculated for Intelligent Maintenance Mode

New Features in Version 7

• CSV Server Import – Now you can schedule a list of Windows or Unix computers for Maintenance Mode using a CSV (Comma Separated Values) file.

• Intelligent Maintenance Mode – Before a computer or object is put into maintenance mode it is checked to see if it’s already in maintenance mode. If the existing maintenance window is longer then the new maintenance window, the existing window is kept.

• Unix/Linux Support for Immediate Maintenance Mode – Unix Admins can now use the MMNow page to immediately put servers into Maintenance Mode using a web browser or wget from a shell script.

• Management Servers in a Group are prevented from going into Maintenance Mode – If a group is scheduled for maintenance mode that contains Management Servers. The Management Servers are now skipped from going into Maintenance Mode.

• New Jobs Report Fixed – Some users were experiencing an issue where the MMScheduler box was grayed out. This has now been resolved.

• Improved permissions tool – It is now easier to grant new permissions as existing users and groups show up when the tool is opened.

• Improved Logging – Events are created when a Schedule Maintenance Job is run or when a user puts a server into maintenance mode immediately using MM Now

CSV Server Import

CSV File

Unix/Linux Support for MMNow Page

Calling One Click MM using wget from a Unix/Linux Shell Script

wget http://xom01/MMweb/OneClick.aspx

Calling One Click MM using Invoke-WebRequest from PowerShell script on a Windows Computer

Invoke-WebRequest http://xOM01/MMweb/OneClick.aspx -UseBasicParsing

New Jobs Report Fixed

Improved permissions tool

Improved Logging

Customer was asking why they were not getting alerts for SQL Agent Job failures.  They had enabled individual SQL Agent Job Discoveries that is talked about in Kevin’s post.

By default the Last Run Status monitor does not alert.  So they created an override to enable Generates Alert.

Problem was no alerts came through.  Looking at Alert severity it says “Critical” and Alert on State it says “The monitor is in a critical health state”.  Looks good to me.

But still no alerts.

Lets take a look at the monitor.  The monitor is in a warning state.  That is weird I thought Alert severity is “Critical” from the above screenshot.

Lets take a look at the monitor.  I right click and go to monitor properties.  There is no Critical Health State, only Warning and Healthy for this monitor.  Looks like a bug to me or bad MP practices.

If I look back at my Overrides I can see that this monitor will only Alert On State if “Alert if it is in a Critical Health State”

Well it will never go into a “Critical Health state” because it doesn’t exist.   There is no way to override the monitors health.  This is not usually an issue because most monitors are either two state Health and Critical, or three state, Healthy, Warning, and Critical.

So what to do?

The only way to get the monitor to alert is to create an override so that Alert on State is changed to Warning.

Now when the monitor changes to a “Warning state” I get an Alert.

But why is it a “Warning”?  Not really sure.  The default behavior is Alert severity is “Critical”

If I create another override for “Alert severity” Critical

Now I get a critical alert.

But the monitor is still in a warning state and could be confusing to the end user.

I will file a bug to get this changed.

Today it was officially announced that Microsoft has acquired BlueStripe software, which has been a key partner for Operations Manager and Azure for monitoring application performance and availability of distributed business applications seamlessly and comprehensively.

You can read more of the announcement here

http://blogs.microsoft.com/blog/2015/06/10/microsoft-acquires-bluestripe-to-help-customers-improve-application-visibility-and-management-across-the-datacenter-and-cloud/

Ran into this issue today trying to upgrade from SCOM RTM to SCOM SP1

I got the standard 1603 error in the OpsMgrSetupWizard.log

[15:44:20]: Error: :LaunchMSI: MSI D:\Setup\AMD64\Server\OMServer.msi returned error 1603
[15:44:20]: Error: :ProcessInstalls: Install Item Management Server failed to install. We did not launch the post process delegate.
[15:44:20]: Always: :SetErrorType: Setting VitalFailure. currentInstallItem: Management Server

Opening up the OMServer.log one of my colleagues Stefan Wuchenauer identified the issue.

MSI (s) (18:18) [15:43:33:134]: Executing op: CustomActionSchedule(Action=_Rollback_UninstallBIDMOF.DB577B04_77CF_4F29_BA24_844EA640D17E,ActionType=3393,Source=BinaryData,Target=UninstallMOF,CustomActionData=C:\Program Files\System Center 2012\Operations Manager\Server\mom_tracing.mof)
MSI (s) (18:18) [15:43:33:136]: Executing op: ActionStart(Name=_InstallBIDMOF.DB577B04_77CF_4F29_BA24_844EA640D17E,,)
MSI (s) (18:18) [15:43:33:138]: Executing op: CustomActionSchedule(Action=_InstallBIDMOF.DB577B04_77CF_4F29_BA24_844EA640D17E,ActionType=3073,Source=BinaryData,Target=InstallMOF,CustomActionData=C:\Program Files\System Center 2012\Operations Manager\Server\mom_tracing.mof)
MSI (s) (18:E8) [15:43:33:172]: Invoking remote custom action. DLL: C:\Windows\Installer\MSIDE96.tmp, Entrypoint: InstallMOF
InstallMOF: Custom Action Data. C:\Program Files\System Center 2012\Operations Manager\Server\mom_tracing.mof
CompileMOF: CompileFile() failed. Error Code: 0x80041002. C:\Program Files\System Center 2012\Operations Manager\Server\mom_tracing.mof
MSI (s) (18:78) [15:43:33:399]: I/O on thread 1084 could not be cancelled. Error: 1168
MSI (s) (18:78) [15:43:33:399]: I/O on thread 2704 could not be cancelled. Error: 1168
MSI (s) (18:78) [15:43:33:399]: I/O on thread 3316 could not be cancelled. Error: 1168
MSI (s) (18:78) [15:43:33:399]: I/O on thread 4716 could not be cancelled. Error: 1168
MSI (s) (18:78) [15:43:33:399]: I/O on thread 2840 could not be cancelled. Error: 1168
MSI (s) (18:78) [15:43:33:399]: I/O on thread 3720 could not be cancelled. Error: 1168
MSI (s) (18:78) [15:43:33:399]: I/O on thread 2924 could not be cancelled. Error: 1168
MSI (s) (18:78) [15:43:33:399]: I/O on thread 1324 could not be cancelled. Error: 1168
MSI (s) (18:78) [15:43:33:399]: I/O on thread 4736 could not be cancelled. Error: 1168
MSI (s) (18:78) [15:43:33:399]: I/O on thread 4496 could not be cancelled. Error: 1168
MSI (s) (18:78) [15:43:33:399]: I/O on thread 4048 could not be cancelled. Error: 1168
MSI (s) (18:78) [15:43:33:399]: I/O on thread 3100 could not be cancelled. Error: 1168
MSI (s) (18:78) [15:43:33:399]: I/O on thread 1768 could not be cancelled. Error: 1168
MSI (s) (18!4C) [15:43:33:400]: Product: System Center Operations Manager 2012 Server — Error 25218.Failed to uninstall SDK MOF. Error Code: -2147217406 (IDispatch error #3586).

Error 25218.Failed to uninstall SDK MOF. Error Code: -2147217406 (IDispatch error #3586).
CustomAction _InstallBIDMOF.DB577B04_77CF_4F29_BA24_844EA640D17E returned actual error code 1603 (note this may not be 100% accurate if translation happened inside sandbox)

Here is the fix. (*Note* The blog post says SCOM 2007 but it works for SCOM 2012 as well.)

http://blogs.technet.com/b/operationsmgr/archive/2009/05/05/opsmgr-2007-error-25218-failed-to-uninstall-sdk-mof-when-installing-sp1.aspx

Resolution: To resolve this issue follow the steps below:

1. Open a CMD prompt and navigate to the “C:\Windows\System32\Wbem” folder.

2. Run the following command:

for /f %s in (‘dir /b *.mof *.mfl’) do mofcomp %s
for /f %s in (‘dir /b /s *.dll’) do regsvr32 /s %s

3. Try installing the SP1 upgrade again.

I ran into this error today trying to install SCOM 2012.  “The target principal name is incorrect. Cannot generate SSPI context.”

The Error from the Setup Wizard was “This SQL server could not be found. Please check that this SQL server is running and that the current user has login permissions.”

Then looking at the OpsMgrSetupWizard.log located at %LocalAppData%\SCOM\LOGS\

I found this error

[17:50:31]: Always: :OpenAndTestSqlConnection: connection string – Data source=vDB01,1433;Integrated Security=true;Connection Timeout=5
[17:50:31]: Debug: :SqlConnectionReady returned False.
[17:50:31]: Info: :Info:DatabaseConfigurationPage: DB connection attempt completed.
[17:50:31]: Info: :Info:DatabaseConfigurationPage: DB connection attempt completed.
[17:50:32]: Always: :DatabaseConfigurationPage: Attempting to connect to database using server\instance vDB01. If we need it, the port is 1433
[17:50:32]: Debug: :MSSQLSERVER on server vDB01 is in a running state
[17:50:32]: Info: :Info:Opening/Testing Sql Connection on vDB01, port:
[17:50:32]: Debug: :Connection was not open. We will try to open it.
[17:50:32]: Error: :Error opening SQL connection.: Threw Exception.Type: System.Data.SqlClient.SqlException, Exception Error Code: 0x80131904, Exception.Message: The target principal name is incorrect. Cannot generate SSPI context.
[17:50:32]: Error: :StackTrace: at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)
at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal& connection)
at System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)
at System.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
at System.Data.SqlClient.SqlConnection.TryOpenInner(TaskCompletionSource`1 retry)
at System.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry)
at System.Data.SqlClient.SqlConnection.Open()
at Microsoft.EnterpriseManagement.OperationsManager.Setup.Common.SetupValidationHelpers.OpenAndTestSqlConnection(DbConnection sqlConnection, Int32 retries)

After some research I concluded that is must be an SPN issue.

I am using a service account for SQL.  So I attempted to manually set the SPN for the service account.

But I ran into this issue.  “The operation failed because the SPN value provided for addition/modification is not unique forest-wide.”

Turns out the computer account already had this SPN set.

So I removed the SPN for the computer account and added it to the SQL Service account.

Then the wizard let me continue.

Here are the new features in V6:

• Dedicated Linux / Unix Page –  Linux and Unix admins now have a dedicated page to schedule maintenance mode.  They can search and select multiple *NIX computers for maintenance mode.

• Improved Classes Page – The new classes page is much faster and uses the display name of objects.  It also has search and the ability to schedule multiple objects of a class.

• Classes Page now accepts a parameter  – When opening the classes page you can now specify an ObjectID as a parameter and it will find the object automatically for easy scheduling or integration in to other systems.

Dedicated Linux / Unix Page

Improved Classes Page

Classes Page now accepts a parameter

In SCOM 2012 R2 the parameters for a manual command line agent install have been updated.  One new required parameter in SCOM 2012 R2 is AcceptEndUserLicenseAgreement=1

To deploy 32-Bit and 64-Bit SCOM 2012 R2 Agents using SCCM in one package here is batch script updated for SCOM 2012 R2

@echo off

Set RegQry=HKLM\Hardware\Description\System\CentralProcessor\0

REG.exe Query %RegQry% > checkOS.txt

Find /i "x86" < CheckOS.txt > StringCheck.txt

If %ERRORLEVEL% == 0 (
               Echo "This is 32 Bit Operating System"
               msiexec.exe /i \\YOURFILESERVER\AgentManagement\x86\MOMAgent.msi /qn /l*v c:\MOMAgentinstall.log USE_SETTINGS_FROM_AD=0 USE_MANUALLY_SPECIFIED_SETTINGS=1 MANAGEMENT_GROUP=YOURMANAGEMENTGROUPNAME MANAGEMENT_SERVER_DNS=YOURSCOMMSSERVER.YOURDOMAIN.COM ACTIONS_USE_COMPUTER_ACCOUNT=1 AcceptEndUserLicenseAgreement=1
) ELSE (
               Echo "This is 64 Bit Operating System"
               msiexec.exe /i \\YOURFILESERVER\AgentManagement\amd64\MOMAgent.msi /qn /l*v c:\MOMAgentinstall.log USE_SETTINGS_FROM_AD=0 USE_MANUALLY_SPECIFIED_SETTINGS=1 MANAGEMENT_GROUP=YOURMANAGEMENTGROUPNAME MANAGEMENT_SERVER_DNS=YOURSCOMMSSERVER.YOURDOMAIN.COM  ACTIONS_USE_COMPUTER_ACCOUNT=1 AcceptEndUserLicenseAgreement=1
)

Download: https://www.scom2k7.com/downloads/SCOM2012Agent.bat.txt

Here is a list of all the parameters for a SCOM 2012 Agent install from https://technet.microsoft.com/en-us/library/hh230736.aspx