Drastically reduce alert noise in SCOM

I noticed a way to reduce our alert noise by up to about 80%. This may seem very obvious but it wasn’t to me till after I realized it.

In my environment we are using a ton of different applications by various software vendors. Some of these applications automatically restart their windows services during the day and night. Other times we will get a heartbeat notification in the middle of the night.

With the majority of these notifications we will get an open and then a closed alert almost immediately. In my environment SCOM is setup to send an SMS alert to the on-call admin. This meant that the on-call admin would often be getting woken up in the middle of the night for an issue that automatically resolved itself.

I wanted to reduce the number of alerts that the on-call person (which is sometimes me). I came up with the idea to set alert aging on initial alert by 5 minutes.

We were already using alert aging for escalation but by aging the initial alert drastically reduced the number of alerts that the on-call person gets. The majority of the Open and Closed alerts are no longer bothering the on-call person in middle of the night

Being that SCOM is state based anything that is really down or causing an issue will still page the on-call person.