How to get rid of Observer ran too long Warnings.

[GitTorre]

Up until the latest version of FabricObserver (3.2.0), you may run into an issue where FO puts itself into warning state with a report like this:

AppObserver on node node000005 has exceeded its specified Maximum run time of 3600 seconds. This means something is wrong with AppObserver. It will not be run again. Please look into it.

Unfortunately, this was (mis)implemented with an infinite TTL. To get rid of this, simply connect to the cluster and run this PS command:

Send-ServiceFabricApplicationHealthReport -HealthState Ok -SourceId "ObserverManager" -HealthProperty "AppObserver_HealthState" -Description "clearing health report" -TimeToLiveSec 15 -Immediate -RemoveWhenExpired

For any observer that hits this, you simply adjust the script to include the observer name in the -HealthProperty arg of the script. So, the generalized command is:

Send-ServiceFabricApplicationHealthReport -HealthState Ok -SourceId "ObserverManager" -HealthProperty "[Observer name]_HealthState" -Description "clearing health report" -TimeToLiveSec 15 -Immediate -RemoveWhenExpired

This has been fixed (TTL = 15 minutes, not forever, but the observer will not run again until you restart the service - so, there is still a bug in the current version) in 3.2.0 and reimplemented in the upcoming version, 3.2.1 (5 minute TTL and a better approach to solving the problem: the observer will run again since its current run will be aborted).

Hope this helps,

Charles