[Obsolete] FabricObserver 3.1.8, ClusterObserver 2.1.6, FabricObserverWebApi 2.0.1 - Update ASAP

FabricObserver 3.1.8, ClusterObserver 2.1.6, FabricObserverWebApi 2.0.1- sfpkgs with Microsoft-signed binaries. Nupkgs (FO, CO) are located in the nuget.org package gallery.

Changes

FabricObserver 3.1.8 - Update ASAP.

• Important Bug fix in Application Parameter Update logic.

• Bug fix in WindowsCpuUtilizationProvider.

• ApplicationManifest and Settings files have updated Parameters.

• New TelemetryData member: ProcessId.

• Updated health event clearing code when FO exits (gracefully) or parameters are updated.

• CSV Data logging update: Support for multiple file output (time stamped files created per observer run.) Only AppObs, FSO and NodeObs support CSV logging. The default write behavior has not changed to not break existing usage. See Settings.xml for comments. This feature (writing CSV files) can be turned on/off via application parameter updates.

• Log lifetime management: FO will prune old log files based on configured lifetime (see Settings.xml, ObserverManagerConfiguration section).

• Fixed bugs in port monitoring logic. It is now much more accurate on Windows.

• FO will only locally log Health Report data if EnableVerboseLogging is enabled or the FOWebApi app is deployed. It is best to only turn on verbose logging while debugging. You do not need to eat unnecessary disk in production. Just emit telemetry or read from the SF Health Event Store to get details about some warning that you want to handle. The HeatlthEvent.HealthInformation.Description property of any health event FO generates will always be a serialized (JSON) TelemetryData instance.

• Updated default monitor durations (decreased values).

• General code maintenance.

NOTE: There is a known issue with ETW where you will see EventSource errors in ETW data that look like this:

ERROR: Exception in Command Processing for EventSource FabricObserverETWProvider:
An instance of EventSource with Guid 0012c99c-680d-5235-03c3-7d3d6e193767 already exists.

This does not break ETW data transmission and will be fixed in 3.1.9. Again, this does not break ObserverLogger.LogEtw function nor is the exception bubbled up to FO process. This is an FYI.

ClusterObserver 2.1.6 - Requires FO 3.1.8 - If you use this service, update ASAP.

Important Bug fixes and added support for FO 3.1.8 (TelemetryData).

FabricObserverWebApi 2.0.1 - If you use this service, update ASAP.

• Bug fixes.
• Support for FO 3.1.8.

FabricObserver 3.1.7 [Obsolete]

FabricObserver 3.1.7 - sfpkgs with Microsoft-signed binaries. nupkgs are located in the nuget.org package gallery.

Changes

• Bug fix in AppObserver's ServiceIncludeList/ServiceExcludeList functionality.

[Obsolete] FabricObserver 3.1.6, ClusterObserver 2.1.5

FabricObserver 3.1.6, ClusterObserver 2.1.5 - sfpkgs with Microsoft-signed binaries. nupkgs are located in the nuget.org package gallery

Breaking Changes

• Each observer must now be individually configured to emit ETW. If you rely on ETW in your deployments, then you will need to enable ETW in both Settings.xml (ObserverManager configuration settings section, EnableETWProvider param) and on any observer you want to emit ETW in ApplicationManifest.xml (ETW section). Note that, as before, when you enable ETW (or Telemetry for that matter) on an observer, all measurements will be emitted, regardless of health state warnings/Ok clears, etc... The point of doing this is to collect both the raw resource usage data that FO creates and the health state information it produces when some specified target is doing bad things.

For plugin authors, or if you forked FO and modified observers, you must update your code to reflect these changes. You must set the EtwEnabled setting on any observer where you want to emit ETW. This is done in ApplicationManifest.xml, ETW section.

In your source code, please make the following change:

Logger.EtwLogger?.Write 

Must be replaced with

ObserverLogger.LogEtw

This is because FO's Logger class has been modified to have a private EventSource logger instance and the public API is now just a function call on ObserverBase's automatically constructed (during initialization for any observer instance) ObserverLogger instance.

If you are using Telemetry (LogAnalytics or ApplicationInsights) in your plugins or in forked observer impls, you will need to make the following change:

if (IsTelemetryProviderEnabled && IsObserverTelemetryEnabled)
{
     ...
}

Must be changed to

if (IsTelemetryEnabled)
{
    ...
}

• Removed ObserverBase's HealthReportProperties and HealthReportSourceIds List members. These were never a good idea and added unnecessary complexity with little real benefit. They are gone. This means your code will not compile if you use them. If you want to supply a custom property or source id for a health report you manually generate than just do so. They are both string properties on HealthInformation. If you utilize ProcessDataReportHealth function in ObserverBase, it will automatically take care of creating the meaningful source id and a readable, useful property value for any health event it generates.

Other Important Changes

• Fixed bug in ObserverBase.ProcessDataReportHealth function that prevented multiple warnings for a single app showing up correctly in SFX, Details view. The right events were created, however.

• Fixed important bugs in AppObserver that some may not have hit, but you would eventually. Please update to this version ASAP.

• New Feature: AppObserver will now monitor the service processes for all apps running on the node with a simple single configuration object in AppObserver.config.json file. See Documentation (and the current AppObserver.config.json file) for examples. This means you can supply one set of threshold settings that apply to all application services deployed to the node where an FO instance is running.

This new feature simplifies the notion of shared threshold settings and eliminates the need to duplicate the same threshold settings across the object array of application settings in the configuration JSON. Note that you can specify "overrides" for the "global" settings by simply also supplying a target specific configuration item with the same threshold type supplied (again, you can see an example of this in the documentation and new default AppObserver.config.json file). You can also supply a new appExcludeList/appIncludeList setting to filter out/in app targets. This support is not a breaking change and will have no impact on your currently-written AppObserver configurations

Example of an all-app target configuration item (note, you can supply "*" or "all" (case is not important) for the value. "*" is used below for extreme brevity (some like this), which means the configuration is for all apps currently deployed to the node):

[
  {
    "targetApp": "*",
    "appExcludeList": "fabric:/SomeApp42",
    "cpuWarningLimitPercent": 75,
    "networkWarningActivePorts": 1800,
    "networkWarningEphemeralPorts": 1400,
    "warningOpenFileHandles": 4000
  },
  {
    "targetApp": "fabric:/CpuStress",
    "cpuWarningLimitPercent": 25
  }
]

• ObserverBase.SetHealthReportTimeToLive function has been renamed to GetHealthReportTimeToLive

• TelemetryData member HealthEventDescription has been renamed to Description. Since ClusterObserver depends upon this type (and maintains an exact duplicate of its impl), ClusterObserver has been modified to support the renaming. Note: If you deploy ClusterObserver along with FO, you will need to deploy the latest ClusterObserver (2.1.5) in order for it to function correctly if you have deployed FO 3.1.6, which you should! Please upgrade both ASAP.

• All of these changes are reflected in the SampleObserverPlugin project in the FO repo as well as ContainerObserver.

• Using.md has been updated to reflect new feature in AppObserver.

FabricObserverWeb 2.0.0

FabricObserverWeb 2.0.0 sfpkgs and nupkgs with Microsoft-signed binaries.

FabricObserverWeb 2.0.0

FabricObserverWeb 2.0.0 adds support for the latest versions of FO and has been updated for the first time in several months including fixing some rather odd code.., which alone is deserving of 2.0.0 status. As you can see, FO build pipeline now produces FOW packages with signed binaries, as per your requests. Thanks for your feedback. Note that FOW is not a regularly-updated technology, but it is not forgotten. It is mainly useful for querying health data that observers produce within the confines of the cluster's internal network, all in JSONy ways. It is designed to work on both Windows and Linux. It was not designed with production environments in mind, but it is probably fine to use it in production nonetheless. It is a simple web API app (.NET Core 3.1).

FabricObserver 3.1.0 [Obsolete]

FabricObserver 3.1.0 sfpkgs and nupkgs with Microsoft-signed binaries.

FabricObserver 3.1.0 introduces a refactored public API implementation (housed in a .NET Standard 2.0 library) that will BREAK existing plugins and forked default observer impls. The changes required to fix the breakage are trivial, however. Please see the SampleObserver project for a complete sample observer plugin implementation with code comments and readme with examples of the new ObserverBase and observer constructor format. If you did not fork the codebase or write a plugin, then the changes will have no impact on you.

This was done to support .NET Standard 2.0 observer plug-ins. All shared observer APIs now live in FabricObserver.Extensibility.dll. Observer plugin impls no longer reference FabricObserver.dll.

ObserverBase, utilities, interfaces reside in the new library.

Namespaces have not been changed in the new library so the impact will be minimal. Essentially, this will only break plugins or forks of the codebase that have observer impls that do not have the now required constructor signature, which is shared by ObserverBase. Please read the updated documentation for more information.

In a nutshell:

Observers must have a public constructor that takes two parameters, a FabricClient instance and a StatelessServiceContext instance:

using System.Threading;
using System.Threading.Tasks;
using FabricObserver.Observers.Utilities;
using FabricObserver.Observers.Utilities.Telemetry;

namespace FabricObserver.Observers
{
    public class SampleNewObserver : ObserverBase
    {
        // FabricObserver will inject the FabricClient and StatelessServiceContext instances at runtime.        
        public SampleNewObserver(FabricClient fabricClient, StatelessServiceContext context)
          : base(fabricClient, context)
        {
            //... Your impl.
        }

        public override async Task ObserveAsync(CancellationToken token)
        {
            //... Your impl.
        }

        public override async Task ReportAsync(CancellationToken token)
        {
            //... Your impl.
        }
    }
 }

Observer plugin startup class has a new ConfigureServices design:

using Microsoft.Extensions.DependencyInjection;
using FabricObserver.Observers;
using System.Fabric;

[assembly: FabricObserver.FabricObserverStartup(typeof(SampleNewObserverStartup))]
namespace FabricObserver.Observers
{
    public class SampleNewObserverStartup : IFabricObserverStartup
    {
        // FabricObserver will inject the FabricClient and StatelessServiceContext instances at runtime.
        public void ConfigureServices(IServiceCollection services, FabricClient fabricClient, StatelessServiceContext context)
        {
            services.AddScoped(typeof(ObserverBase), s => new SampleNewObserver(fabricClient, context));
        }
    }
}

If you are using ObserverHealthReporter in your current plugin, you will need to modify how you construct it:

// FabricClientInstance is a built-in object in ObserverBase. 
var healthReporter = new ObserverHealthReporter(ObserverLogger, FabricClientInstance);