VM Scale set is in a Failed ProvisioningState, because of an extension

[Petkoii0]

We've discovered there is a VM Scale set, that seems to be in "failed" ProvisioningState. The nodes are running fine, but one extension on the VMSS has been reported as failed. Unfortunately this is a managed service and we don't have much of an access to troubleshoot this further. However we checked the state of the VMSS via the API with the command below:

az vmss get-instance-view --resource-group groupName --name vmssName

      "name": "AKSLinuxExtension",
      "statusesSummary": [
        {
          "code": "ProvisioningState/succeeded",
          "count": 1
        },
        {
          "code": "ProvisioningState/failed",
          "count": 1
        }
      ]
    }
  ],
  "orchestrationServices": null,
  "statuses": [
    {
      "code": "ProvisioningState/failed/VMExtensionHandlerNonTransientError",
      "displayStatus": "Provisioning failed",
      "level": "Error",
      "message": "The handler for VM extension type 'Microsoft.AKS.Compute.AKS.Linux.AKSNode' has reported terminal failure for VM extension 'AKSLinuxExtension' with error message: 'Failed to download artifacts: [ExtensionDownloadError] Timeout downloading extension package. Elapsed: 0:08:04.405546 URIs tried: 2/3. Last error: [HttpError] Download failed both on the primary and fallback channels. IOError [Errno -3] Temporary failure in name resolution -- 6 attempts made]'.\r\n    \r\n'Manifest download failed for the extension. 

When we ran on all VMSS nodes the same command it showed that everything looks fine:

     "name": "AKSLinuxExtension",
     "statuses": [
       {
         "code": "ProvisioningState/succeeded",
         "displayStatus": "Provisioning succeeded",
         "level": "Info",
         "message": "Successfully enabled Compute.AKS.Linux.AKSNode extension",
         "time": null
       }
     ],
     "substatuses": null,
     "type": "Microsoft.AKS.Compute.AKS.Linux.AKSNode",
     "typeHandlerVersion": "1.56"
   }

The question here is - how do we fix the error of the VMSS? Will restarting it fix the issue without a downtime, or there is another more pragmatic way to clear this error?

[vishalgupta-microsoft]

+1 Facing exactly same issue where the PUT operation on VMSS shows success even though the individual VMs are having extension provisioning as Failed. The VMSS rolling upgrade is also moving forward from instance to instance when it shouldn't in this case.

VM instance state

  {
      "name": "DSCConfiguration",
      "type": "Microsoft.Powershell.DSC",
      "typeHandlerVersion": "2.83.5",
      "substatuses": [
        {
          "code": "ComponentStatus/DscConfigurationLog/failed/8",
          "level": "Error",

VMSS extension state

{
  "name": "DSCConfiguration",
  "id": "/subscriptions/<subId>/resourceGroups/<rg>/providers/Microsoft.Compute/virtualMachineScaleSets/<vmssname>/extensions/DSCConfiguration",
  "type": "Microsoft.Compute/virtualMachineScaleSets/extensions",
  "properties": {
    "provisioningState": "Succeeded",
    "autoUpgradeMinorVersion": true,
    "provisionAfterExtensions": [
      "Microsoft.Azure.Geneva.GenevaMonitoring"
    ],
    "forceUpdateTag": "UpdateARMDeployment-2024-10-14T11-49-54",
    "publisher": "Microsoft.Powershell",
    "type": "DSC",
    "typeHandlerVersion": "2.73",
    "settings": {