Worker Pool referenced from a release only, can be deleted and the release cannot be deployed.

[benPearce1]

Severity

Blocking some customers, no workaround available

Version

Confirmed as affecting 2024.3 and 2024.4.

Latest Version

I could reproduce the problem in the latest build

What happened?

A worker pool can be deleted if it is referenced in the deployment process for a release, but not in a current deployment process

Reproduction

Create a worker pool
Reference this worker pool from a deployment step
Create a release
Modify the worker pool in the step to be a different worker pool
Delete the worker pool
Try to deploy the release
Deployment will fail with error

Error and Stacktrace

Exception of type 'Octopus.Core.Model.Exceptions.EntityNotFoundException' was thrown.
Octopus.Core.Model.Exceptions.EntityNotFoundException
   at Octopus.Core.Persistence.Database.DatabaseDocumentStore`2.Get(TKey id) in DatabaseDocumentStore.cs:line 63
   at Octopus.Core.Persistence.Git.SlugDocumentStoreDecorator`2.Get(TKey id) in SlugDocumentStoreDecorator.cs:line 57
   at Octopus.Core.Persistence.Database.FullTableCacheDocumentStoreDecorator`2.Get(TKey id) in FullTableCacheDocumentStoreDecorator.cs:line 67
   at Octopus.Core.Persistence.Auditing.AuditingDocumentStoreDecorator`2.Get(TKey id) in AuditingDocumentStoreDecorator.cs:line 84
   at Octopus.Core.Persistence.Database.Deletion.DeleteRelatedDocumentsDocumentStoreDecorator`2.Get(TKey id) in DeleteRelatedDocumentsDocumentStoreDecorator.cs:line 48
   at Octopus.Core.Persistence.Database.Deletion.VetoDocumentStoreDecorator`2.Get(TKey id) in VetoDocumentStoreDecorator.cs:line 47
   at Octopus.Core.Persistence.Database.AccessCheckingDocumentStoreDecorator`2.Get(TKey id) in AccessCheckingDocumentStoreDecorator.cs:line 43
   at Octopus.Core.Persistence.DocumentStore`2.Get(TKey id) in DocumentStore.cs:line 91
   at Octopus.Server.Orchestration.ServerTasks.Deploy.ExecutionPlanner.GetWorkerPoolIdFromAction(ExecutionAction action, IImmutableVariableDictionary actionVariables) in ExecutionPlanner.cs:line 81
   at Octopus.Server.ExecutionsV2.Model.BaseExecutionPlanner.AddNonSkippedActionsToPlan(ExecutionPlan plan, ExecutionStep step, PlannedStep plannedStep, ITaskLog taskLog) in BaseExecutionPlanner.cs:line 531
   at Octopus.Server.ExecutionsV2.Model.BaseExecutionPlanner.PlanSteps(ExecutionPlan plan, IList`1 steps, ITaskLog taskLog)+MoveNext() in BaseExecutionPlanner.cs:line 431
   at System.Collections.Generic.LargeArrayBuilder`1.AddRange(IEnumerable`1 items)
   at System.Collections.Generic.EnumerableHelpers.ToArray[T](IEnumerable`1 source)
   at Octopus.Server.ExecutionsV2.Model.BaseExecutionPlanner.Initialize(ITaskLog taskLog, CancellationToken cancellationToken) in BaseExecutionPlanner.cs:line 202
   at Octopus.Server.Orchestration.ServerTasks.Deploy.ExecutionPlanner.Initialize(ITaskLog taskLog, CancellationToken cancellationToken) in ExecutionPlanner.cs:line 65
   at Octopus.Server.Orchestration.ServerTasks.Deploy.DeploymentPlanService.CreateOrResumePlan(DeploymentId deploymentId, ITaskLog taskLog, CancellationToken cancellationToken) in DeploymentPlanService.cs:line 116
   at Octopus.Server.Orchestration.ServerTasks.Deploy.DeploymentTaskControllerService.<>c__DisplayClass2_0.<CreateOrResumePlan in DeploymentTaskControllerService.cs:line 26
   at Octopus.Core.Infrastructure.UnitsOfWork.UnitOfWorkExecutor.<>c__DisplayClass13_0`3.<Execute in UnitOfWorkExecutor.cs:line 284
   at Octopus.Core.Infrastructure.UnitsOfWork.UnitOfWorkExtensionMethods.Do[T](IUnitOfWork unitOfWork, Func`1 action, CancellationToken cancellationToken, String name) in UnitOfWorkExtensionMethods.cs:line 15
   at Octopus.Core.Infrastructure.UnitsOfWork.UnitOfWorkExtensionMethods.Do[T](IUnitOfWork unitOfWork, Func`1 action, CancellationToken cancellationToken, String name) in UnitOfWorkExtensionMethods.cs:line 21
   at Octopus.Core.Infrastructure.UnitsOfWork.UnitOfWorkExecutor.Execute[T1,T2,TResult](Func`4 action, CancellationToken cancellationToken, String name) in UnitOfWorkExecutor.cs:line 279
   at Octopus.Core.Infrastructure.UnitsOfWork.UnitOfWorkExecutor.Execute[T1,T2,TResult](Func`4 action, CancellationToken cancellationToken, String name) in UnitOfWorkExecutor.cs:line 279
   at Octopus.Server.Orchestration.ServerTasks.Deploy.DeploymentTaskControllerService.CreateOrResumePlan(DeploymentId deploymentId, ITaskLog taskLog, CancellationToken cancellationToken) in DeploymentTaskControllerService.cs:line 23
   at Octopus.Server.Orchestration.ServerTasks.Deploy.DeploymentTaskController.Execute(CancellationToken cancellationToken) in DeploymentTaskController.cs:line 32
   at Octopus.Server.Orchestration.ServerTasks.RunningTask.WorkerTask(CancellationToken cancellationToken) in RunningTask.cs:line 199
   at Octopus.Server.Orchestration.ServerTasks.RunningTask.WorkerTask(CancellationToken cancellationToken) in RunningTask.cs:line 206

More Information

No response

Workaround

Recreating the release with the same package version numbers might be an option, but if the deployment process has changed this might not be a valid approach.

The broken deployment process snapshot can be updated directly in the database, this is not advisable unless you know what you are doing.

SQL command to fix up the old worker pool reference in old releases:

DECLARE @deletedWorkerPoolId nvarchar(50) = 'WorkerPools-1463'
DECLARE @replacementWorkerPoolId nvarchar(50) = 'WorkerPools-1462'
DECLARE @spaceId nvarchar(50) = "Spaces-1"
UPDATE [DeploymentProcess] 
SET JSON = replace(JSON, '"WorkerPoolId":"' + @deletedWorkerPoolId + '"', '"WorkerPoolId":"' + @replacementWorkerPoolId +'"')
WHERE IsFrozen = 1 and SpaceId = @spaceId