Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

VM cross cluster migration not working #10078

Open
ahmedali6 opened this issue Dec 10, 2024 · 2 comments
Open

VM cross cluster migration not working #10078

ahmedali6 opened this issue Dec 10, 2024 · 2 comments
Labels
component:migration
Milestone
4.19.2

Comments

@ahmedali6
Copy link
Contributor

ahmedali6 commented Dec 10, 2024 •
edited
Loading

ISSUE TYPE
  • Bug Report
COMPONENT NAME
API
CLOUDSTACK VERSION
4.19.1.2
CONFIGURATION

Advanced Networking

OS / ENVIRONMENT

OS: Red Hat 9
Fresh installation of CLoudStack

SUMMARY

When trying to migrate a vm between clusters the migration fails

STEPS TO REPRODUCE
Create a Zone
Create a Cluster A
Create a Cluster B
Create a cluster-wide primary storage for cluster A
Create a cluster-wide primary storage for cluster B
Create a VM in cluster A
Migrate the VM with storage to cluster B
EXPECTED RESULTS
The instance to be migrated to cluster B
ACTUAL RESULTS
The migration fails and a volume create and stuck in `Migrating` state

image

image

Management server logs

2024-12-10 09:27:04,052 DEBUG [c.c.a.ApiServlet] (qtp1390913202-391:[ctx-434f8d2a, ctx-fb760dc7]) (logid:1efe446c) ===END===  10.0.0.43 -- GET  command=readyForShutdown&response=json&sessionkey=oxYEmliCuKzvEtObrqHQyNyP678
2024-12-10 09:27:04,201 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 2-2287: Seq 2-2287:  { Cmd , MgmtId: -1, via: 2, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"2","_loadInfo":"{
  "connections": [],
  "removedSessions": []
}","wait":"0","bypassHostMaintenance":"false"}}] }
2024-12-10 09:27:04,212 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-4:[]) (logid:) SeqA 2-: Sending 2287
2024-12-10 09:27:04,234 DEBUG [c.c.a.t.Request] (AgentManager-Handler-6:[]) (logid:) Seq 4-7171982406587515702: Processing:  { Ans: , MgmtId: 52231313017, via: 4, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":"false","details":"com.cloud.utils.exception.CloudRuntimeException: Can't find volume:88ea6edb-5cf2-413b-abc3-9be3b6177813
	at com.cloud.hypervisor.kvm.storage.LibvirtStoragePool.getPhysicalDisk(LibvirtStoragePool.java:159)
	at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.getPhysicalDisk(KVMStoragePoolManager.java:369)
	at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyTargetsCommandWrapper.execute(LibvirtModifyTargetsCommandWrapper.java:57)
	at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyTargetsCommandWrapper.execute(LibvirtModifyTargetsCommandWrapper.java:38)
	at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
	at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1945)
	at com.cloud.agent.Agent.processRequest(Agent.java:686)
	at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1109)
	at com.cloud.utils.nio.Task.call(Task.java:83)
	at com.cloud.utils.nio.Task.call(Task.java:29)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)
","wait":"0","bypassHostMaintenance":"false"}}] }
2024-12-10 09:27:04,235 DEBUG [c.c.a.t.Request] (Work-Job-Executor-5:[ctx-d8c8e5df, job-22/job-23, ctx-107ec426]) (logid:6c4e0747) Seq 4-7171982406587515702: Received:  { Ans: , MgmtId: 52231313017, via: 4(localhost.localdomain), Ver: v1, Flags: 10, { Answer } }
2024-12-10 09:27:04,235 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (Work-Job-Executor-5:[ctx-d8c8e5df, job-22/job-23, ctx-107ec426]) (logid:6c4e0747) Details from executing class com.cloud.agent.api.ModifyTargetsCommand: com.cloud.utils.exception.CloudRuntimeException: Can't find volume:88ea6edb-5cf2-413b-abc3-9be3b6177813
	at com.cloud.hypervisor.kvm.storage.LibvirtStoragePool.getPhysicalDisk(LibvirtStoragePool.java:159)
	at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.getPhysicalDisk(KVMStoragePoolManager.java:369)
	at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyTargetsCommandWrapper.execute(LibvirtModifyTargetsCommandWrapper.java:57)
	at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyTargetsCommandWrapper.execute(LibvirtModifyTargetsCommandWrapper.java:38)
	at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
	at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1945)
	at com.cloud.agent.Agent.processRequest(Agent.java:686)
	at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1109)
	at com.cloud.utils.nio.Task.call(Task.java:83)
	at com.cloud.utils.nio.Task.call(Task.java:29)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)

2024-12-10 09:27:04,267 DEBUG [o.a.c.s.v.VolumeServiceImpl] (Work-Job-Executor-5:[ctx-d8c8e5df, job-22/job-23, ctx-107ec426]) (logid:6c4e0747) Failed to copy volume java.lang.ClassCastException: class com.cloud.agent.api.Answer cannot be cast to class com.cloud.agent.api.ModifyTargetsAnswer (com.cloud.agent.api.Answer and com.cloud.agent.api.ModifyTargetsAnswer are in unnamed module of loader 'app')
	at org.apache.cloudstack.storage.motion.StorageSystemDataMotionStrategy.sendModifyTargetsCommand(StorageSystemDataMotionStrategy.java:2401)
	at org.apache.cloudstack.storage.motion.StorageSystemDataMotionStrategy.connectHostToVolume(StorageSystemDataMotionStrategy.java:2366)
	at org.apache.cloudstack.storage.motion.StorageSystemDataMotionStrategy.copyAsync(StorageSystemDataMotionStrategy.java:2075)
	at org.apache.cloudstack.storage.motion.DataMotionServiceImpl.copyAsync(DataMotionServiceImpl.java:136)
	at org.apache.cloudstack.storage.volume.VolumeServiceImpl.migrateVolumes(VolumeServiceImpl.java:2334)
	at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolumes(VolumeOrchestrator.java:1438)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:569)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
	at jdk.proxy3/jdk.proxy3.$Proxy266.migrateVolumes(Unknown Source)
	at com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:3295)
	at com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:5515)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:569)
	at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:106)
	at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5591)
	at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:99)
	at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:652)
	at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
	at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
	at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:600)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)

2024-12-10 09:27:04,269 ERROR [o.a.c.e.o.VolumeOrchestrator] (Work-Job-Executor-5:[ctx-d8c8e5df, job-22/job-23, ctx-107ec426]) (logid:6c4e0747) Failed to migrate VM [VM instance {"id":3,"instanceName":"i-2-3-VM","type":"User","uuid":"57df225e-a02f-4249-b731-fd48767a442f"}] along with its volumes due to [java.lang.ClassCastException: class com.cloud.agent.api.Answer cannot be cast to class com.cloud.agent.api.ModifyTargetsAnswer (com.cloud.agent.api.Answer and com.cloud.agent.api.ModifyTargetsAnswer are in unnamed module of loader 'app')].
2024-12-10 09:27:04,270 INFO  [c.c.v.ClusteredVirtualMachineManagerImpl] (Work-Job-Executor-5:[ctx-d8c8e5df, job-22/job-23, ctx-107ec426]) (logid:6c4e0747) Migration was unsuccessful.  Cleaning up: VM instance {"id":3,"instanceName":"i-2-3-VM","type":"User","uuid":"57df225e-a02f-4249-b731-fd48767a442f"}
2024-12-10 09:27:04,305 DEBUG [o.a.c.e.o.VolumeOrchestrator] (Work-Job-Executor-5:[ctx-d8c8e5df, job-22/job-23, ctx-107ec426]) (logid:6c4e0747) Releasing [1] volumes for VM [VM instance {"id":3,"instanceName":"i-2-3-VM","type":"User","uuid":"57df225e-a02f-4249-b731-fd48767a442f"}] from host [Host {"id":4,"name":"localhost.localdomain","type":"Routing","uuid":"f20e35b7-a4b1-445a-bb3f-fd267760eb57"}].
2024-12-10 09:27:04,323 WARN  [c.c.a.AlertManagerImpl] (Work-Job-Executor-5:[ctx-d8c8e5df, job-22/job-23, ctx-107ec426]) (logid:6c4e0747) alertType=[17] dataCenterId=[1] podId=[1] clusterId=[null] message=[Unable to migrate vm i-2-3-VM from host localhost.localdomain in zone zone-1 and pod zone-1].
2024-12-10 09:27:04,327 DEBUG [c.c.a.AlertManagerImpl] (Work-Job-Executor-5:[ctx-d8c8e5df, job-22/job-23, ctx-107ec426]) (logid:6c4e0747) Have already sent: 1 emails for alert type '17' -- skipping send email
2024-12-10 09:27:04,331 DEBUG [c.c.v.ClusteredVirtualMachineManagerImpl] (Work-Job-Executor-5:[ctx-d8c8e5df, job-22/job-23, ctx-107ec426]) (logid:6c4e0747) VM instance {"id":3,"instanceName":"i-2-3-VM","type":"User","uuid":"57df225e-a02f-4249-b731-fd48767a442f"} is a [User], returning null for control Nic IP.
2024-12-10 09:27:04,351 DEBUG [c.c.a.m.ClusteredAgentAttache] (Work-Job-Executor-5:[ctx-d8c8e5df, job-22/job-23, ctx-107ec426]) (logid:6c4e0747) Seq 4-7171982406587515719: Routed from 52231313017
2024-12-10 09:27:04,352 DEBUG [c.c.a.t.Request] (Work-Job-Executor-5:[ctx-d8c8e5df, job-22/job-23, ctx-107ec426]) (logid:6c4e0747) Seq 4-7171982406587515719: Sending  { Cmd , MgmtId: 52231313017, via: 4(localhost.localdomain), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":"false","checkBeforeCleanup":"false","forceStop":"false","vlanToPersistenceMap":{"650":"true"},"volumesToDisconnect":[],"vmName":"i-2-3-VM","executeInSequence":"false","wait":"0","bypassHostMaintenance":"false"}}] }
2024-12-10 09:27:04,416 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-5:[ctx-d8c8e5df, job-22/job-23, ctx-107ec426]) (logid:6c4e0747) VM instance {"id":3,"instanceName":"i-2-3-VM","type":"User","uuid":"57df225e-a02f-4249-b731-fd48767a442f"} state transited from [Migrating] to [Running] with event [OperationFailed]. VM's original host: Host {"id":1,"name":"localhost.localdomain","type":"Routing","uuid":"b1df0bbe-ab48-4d5e-8a49-af9adc95ab3a"}, new host: Host {"id":1,"name":"localhost.localdomain","type":"Routing","uuid":"b1df0bbe-ab48-4d5e-8a49-af9adc95ab3a"}, host before state transition: Host {"id":4,"name":"localhost.localdomain","type":"Routing","uuid":"f20e35b7-a4b1-445a-bb3f-fd267760eb57"}
2024-12-10 09:27:04,436 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-5:[ctx-d8c8e5df, job-22/job-23, ctx-107ec426]) (logid:6c4e0747) Hosts's actual total CPU: 9200 and CPU after applying overprovisioning: 9200
2024-12-10 09:27:04,437 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-5:[ctx-d8c8e5df, job-22/job-23, ctx-107ec426]) (logid:6c4e0747) Hosts's actual total RAM: (6.98 GB) 7492751360 and RAM after applying overprovisioning: (6.98 GB) 7492751360
2024-12-10 09:27:04,445 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-5:[ctx-d8c8e5df, job-22/job-23, ctx-107ec426]) (logid:6c4e0747) release cpu from host: 4, old used: 1000,reserved: 0, actual total: 9200, total with overprovisioning: 9200; new used: 500,reserved:0; movedfromreserved: false,moveToReserveredfalse
2024-12-10 09:27:04,447 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-5:[ctx-d8c8e5df, job-22/job-23, ctx-107ec426]) (logid:6c4e0747) release mem from host: 4, old used: (1.00 GB) 1073741824,reserved: (0 bytes) 0, total: (6.98 GB) 7492751360; new used: (512.00 MB) 536870912,reserved:(0 bytes) 0; movedfromreserved: false,moveToReserveredfalse
2024-12-10 09:27:04,476 ERROR [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-5:[ctx-d8c8e5df, job-22/job-23, ctx-107ec426]) (logid:6c4e0747) Invocation exception, caused by: com.cloud.utils.exception.CloudRuntimeException: Failed to migrate VM [VM instance {"id":3,"instanceName":"i-2-3-VM","type":"User","uuid":"57df225e-a02f-4249-b731-fd48767a442f"}] along with its volumes due to [java.lang.ClassCastException: class com.cloud.agent.api.Answer cannot be cast to class com.cloud.agent.api.ModifyTargetsAnswer (com.cloud.agent.api.Answer and com.cloud.agent.api.ModifyTargetsAnswer are in unnamed module of loader 'app')].
2024-12-10 09:27:04,476 INFO  [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-5:[ctx-d8c8e5df, job-22/job-23, ctx-107ec426]) (logid:6c4e0747) Rethrow exception com.cloud.utils.exception.CloudRuntimeException: Failed to migrate VM [VM instance {"id":3,"instanceName":"i-2-3-VM","type":"User","uuid":"57df225e-a02f-4249-b731-fd48767a442f"}] along with its volumes due to [java.lang.ClassCastException: class com.cloud.agent.api.Answer cannot be cast to class com.cloud.agent.api.ModifyTargetsAnswer (com.cloud.agent.api.Answer and com.cloud.agent.api.ModifyTargetsAnswer are in unnamed module of loader 'app')].
2024-12-10 09:27:04,477 DEBUG [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-5:[ctx-d8c8e5df, job-22/job-23]) (logid:6c4e0747) Done with run of VM work job: com.cloud.vm.VmWorkMigrateWithStorage for VM 3, job origin: 22
2024-12-10 09:27:04,478 ERROR [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-5:[ctx-d8c8e5df, job-22/job-23]) (logid:6c4e0747) Unable to complete AsyncJobVO: {id:23, userId: 2, accountId: 2, instanceType: null, instanceId: null, cmd: com.cloud.vm.VmWorkMigrateWithStorage, cmdInfo: rO0ABXNyACVjb20uY2xvdWQudm0uVm1Xb3JrTWlncmF0ZVdpdGhTdG9yYWdlsew9z6UxtXMCAANKAApkZXN0SG9zdElkSgAJc3JjSG9zdElkTAAMdm9sdW1lVG9Qb29sdAAPTGphdmEvdXRpbC9NYXA7eHIAE2NvbS5jbG91ZC52bS5WbVdvcmufmbZW8CVnawIABEoACWFjY291bnRJZEoABnVzZXJJZEoABHZtSWRMAAtoYW5kbGVyTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO3hwAAAAAAAAAAIAAAAAAAAAAgAAAAAAAAADdAAZVmlydHVhbE1hY2hpbmVNYW5hZ2VySW1wbAAAAAAAAAAEAAAAAAAAAAFzcgARamF2YS51dGlsLkhhc2hNYXAFB9rBwxZg0QMAAkYACmxvYWRGYWN0b3JJAAl0aHJlc2hvbGR4cD9AAAAAAAAMdwgAAAAQAAAAAXNyAA5qYXZhLmxhbmcuTG9uZzuL5JDMjyPfAgABSgAFdmFsdWV4cgAQamF2YS5sYW5nLk51bWJlcoaslR0LlOCLAgAAeHAAAAAAAAAAA3NxAH4ACAAAAAAAAAACeA, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 52231313017, completeMsid: null, lastUpdated: null, lastPolled: null, created: Tue Dec 10 09:21:59 EST 2024, removed: null}, job origin: 22 com.cloud.utils.exception.CloudRuntimeException: Failed to migrate VM [VM instance {"id":3,"instanceName":"i-2-3-VM","type":"User","uuid":"57df225e-a02f-4249-b731-fd48767a442f"}] along with its volumes due to [java.lang.ClassCastException: class com.cloud.agent.api.Answer cannot be cast to class com.cloud.agent.api.ModifyTargetsAnswer (com.cloud.agent.api.Answer and com.cloud.agent.api.ModifyTargetsAnswer are in unnamed module of loader 'app')].
	at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolumes(VolumeOrchestrator.java:1444)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:569)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
	at jdk.proxy3/jdk.proxy3.$Proxy266.migrateVolumes(Unknown Source)
	at com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:3295)
	at com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:5515)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:569)
	at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:106)
	at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5591)
	at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:99)
	at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:652)
	at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
	at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
	at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:600)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)

2024-12-10 09:27:04,512 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-5:[ctx-d8c8e5df, job-22/job-23]) (logid:6c4e0747) Complete async job-23, jobStatus: FAILED, resultCode: 0, result: rO0ABXNyAC9jb20uY2xvdWQudXRpbHMuZXhjZXB0aW9uLkNsb3VkUnVudGltZUV4Y2VwdGlvblZNT3AAAAACAwABSQALY3NFcnJvckNvZGV4cgAaamF2YS5sYW5nLlJ1bnRpbWVFeGNlcHRpb26eXwZHCjSD5QIAAHhyABNqYXZhLmxhbmcuRXhjZXB0aW9u0P0fPho7HMQCAAB4cgATamF2YS5sYW5nLlRocm93YWJsZdXGNSc5d7jLAwAETAAFY2F1c2V0ABVMamF2YS9sYW5nL1Rocm93YWJsZTtMAA1kZXRhaWxNZXNzYWdldAASTGphdmEvbGFuZy9TdHJpbmc7WwAKc3RhY2tUcmFjZXQAHltMamF2YS9sYW5nL1N0YWNrVHJhY2VFbGVtZW50O0wAFHN1cHByZXNzZWRFeGNlcHRpb25zdAAQTGphdmEvdXRpbC9MaXN0O3hwcQB-AAh0AZBGYWlsZWQgdG8gbWlncmF0ZSBWTSBbVk0gaW5zdGFuY2UgeyJpZCI6MywiaW5zdGFuY2VOYW1lIjoiaS0yLTMtVk0iLCJ0eXBlIjoiVXNlciIsInV1aWQiOiI1N2RmMjI1ZS1hMDJmLTQyNDktYjczMS1mZDQ4NzY3YTQ0MmYifV0gYWxvbmcgd2l0aCBpdHMgdm9sdW1lcyBkdWUgdG8gW2phdmEubGFuZy5DbGFzc0Nhc3RFeGNlcHRpb246IGNsYXNzIGNvbS5jbG91ZC5hZ2VudC5hcGkuQW5zd2VyIGNhbm5vdCBiZSBjYXN0IHRvIGNsYXNzIGNvbS5jbG91ZC5hZ2VudC5hcGkuTW9kaWZ5VGFyZ2V0c0Fuc3dlciAoY29tLmNsb3VkLmFnZW50LmFwaS5BbnN3ZXIgYW5kIGNvbS5jbG91ZC5hZ2VudC5hcGkuTW9kaWZ5VGFyZ2V0c0Fuc3dlciBhcmUgaW4gdW5uYW1lZCBtb2R1bGUgb2YgbG9hZGVyICdhcHAnKV0udXIAHltMamF2YS5sYW5nLlN0YWNrVHJhY2VFbGVtZW50OwJGKjw8_SI5AgAAeHAAAAAhc3IAG2phdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudGEJxZomNt2FAgAIQgAGZm9ybWF0SQAKbGluZU51bWJlckwAD2NsYXNzTG9hZGVyTmFtZXEAfgAFTAAOZGVjbGFyaW5nQ2xhc3NxAH4ABUwACGZpbGVOYW1lcQB-AAVMAAptZXRob2ROYW1lcQB-AAVMAAptb2R1bGVOYW1lcQB-AAVMAA1tb2R1bGVWZXJzaW9ucQB-AAV4cAEAAAWkdAADYXBwdAA9b3JnLmFwYWNoZS5jbG91ZHN0YWNrLmVuZ2luZS5vcmNoZXN0cmF0aW9uLlZvbHVtZU9yY2hlc3RyYXRvcnQAF1ZvbHVtZU9yY2hlc3RyYXRvci5qYXZhdAAObWlncmF0ZVZvbHVtZXNwcHNxAH4ADAL____-cHQALWpkay5pbnRlcm5hbC5yZWZsZWN0Lk5hdGl2ZU1ldGhvZEFjY2Vzc29ySW1wbHQAHU5hdGl2ZU1ldGhvZEFjY2Vzc29ySW1wbC5qYXZhdAAHaW52b2tlMHQACWphdmEuYmFzZXQABzE3LjAuMTNzcQB-AAwCAAAATXBxAH4AE3EAfgAUdAAGaW52b2tlcQB-ABZxAH4AF3NxAH4ADAIAAAArcHQAMWpkay5pbnRlcm5hbC5yZWZsZWN0LkRlbGVnYXRpbmdNZXRob2RBY2Nlc3NvckltcGx0ACFEZWxlZ2F0aW5nTWV0aG9kQWNjZXNzb3JJbXBsLmphdmFxAH4AGXEAfgAWcQB-ABdzcQB-AAwCAAACOXB0ABhqYXZhLmxhbmcucmVmbGVjdC5NZXRob2R0AAtNZXRob2QuamF2YXEAfgAZcQB-ABZxAH4AF3NxAH4ADAEAAAFYcQB-AA50AChvcmcuc3ByaW5nZnJhbWV3b3JrLmFvcC5zdXBwb3J0LkFvcFV0aWxzdAANQW9wVXRpbHMuamF2YXQAHmludm9rZUpvaW5wb2ludFVzaW5nUmVmbGVjdGlvbnBwc3EAfgAMAQAAAMZxAH4ADnQAPG9yZy5zcHJpbmdmcmFtZXdvcmsuYW9wLmZyYW1ld29yay5SZWZsZWN0aXZlTWV0aG9kSW52b2NhdGlvbnQAH1JlZmxlY3RpdmVNZXRob2RJbnZvY2F0aW9uLmphdmF0AA9pbnZva2VKb2lucG9pbnRwcHNxAH4ADAEAAACjcQB-AA5xAH4AJXEAfgAmdAAHcHJvY2VlZHBwc3EAfgAMAQAAAGFxAH4ADnQAP29yZy5zcHJpbmdmcmFtZXdvcmsuYW9wLmludGVyY2VwdG9yLkV4cG9zZUludm9jYXRpb25JbnRlcmNlcHRvcnQAIEV4cG9zZUludm9jYXRpb25JbnRlcmNlcHRvci5qYXZhcQB-ABlwcHNxAH4ADAEAAAC6cQB-AA5xAH4AJXEAfgAmcQB-AClwcHNxAH4ADAEAAADXcQB-AA50ADRvcmcuc3ByaW5nZnJhbWV3b3JrLmFvcC5mcmFtZXdvcmsuSmRrRHluYW1pY0FvcFByb3h5dAAXSmRrRHluYW1pY0FvcFByb3h5LmphdmFxAH4AGXBwc3EAfgAMAP____9wdAAUamRrLnByb3h5My4kUHJveHkyNjZwcQB-ABF0AApqZGsucHJveHkzcHNxAH4ADAEAAAzfcQB-AA50ACZjb20uY2xvdWQudm0uVmlydHVhbE1hY2hpbmVNYW5hZ2VySW1wbHQAHlZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwuamF2YXQAHW9yY2hlc3RyYXRlTWlncmF0ZVdpdGhTdG9yYWdlcHBzcQB-AAwBAAAVi3EAfgAOcQB-ADVxAH4ANnEAfgA3cHBzcQB-AAwC_____nBxAH4AE3EAfgAUcQB-ABVxAH4AFnEAfgAXc3EAfgAMAgAAAE1wcQB-ABNxAH4AFHEAfgAZcQB-ABZxAH4AF3NxAH4ADAIAAAArcHEAfgAbcQB-ABxxAH4AGXEAfgAWcQB-ABdzcQB-AAwCAAACOXBxAH4AHnEAfgAfcQB-ABlxAH4AFnEAfgAXc3EAfgAMAQAAAGpxAH4ADnQAImNvbS5jbG91ZC52bS5WbVdvcmtKb2JIYW5kbGVyUHJveHl0ABpWbVdvcmtKb2JIYW5kbGVyUHJveHkuamF2YXQAD2hhbmRsZVZtV29ya0pvYnBwc3EAfgAMAQAAFddxAH4ADnEAfgA1cQB-ADZxAH4AQHBwc3EAfgAMAQAAAGNxAH4ADnQAIGNvbS5jbG91ZC52bS5WbVdvcmtKb2JEaXNwYXRjaGVydAAYVm1Xb3JrSm9iRGlzcGF0Y2hlci5qYXZhdAAGcnVuSm9icHBzcQB-AAwBAAACjHEAfgAOdAA_b3JnLmFwYWNoZS5jbG91ZHN0YWNrLmZyYW1ld29yay5qb2JzLmltcGwuQXN5bmNKb2JNYW5hZ2VySW1wbCQ1dAAYQXN5bmNKb2JNYW5hZ2VySW1wbC5qYXZhdAAMcnVuSW5Db250ZXh0cHBzcQB-AAwBAAAAMXEAfgAOdAA-b3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5NYW5hZ2VkQ29udGV4dFJ1bm5hYmxlJDF0ABtNYW5hZ2VkQ29udGV4dFJ1bm5hYmxlLmphdmF0AANydW5wcHNxAH4ADAEAAAA4cQB-AA50AEJvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0LmltcGwuRGVmYXVsdE1hbmFnZWRDb250ZXh0JDF0ABpEZWZhdWx0TWFuYWdlZENvbnRleHQuamF2YXQABGNhbGxwcHNxAH4ADAEAAABncQB-AA50AEBvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0LmltcGwuRGVmYXVsdE1hbmFnZWRDb250ZXh0cQB-AFB0AA9jYWxsV2l0aENvbnRleHRwcHNxAH4ADAEAAAA1cQB-AA5xAH4AU3EAfgBQdAAOcnVuV2l0aENvbnRleHRwcHNxAH4ADAEAAAAucQB-AA50ADxvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGVxAH4ATHEAfgBNcHBzcQB-AAwBAAACWHEAfgAOcQB-AEdxAH4ASHEAfgBNcHBzcQB-AAwCAAACG3B0AC5qYXZhLnV0aWwuY29uY3VycmVudC5FeGVjdXRvcnMkUnVubmFibGVBZGFwdGVydAAORXhlY3V0b3JzLmphdmFxAH4AUXEAfgAWcQB-ABdzcQB-AAwCAAABCHB0AB9qYXZhLnV0aWwuY29uY3VycmVudC5GdXR1cmVUYXNrdAAPRnV0dXJlVGFzay5qYXZhcQB-AE1xAH4AFnEAfgAXc3EAfgAMAgAABHBwdAAnamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9ydAAXVGhyZWFkUG9vbEV4ZWN1dG9yLmphdmF0AAlydW5Xb3JrZXJxAH4AFnEAfgAXc3EAfgAMAgAAAntwdAAuamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9yJFdvcmtlcnEAfgBicQB-AE1xAH4AFnEAfgAXc3EAfgAMAgAAA0hwdAAQamF2YS5sYW5nLlRocmVhZHQAC1RocmVhZC5qYXZhcQB-AE1xAH4AFnEAfgAXc3IAH2phdmEudXRpbC5Db2xsZWN0aW9ucyRFbXB0eUxpc3R6uBe0PKee3gIAAHhweAAAEJp3CAAAAAAAAAAAeA
2024-12-10 09:27:04,514 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-5:[ctx-d8c8e5df, job-22/job-23]) (logid:6c4e0747) Publish async job-23 complete on message bus
2024-12-10 09:27:04,514 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-5:[ctx-d8c8e5df, job-22/job-23]) (logid:6c4e0747) Wake up jobs related to job-23
2024-12-10 09:27:04,514 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-5:[ctx-d8c8e5df, job-22/job-23]) (logid:6c4e0747) Update db status for job-23
2024-12-10 09:27:04,535 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-5:[ctx-d8c8e5df, job-22/job-23]) (logid:6c4e0747) Wake up jobs joined with job-23 and disjoin all subjobs created from job- 23
2024-12-10 09:27:04,630 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl$5] (Work-Job-Executor-5:[ctx-d8c8e5df, job-22/job-23]) (logid:6c4e0747) Done executing com.cloud.vm.VmWorkMigrateWithStorage for job-23
2024-12-10 09:27:04,644 INFO  [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-5:[ctx-d8c8e5df, job-22/job-23]) (logid:6c4e0747) Remove job-23 from job monitoring
2024-12-10 09:27:04,676 WARN  [o.a.c.m.w.WebhookServiceImpl] (API-Job-Executor-15:[ctx-f549996f, job-22, ctx-7c67f785]) (logid:6c4e0747) Skipping delivering event [ID: null, description: {"event":"VM.MIGRATE","status":"Completed"}] to any webhook as account ID is missing
2024-12-10 09:27:04,676 WARN  [o.a.c.f.e.EventDistributorImpl] (API-Job-Executor-15:[ctx-f549996f, job-22, ctx-7c67f785]) (logid:6c4e0747) Failed to publish event [category: ActionEvent, type: VM.MIGRATE] on bus webhookEventBus
2024-12-10 09:27:04,713 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-15:[ctx-f549996f, job-22]) (logid:6c4e0747) Unexpected exception while executing org.apache.cloudstack.api.command.admin.vm.MigrateVirtualMachineWithVolumeCmd com.cloud.utils.exception.CloudRuntimeException: Failed to migrate VM [VM instance {"id":3,"instanceName":"i-2-3-VM","type":"User","uuid":"57df225e-a02f-4249-b731-fd48767a442f"}] along with its volumes due to [java.lang.ClassCastException: class com.cloud.agent.api.Answer cannot be cast to class com.cloud.agent.api.ModifyTargetsAnswer (com.cloud.agent.api.Answer and com.cloud.agent.api.ModifyTargetsAnswer are in unnamed module of loader 'app')].
	at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolumes(VolumeOrchestrator.java:1444)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:569)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
	at jdk.proxy3/jdk.proxy3.$Proxy266.migrateVolumes(Unknown Source)
	at com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:3295)
	at com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:5515)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:569)
	at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:106)
	at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5591)
	at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:99)
	at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:652)
	at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
	at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
	at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:600)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)

2024-12-10 09:27:04,725 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-15:[ctx-f549996f, job-22]) (logid:6c4e0747) Complete async job-22, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed to migrate VM [VM instance {"id":3,"instanceName":"i-2-3-VM","type":"User","uuid":"57df225e-a02f-4249-b731-fd48767a442f"}] along with its volumes due to [java.lang.ClassCastException: class com.cloud.agent.api.Answer cannot be cast to class com.cloud.agent.api.ModifyTargetsAnswer (com.cloud.agent.api.Answer and com.cloud.agent.api.ModifyTargetsAnswer are in unnamed module of loader 'app')]."}
2024-12-10 09:27:04,728 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-15:[ctx-f549996f, job-22]) (logid:6c4e0747) Publish async job-22 complete on message bus
2024-12-10 09:27:04,728 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-15:[ctx-f549996f, job-22]) (logid:6c4e0747) Wake up jobs related to job-22
2024-12-10 09:27:04,728 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-15:[ctx-f549996f, job-22]) (logid:6c4e0747) Update db status for job-22
2024-12-10 09:27:04,731 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-15:[ctx-f549996f, job-22]) (logid:6c4e0747) Wake up jobs joined with job-22 and disjoin all subjobs created from job- 22
2024-12-10 09:27:04,789 DEBUG [c.c.a.ApiServer] (API-Job-Executor-15:[ctx-f549996f, job-22]) (logid:6c4e0747) Retrieved cmdEventType from job info: VM.MIGRATE
2024-12-10 09:27:04,850 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl$5] (API-Job-Executor-15:[ctx-f549996f, job-22]) (logid:6c4e0747) Done executing org.apache.cloudstack.api.command.admin.vm.MigrateVirtualMachineWithVolumeCmd for job-22
2024-12-10 09:27:04,850 INFO  [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-15:[ctx-f549996f, job-22]) (logid:6c4e0747) Remove job-22 from job monitoring
2024-12-10 09:27:05,248 DEBUG [c.c.a.t.Request] (AgentManager-Handler-5:[]) (logid:) Seq 4-7171982406587515719: Processing:  { Ans: , MgmtId: 52231313017, via: 4, Ver: v1, Flags: 10, [{"com.cloud.agent.api.StopAnswer":{"result":"true","wait":"0","bypassHostMaintenance":"false"}}] }
2024-12-10 09:27:05,249 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-5:[]) (logid:) Seq 4-7171982406587515719: Unable to find listener.

Agent logs

2024-12-10 05:50:28,601 INFO  [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:[]) (logid:fed9001a) Trying to fetch storage pool ef8eef61-8e8c-3b13-9f04-2d854a6e0fc6 from libvirt
2024-12-10 05:50:28,612 WARN  [cloud.agent.Agent] (agentRequest-Handler-1:[]) (logid:fed9001a) Caught: com.cloud.utils.exception.CloudRuntimeException: Can't find volume:efedc0b5-8674-4bd8-a723-7c971c5bce40
	at com.cloud.hypervisor.kvm.storage.LibvirtStoragePool.getPhysicalDisk(LibvirtStoragePool.java:159)
	at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.getPhysicalDisk(KVMStoragePoolManager.java:369)
	at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyTargetsCommandWrapper.execute(LibvirtModifyTargetsCommandWrapper.java:57)
	at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyTargetsCommandWrapper.execute(LibvirtModifyTargetsCommandWrapper.java:38)
	at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
	at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1945)
	at com.cloud.agent.Agent.processRequest(Agent.java:686)
	at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1109)
	at com.cloud.utils.nio.Task.call(Task.java:83)
	at com.cloud.utils.nio.Task.call(Task.java:29)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)

2024-12-10 05:50:28,758 INFO  [kvm.resource.LibvirtConnection] (agentRequest-Handler-2:[]) (logid:fed9001a) No existing libvirtd connection found. Opening a new one
2024-12-10 05:50:28,764 WARN  [kvm.resource.LibvirtConnection] (agentRequest-Handler-2:[]) (logid:fed9001a) Can not find a connection for Instance i-2-3-VM. Assuming the default connection.
2024-12-10 05:50:29,473 WARN  [kvm.resource.LibvirtKvmAgentHook] (agentRequest-Handler-2:[]) (logid:fed9001a) Groovy script '/etc/cloudstack/agent/hooks/libvirt-vm-state-change.groovy' is not available. Transformations will not be applied.
2024-12-10 05:50:29,473 WARN  [kvm.resource.LibvirtKvmAgentHook] (agentRequest-Handler-2:[]) (logid:fed9001a) Groovy scripting engine is not initialized. Data transformation skipped.
2024-12-10 05:50:55,629 INFO  [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:[]) (logid:a8dc1538) Trying to fetch storage pool ef8eef61-8e8c-3b13-9f04-2d854a6e0fc6 from libvirt
2024-12-10 05:50:55,633 INFO  [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:[]) (logid:a8dc1538) Asking libvirt to refresh storage pool ef8eef61-8e8c-3b13-9f04-2d854a6e0fc6
@DaanHoogland DaanHoogland added this to the 4.19.2 milestone Dec 10, 2024
@weizhouapache
Copy link
Member

the error is

Can't find volume:88ea6edb-5cf2-413b-abc3-9be3b6177813
	at com.cloud.hypervisor.kvm.storage.LibvirtStoragePool.getPhysicalDisk(LibvirtStoragePool.java:159)
	at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.getPhysicalDisk(KVMStoragePoolManager.java:369)
	at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyTargetsCommandWrapper.execute(LibvirtModifyTargetsCommandWrapper.java:57)
	at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyTargetsCommandWrapper.execute(LibvirtModifyTargetsCommandWrapper.java:38)
	at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
	at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1945)

what storage do you use ? @ahmedali6

@ahmedali6
Copy link
Contributor Author

@weizhouapache NFS Storage

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
component:migration
Projects
Apache CloudStack BugFest - Issues
Status: Todo
Milestone
4.19.2
Development

No branches or pull requests
3 participants
@DaanHoogland @weizhouapache @ahmedali6