Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Linstor VM/Volume snapshots #9360

Closed
UAnton opened this issue Jul 9, 2024 · 5 comments · Fixed by #10219
Closed

Linstor VM/Volume snapshots #9360

UAnton opened this issue Jul 9, 2024 · 5 comments · Fixed by #10219
Assignees
@rp- @UAnton
Labels
3rd-party:linstor has-3rdparty-components
Milestone
4.19.2

Comments

@UAnton
Copy link

UAnton commented Jul 9, 2024

ISSUE TYPE
  • Other
COMPONENT NAME
Storage
CLOUDSTACK VERSION
4.19.0.2
CONFIGURATION
OS / ENVIRONMENT
SUMMARY

How can I create snapshots/backups?

STEPS TO REPRODUCE
Try to create a VM/Volume snapshot
EXPECTED RESULTS
Snapshot created
ACTUAL RESULTS
get error - Error while taking snapshot
@UAnton
Copy link
Author

UAnton commented Jul 9, 2024 

2024-07-09 10:18:15,953 ERROR [o.a.c.a.c.u.s.CreateSnapshotCmd] (API-Job-Executor-4:ctx-753b7118 job-52 ctx-7d1d55c9) (logid:feb78585) Failed to create snapshot due to an internal error creating snapshot for volume 12bf6956-591f-4f53-a1a1-36cb932752e0
java.lang.RuntimeException: Unexpected exception
        at com.cloud.storage.VolumeApiServiceImpl.takeSnapshotInternal(VolumeApiServiceImpl.java:3528)
        at com.cloud.storage.VolumeApiServiceImpl.takeSnapshot(VolumeApiServiceImpl.java:3445)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
        at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:52)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
        at com.sun.proxy.$Proxy236.takeSnapshot(Unknown Source)
        at org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:228)
        at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:172)
        at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:112)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:654)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:602)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: com.cloud.utils.exception.CloudRuntimeException: Unable to unhide zfs snapshot device.
        at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:382)
        at org.apache.cloudstack.storage.snapshot.DefaultSnapshotStrategy.backupSnapshot(DefaultSnapshotStrategy.java:193)
        at com.cloud.storage.snapshot.SnapshotManagerImpl.backupSnapshotToSecondary(SnapshotManagerImpl.java:1439)
        ... 45 more
2024-07-09 10:18:15,955 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-4:ctx-753b7118 job-52) (logid:feb78585) Complete async job-52, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed to create snapshot due to an internal error creating snapshot for volume 12bf6956-591f-4f53-a1a1-36cb932752e0"}

@DaanHoogland
Copy link
Contributor

DaanHoogland commented Jul 12, 2024 •
edited
Loading

I see I made some assumption that might not be true. Is this a linstor environment @UAnton ? what storage type(s) did you try this with?

(I was looking at the title, so maybe not so wrong after all)

can you have a look as well @rp- ?

@DaanHoogland DaanHoogland added this to the 4.19.2.0 milestone Jul 12, 2024
@UAnton
Copy link
Author

UAnton commented Jul 12, 2024

@DaanHoogland I use Linstor. When I try to take a snapshot, I get an error in the UI and don’t see the snapshot, but the snapshot is created in the Linstor storage

@rp-
Copy link
Contributor

rp- commented Jul 15, 2024

Well the exception kinda tells what the problem is:
Caused by: com.cloud.utils.exception.CloudRuntimeException: Unable to unhide zfs snapshot device.

We copy the data to secondary storage, from the zfs snapshot. But zfs snapshots are not directly accessible by default
and you have to unhide the block device. For some reason the unhide fails on your setup.
Can you try to manually unhide such a snapshot to see the error? e.g.:
zfs set snapdev=visible <zfs-snapshotname>

Or we need the agent debug output of the failed request.

@rp- rp- mentioned this issue Jan 20, 2025
Merged
14 tasks
@rp- rp- linked a pull request Jan 20, 2025 that will close this issue
linstor: Fix ZFS snapshot backup #10219
Merged
14 tasks
@harikrishna-patnala
Copy link
Contributor

#10219 related PR is merged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@rp- rp-

@UAnton UAnton

Labels
3rd-party:linstor has-3rdparty-components
Projects
Apache CloudStack BugFest - Issues
Status: Done
Milestone
4.19.2
Development

Successfully merging a pull request may close this issue.

linstor: Fix ZFS snapshot backup LINBIT/cloudstack
4 participants
@rp- @UAnton @DaanHoogland @harikrishna-patnala