Zfs: ZFS resume send with token errors out

[seonwoolee]

System information

Type | Version/Name
--- | ---
Distribution Name | Arch Linux
Distribution Version | N/A, rolling release
Linux Kernel | 4.19.1-arch1-1-ARCH
Architecture | x86_64
ZFS Version | 0.8.0-rc2
SPL Version | 0.8.0-rc2

Describe the problem you're observing

I did a zfs send recv with zfs send --raw --props -I master/replicate/TV@2018-1001-0000_monthly master/replicate/TV@2018-1114-2000_hourly | ssh <remote computer IP> zfs recv -s -o autosnapshot:original=false -u -F which was interrupted.

I then got the receive_resume_token at the remote

tank/local/NAS-Backup/TV  receive_resume_token  1-148e1e3f23-158-789c636064000310a501c49c50360710a715e5e7a69766a6304041b6dc91755922b39515806c762475f94959a9c9250c0c8a507518f26969c5a9250c7000926743924faa2c492d06d21792b1eb2fc987b8e2c8921e75b667af767b20c97382e5f3127353191872138b4b528bf48b520b723293134b52f543c21c8c0c0c2d740d0d0c4d750d80203e253133a712d91dbc0c08ffe72416a5a726e5e42767e76783c424a0f6c3e453739352538052207ddc48e2c9f9b90545a9c5c5105d1000006fc83351

So then I tried to resume this send via zfs send -v -t 1-148e1e3f23-158-789c636064000310a501c49c50360710a715e5e7a69766a6304041b6dc91755922b39515806c762475f94959a9c9250c0c8a507518f26969c5a9250c7000926743924faa2c492d06d21792b1eb2fc987b8e2c8921e75b667af767b20c97382e5f3127353191872138b4b528bf48b520b723293134b52f543c21c8c0c0c2d740d0d0c4d750d80203e253133a712d91dbc0c08ffe72416a5a726e5e42767e76783c424a0f6c3e453739352538052207ddc48e2c9f9b90545a9c5c5105d1000006fc83351 | ssh <remote IP> zfs recv -s -o autosnapshot:original=false -u -F tank/local/NAS-Backup/TV

but I got the error message cannot receive incremental stream: destination tank/local/NAS-Backup/TV contains partially-complete state from "zfs receive -s".

I don't understand this. I'm using the token to resume the send/recv, but it errors out saying there's a partially complete state.

I also tried removing some of the receive options, switching it to zfs recv -s -F tank/local/NAS-Backup/TV, but that didn't work either.

Describe how to reproduce the problem

Not sure.

Include any warning/errors/backtraces from the system logs

When I use the verbose options, I get

resume token contents:
nvlist version: 0
    fromguid = 0x239b146aaec41e6b
    object = 0x21
    offset = 0x0
    bytes = 0x63d0
    toguid = 0xbbeae606278ca4c4
    toname = master/replicate/TV@2018-1015-0000_daily
    largeblockok = 1
    embedok = 1
    compressok = 1
send from master/replicate/TV@2018-1001-0000_monthly to master/replicate/TV@2018-1015-0000_daily estimated size is 162G
TIME        SENT   SNAPSHOT
receiving incremental stream of master/replicate/TV@2018-1015-0000_daily into tank/local/NAS-Backup/TV@2018-1015-0000_daily
cannot receive incremental stream: destination tank/local/NAS-Backup/TV contains partially-complete state from "zfs receive -s".

[seonwoolee]

How is this a support request and not a bug? I'm clearly using the zfs send token as intended, but it's reporting that I'm not using the send token.

[helamonster]

FYI: I just encountered this myself and discovered the cause.

In my case, the zfs recv process was still running on the destination server. Once I killed it, the send/recv with the token worked normally.

I was also able to reproduce this by attempting to start another zfs send -t ... while it was already running.

This is clearly a situation that needs to be handled by any automated send/recv mechanisms.

[kangelos]

@helamonster you are a savior