Lxd: recursive push for pushing entire directories

[mikemccracken]

It'd be nice to be able to lxc push a directory recursively

[srkunze]

+1

[stgraber]

We've gotten some more request for this, it'd indeed be pretty nice to have.

[stgraber]

Going to take a stab at that. Recursive push should be reasonably easy, recursive pull would likely be more difficult.

[stgraber]

But since this issue only mentions push, I'll do that part and we can track recursive pull separately.

[stgraber]

Hmm, I was hoping I could do this entirely from the client without needing a daemon change, but we unfortunately don't create paths leading to the target on push, so we will need a tiny API addition to make this possible which in turns will mean needing support for API extension detection in the client.

[ChrisLundquist]

When adding an LXD builder to Packer I had to implement this to support provisioners.

The overview of my solution is:

$ cd /path/to/send
$ tar -czf - . | lxc file push - container/path/payload.tar.gz
$ lxc exec container -- bash -c "cd path && tar -xzf payload.tar.gz && rm -f payload.tar.gz"

There are probably a few corner cases.

• This isn't atomic.
  
  
  
  • You could make it atomic by uploading and extracting to a tmp directory, then moving it into place.
  
  
• The owner/group might be messed up.
  
  
  
  • You could chown -R "$(whoami)": path/
  
  

On the bright side though, with this approach you don't need any daemon changes.

I ended up with this in Packer:

func (c *Communicator) UploadDir(dst string, src string, exclude []string) error {
        // NOTE:lxc file push doesn't yet support directory uploads.
        // As a work around, we tar up the folder, upload it as a file, then extract it

        tarFileName := "packer-upload.tar.gz"
        os.Chdir(src)
        tar, err := c.CmdWrapper(fmt.Sprintf("tar --exclude %s -czf - .", tarFileName))
        if err != nil {
                return err
        }

        cp, err := c.CmdWrapper(fmt.Sprintf("lxc file push - %s ", filepath.Join(c.ContainerName, dst, tarFileName)))
        if err != nil {
                return err
        }

        tarCmd := ShellCommand(tar)
        cpCmd := ShellCommand(cp)

        cpCmd.Stdin, _ = tarCmd.StdoutPipe()
        log.Printf("Starting tar command: %s", tar)
        err = tarCmd.Start()
        if err != nil {
                return err
        }

        log.Printf("Running cp command: %s", cp)
        err = cpCmd.Run()
        if err != nil {
                log.Printf("Error running cp command: %s", err)
                return err
        }

        err = tarCmd.Wait()
        if err != nil {
                log.Printf("Error running tar command: %s", err)
                return err
        }

        extract := fmt.Sprintf("cd %s && tar -xzf %s", dst, tarFileName)
        log.Printf("Extracting upload: %s", extract)

        extractCmd, _ := c.Execute(extract)
        err = extractCmd.Run()
        if err != nil {
                log.Printf("Error running extract command: %s", err)
                return err
        }

        cleanup := fmt.Sprintf("rm -f %s", filepath.Join(dst, tarFileName))
        log.Printf("Extracting upload: %s", extract)

        cleanupCmd, _ := c.Execute(cleanup)
        err = cleanupCmd.Run()
        if err != nil {
                log.Printf("Error running cleanup command: %s", err)
                return err
        }

        return nil
}

[stgraber]

You could make things slightly simpler with:

tar cf - /path/on/host | lxc exec container -- tar xvf - -C /path/on/target

[stgraber]

That's the workaround I've been using myself so far

[ChrisLundquist]

You could make things slightly simpler with:

I didn't know I could pipe to lxc exec. Nice!

[ChrisLundquist]

@stgraber do you think it would make sense to have lxc file push -r ... or lxc dir push ... be a "macro" for the above, which could be improved later?

[stgraber]

Hmm, it'd be kinda difficult to support. The lxc client works on Windows and OS X too and may be run against containers which do not contain tar, in either case, it'd then fail.

I'd rather we do this right in the API. There isn't very much missing, we just lack the ability to create directories right now.

[ChrisLundquist]

Thank you!

[adbig]

This feature is very usefull. Do you think it will be ported to 2.0.x branch ? We use this stable branch for production purposes .... Thank you in advance !

[stgraber]

No, we don't backport new features and API changes to the LTS branch.