Azure-storage-azcopy: Please make it work on Raspberry Pi!

[alvipeo]

Team!

Please make it work on Raspberry Pi!

Thank you very much!

[Virepri]

Hi, alvipeo. AzCopy is written in Go which supports building for ARM processors. We don't distribute an ARM build at the moment, but you're more than welcome to build it yourself. Would you like instruction on this?

[alvipeo]

Hi! Never tried Go but yes, please.

[silaspepperbyte]

Treid compiling for ARM but sadly the following error pops up:

silas@silas-Virtual-Machine:~/Desktop/azure-storage-azcopy-master$ env GOOS=linux GOARCH=arm go build
# github.com/jiacfan/keyctl
../../go/pkg/mod/github.com/jiacfan/[email protected]/sys_linux.go:100:29: undefined: syscall_keyctl
../../go/pkg/mod/github.com/jiacfan/[email protected]/sys_linux.go:102:36: undefined: syscall_keyctl
../../go/pkg/mod/github.com/jiacfan/[email protected]/sys_linux.go:133:34: undefined: syscall_add_key
../../go/pkg/mod/github.com/jiacfan/[email protected]/sys_linux.go:157:36: undefined: syscall_setfsgid

[alvipeo]

@Virepri , so how do we make it run on Rasp Pi?

[adreed-msft]

It seems as though jiacfan/keyctl has support for 32 & 64-bit system calls, though, no support whatsoever for other architectures. So that would be the trouble with compiling this for any other arch.

I'm looking into the values I need to supply at the moment.

[adreed-msft]

https://github.com/jiacfan/keyctl/pull/2

I've created a pull request containing some of the architectures I've seen asks for + their direct relatives on jiacfan/keyctl.

If you wish to compile AzCopy in the meantime, you can pop into go/src/github.com/jiacfan and replace the keyctl dir in that directory with mine under adreed-msft/keyctl.

[adreed-msft]

As to @alvipeo, pardon me for not providing directions yet. You'll need go installed, but it's a pretty straightforward process to compile.

Below are general commands to compile AzCopy. If you're trying to cross-compile from another machine, you should supply the appropriate GOOS and GOARCH environment variables.

Due to jiacfan/keyctl missing the required values, you should follow the instructions in the last post to substitute it with adreed-msft/keyctl if you wish to build ahead of time.

go get github.com/Azure/azure-storage-azcopy
cd $GOPATH/src/github.com/Azure/azure-storage-azcopy
go install
cd $GOPATH/bin

AzCopy should be located in $GOPATH/bin ($GOPATH is default ~/go) after it's built with these instructions.

[alvipeo]

Thank you! I'll give it a try!

You guys really should include that build in your automation.

[adreed-msft]

Yeah, it would be nice to distribute for more platforms but it's mostly a concern of properly testing all platforms. Perhaps we will start doing so if more of these scenarios pop up-- perhaps this will be a good reason for us to start testing against ARM properly in the near-ish future.

[alvipeo]

I'm sure there is already a demand for AzCopy on Rasp Pi. So with this offer it'll only increase. Better for Azure at the end (start it before others cloud providers do that) ;)

[adreed-msft]

Hi there, Alvipeo. My changes have made it into jiacfan/keyctl now. Folks should be able to compile AzCopy for Linux on ARM32, ARM64, PowerPC32, PowerPC64, and PowerPC64le unofficially now. Enjoy!

[nerijusk]

As to @alvipeo, pardon me for not providing directions yet. You'll need go installed, but it's a pretty straightforward process to compile.

Below are general commands to compile AzCopy. If you're trying to cross-compile from another machine, you should supply the appropriate GOOS and GOARCH environment variables.

Due to jiacfan/keyctl missing the required values, you should follow the instructions in the last post to substitute it with adreed-msft/keyctl if you wish to build ahead of time.

go get github.com/Azure/azure-storage-azcopy
cd $GOPATH/src/github.com/Azure/azure-storage-azcopy
go install
cd $GOPATH/bin

AzCopy should be located in $GOPATH/bin ($GOPATH is default ~/go) after it's built with these instructions.

Unfortunately does not build with the following errors:

root@pi:~/src/go/src/github.com/Azure/azure-storage-azcopy# go install
../azure-pipeline-go/pipeline/core.go:4:2: cannot find package "context" in any of:
        /usr/lib/go/src/pkg/context (from $GOROOT)
        /root/src/go/src/context (from $GOPATH)
../../spf13/cobra/doc/man_docs.go:27:2: cannot find package "github.com/cpuguy83/go-md2man/v2/md2man" in any of:
        /usr/lib/go/src/pkg/github.com/cpuguy83/go-md2man/v2/md2man (from $GOROOT)
        /root/src/go/src/github.com/cpuguy83/go-md2man/v2/md2man (from $GOPATH)
../../minio/minio-go/api.go:45:2: cannot find package "github.com/minio/minio-go/v6/pkg/credentials" in any of:
        /usr/lib/go/src/pkg/github.com/minio/minio-go/v6/pkg/credentials (from $GOROOT)
        /root/src/go/src/github.com/minio/minio-go/v6/pkg/credentials (from $GOPATH)
../../minio/minio-go/api-compose-object.go:31:2: cannot find package "github.com/minio/minio-go/v6/pkg/encrypt" in any of:
        /usr/lib/go/src/pkg/github.com/minio/minio-go/v6/pkg/encrypt (from $GOROOT)
        /root/src/go/src/github.com/minio/minio-go/v6/pkg/encrypt (from $GOPATH)
../../minio/minio-go/api-presigned.go:26:2: cannot find package "github.com/minio/minio-go/v6/pkg/s3signer" in any of:
        /usr/lib/go/src/pkg/github.com/minio/minio-go/v6/pkg/s3signer (from $GOROOT)
        /root/src/go/src/github.com/minio/minio-go/v6/pkg/s3signer (from $GOPATH)
../../minio/minio-go/api-compose-object.go:32:2: cannot find package "github.com/minio/minio-go/v6/pkg/s3utils" in any of:
        /usr/lib/go/src/pkg/github.com/minio/minio-go/v6/pkg/s3utils (from $GOROOT)
        /root/src/go/src/github.com/minio/minio-go/v6/pkg/s3utils (from $GOPATH)
../../minio/minio-go/bucket-notification.go:23:2: cannot find package "github.com/minio/minio-go/v6/pkg/set" in any of:
        /usr/lib/go/src/pkg/github.com/minio/minio-go/v6/pkg/set (from $GOROOT)
        /root/src/go/src/github.com/minio/minio-go/v6/pkg/set (from $GOPATH)
../../../golang.org/x/sys/unix/affinity_linux.go:10:2: cannot find package "math/bits" in any of:
        /usr/lib/go/src/pkg/math/bits (from $GOROOT)
        /root/src/go/src/math/bits (from $GOPATH)
ste/xferLogPolicy.go:8:2: cannot find package "net/http/httptrace" in any of:
        /usr/lib/go/src/pkg/net/http/httptrace (from $GOROOT)
        /root/src/go/src/net/http/httptrace (from $GOPATH)

[JohnRusk]

@nerijusk Which version of Go where you using, and which OS were you running the build on? If your version of Go is too old, IIRC it won't use the module system to restore dependencies. 1.13 is recommended.

Probably don't do the build on the RPi itself. Probably build it on a Linux or Windows machine, setting the GOOS and GOARCH environment variables before building.

[nerijusk]

@nerijusk Which version of Go where you using, and which OS were you running the build on? If your version of Go is too old, IIRC it won't use the module system to restore dependencies. 1.13 is recommended.

Probably don't do the build on the RPi itself. Probably build it on a Linux or Windows machine, setting the GOOS and GOARCH environment variables before building.

$ go version
go version go1.3.3 linux/arm

A bit older than recommended. I'll try to upgrade or build on another machine.

UPDATE
Built on another machine and it worked on my old (gen 2) RPi. Thanks @JohnRusk!

[sadhosOIT]

Thanks, this worked for me!

Hardware: Jetson Nano, ARMv8
OS: Ubuntu 18.04 LTS
Solution works with go1.14.4.linux-arm64.tar.gz

[anterosa]

I get following error

pi@raspberrypi:~ $ go get github.com/Azure/azure-storage-azcopy github.com/Azure/azure-storage-azcopy imports github.com/Azure/azure-storage-azcopy/cmd imports github.com/Azure/azure-storage-blob-go/2018-03-28/azblob: cannot find module providing package github.com/Azure/azure-storage-blob-go/2018-03-28/azblob github.com/Azure/azure-storage-azcopy imports github.com/Azure/azure-storage-azcopy/cmd imports github.com/Azure/azure-storage-file-go/2017-07-29/azfile: cannot find module providing package github.com/Azure/azure-storage-file-go/2017-07-29/azfile

[CJTalbot87]

@anterosa I also have this issue, did you find a solution?