Iotedge: IoT Edge Build on Dev Box AMD64 Deploy to ARM64v8

[juansuerogit]

I have an NVIdia Jetson TX2 that ive successful hooked up to IoT Hub.
its an ARM64v8 ( arch64 ) device.

but my development environment is AMD64.

when i try to build the solution it fails with
standard_init_linux.go:190: exec user process caused "exec format error"

Step 1/12 : FROM mcr.microsoft.com/dotnet/core/sdk:3.0-buster-arm64v8 AS build-env
---> 0ca5aebeb047
Step 2/12 : WORKDIR /app
---> Using cache
---> 6c2033b25d95
Step 3/12 : COPY *.csproj ./
---> Using cache
---> 0ba74f345874
Step 4/12 : RUN dotnet restore
---> Running in 4a648bee3a74
standard_init_linux.go:190: exec user process caused "exec format error"

it works fine when i set my environment to AMD64, it even runs on the Nvidia jetson!
but i cant build it from my local machine.

i got here because i started with the following
https://github.com/toolboc/IntelligentEdgeHOL
then when that didnt work for me i found your article
https://devblogs.microsoft.com/iotdev/develop-and-debug-arm64-iot-edge-modules-in-visual-studio-code-preview/

Expected Behavior

Be able to develop modules in my development machine and just press a button and deploy them to the device.

Current Behavior

not able to build and this push my module because my dev machine is a different architecture than the edge device even though both are 64 bit and bott run Ubuntu 18.04

Steps to Reproduce

1. Preferences Settings.json...

   "azure-iot-edge.platforms": {
   "amd64": [],
   "arm32v7": [],
   "windows-amd64": [],
   "arm64v8": [],
   },
   "azure-iot-edge.version.csharpmodule": "3.0.0-alpha"

2. Set default platform to arm64v8

3. create a new edge solution and C# module
4. try to build and push

Context (Environment)

dev box: ubuntu 18.04
edge device: nvidia jetson tx2

Output of iotedge check

sudo iotedge check
[sudo] password for meta:

Configuration checks

√ config.yaml is well-formed
√ config.yaml has well-formed connection string
√ container engine is installed and functional
√ config.yaml has correct hostname
√ config.yaml has correct URIs for daemon mgmt endpoint
‼ latest security daemon
Installed IoT Edge daemon has version 1.0.8~rc1 but version 1.0.8 is available.
Please see https://aka.ms/iotedge-update-runtime for update instructions.
√ host time is close to real time
√ container time is close to host time
‼ DNS server
Container engine is not configured with DNS server setting, which may impact connectivity to IoT Hub.
Please see https://aka.ms/iotedge-prod-checklist-dns for best practices.
You can ignore this warning if you are setting DNS server per module in the Edge deployment.
‼ production readiness: certificates
Device is using self-signed, automatically generated certs.
Please see https://aka.ms/iotedge-prod-checklist-certs for best practices.
√ production readiness: certificates expiry
‼ production readiness: container engine
Device is not using a production-supported container engine (moby-engine).
Please see https://aka.ms/iotedge-prod-checklist-moby for details.
‼ production readiness: logs policy
Container engine is not configured to rotate module logs which may cause it run out of disk space.
Please see https://aka.ms/iotedge-prod-checklist-logs for best practices.
You can ignore this warning if you are setting log policy per module in the Edge deployment.

Connectivity checks

√ host can connect to and perform TLS handshake with IoT Hub AMQP port
√ host can connect to and perform TLS handshake with IoT Hub HTTPS port
√ host can connect to and perform TLS handshake with IoT Hub MQTT port
√ container on the default network can connect to IoT Hub AMQP port
√ container on the default network can connect to IoT Hub HTTPS port
√ container on the default network can connect to IoT Hub MQTT port
√ container on the IoT Edge module network can connect to IoT Hub AMQP port
√ container on the IoT Edge module network can connect to IoT Hub HTTPS port
√ container on the IoT Edge module network can connect to IoT Hub MQTT port
√ Edge Hub can bind to ports on host

One or more checks raised warnings. Re-run with --verbose for more details.

Click here

// Paste here

Device (Host) Operating System

Ubuntu 18.04

Architecture

amd64

Container Operating System

Linux

Runtime Versions

iotedged

Edge Agent

1.0.8

Edge Hub

1.0.8

Docker

<Run `docker version` (`docker -H npipe:////./pipe/iotedge_moby_engine version` for Moby on Windows) and paste here>

dev box : 18.03.1-ce
device: 18.09.2

Logs

iotedged logs

<Paste here>

edge-agent logs

<Paste here>

edge-hub logs

<Paste here>

Additional Information

[juansuerogit]

So now if i switch my module to
FROM mcr.microsoft.com/dotnet/core/sdk:3.0-buster AS build-env
FROM mcr.microsoft.com/dotnet/core/runtime:3.0-buster-slim

its builds on the dev machine but when device goes to pull and run it the docker logs say
standard_init_linux.go:207: exec user process caused "exec format error"

[philipktlin]

If I understand correctly, you want to build a ARM64 docker image running your .net console app on amd64 machine. To do this, you need to have 2 stages in docker file. I drafted a sample docker file for your reference below.

FROM AS builder

don't run dotnet restore

dotnet publish -r linux-arm64 -c Release -f netcoreapp2.1

FROM
COPY --from=builder
CMD

[juansuerogit]

actually the publish command should be
RUN dotnet publish -r linux-arm -c Release -o out -f netcoreapp3.0
since im trying on raspberry pi which is arm32 ( before i was using nvidia jetson which is arm64)

if i publish to linux-arm64 like you said i get errors.
( on the raspberry pi)

in addition of note is that the build stage should be a 64bit AMD container
e.g. sdk:3.0-stretch
otherwise the publish command wont work.

also dotnet publish actually does an implicit dotnet restore
https://github.com/dotnet/announcements/issues/23

finally here is my dockerfile that worked but its probably not ideal practice to develop and build modules on a dev machine thats a different architecture than the device you will deploy to.
even if its just nodejs or python because some of those node or python dependencies actually compile when you install them

but heres the rub...
there is not version of visual studio code that runs on 32bit CPU!. so even if i bought an extra device just for development, ----->it wouldnt work!!! <------

i guess the only solution is to use visual studio code just to get a shell of the project into source control then i can git clone that thing onto a 32 bit operating system for the rest of the development lifecycle?

FROM mcr.microsoft.com/dotnet/core/sdk:3.0-stretch AS build-env
WORKDIR /app

COPY *.csproj ./

RUN dotnet restore

COPY . ./
RUN dotnet publish -r linux-arm -c Release -o out -f netcoreapp3.0

FROM mcr.microsoft.com/dotnet/core/runtime:3.0-stretch-slim-arm32v7
WORKDIR /app
COPY --from=build-env /app/out ./

RUN useradd -ms /bin/bash moduleuser

USER moduleuser

ENTRYPOINT ["dotnet", "SampleModule51.dll"]

[veyalla]

Also see:
https://devblogs.microsoft.com/iotdev/develop-and-debug-arm64-iot-edge-modules-in-visual-studio-code-preview/

[juansuerogit]

ok had to upgrade to Docker version 19.03.1, build 74b1e89 on the development machine and on the pi
not sure how thats going to work with moby-engine
so ill leave regular docker for now.

otherwise you get an error trying to bind to remote docker over ssh

[juansuerogit]

wow that actually worked.

[nishad1092]

@

wow that actually worked.

Hi @juansuerogit , did that link work which @veyalla has sent. I'm trying to develop the module in my system ( amd architecture) to deploy in Jetson nano (arm64) following this link too, but I'm getting error like( build an image in Windows against a non windows system ) and deployed to the edge but no response on Jetson , Tll me plz will it work if it is developed on a AMD machine for a arm64 when we set VScode environment for Arm64.