Airsim: Unreal engine simulation crash in ubuntu 16.04

[VineeshKK]

Hello,
I am trying to launch the simulation using Unreal engine in Ubuntu. Unreal engine crashed after pressing the play button for simulation. gamemode is et to SimGamemode.
can someone help to resolve this?

setup:

1. Airsim and Unrealengine are built and created the test project: MyProject.uproject
2. Airsim plugin is loaded in the test project.
3. Selected the gamemode as SimGameMode from WorldSettings
4. Press the play button for simulation.
   Did i miss anything here?

titled_1.Untitled_1 (0.002435s)
Signal 11 caught.
Malloc Size=131076 LargeMemoryPoolOffset=131092
CommonLinuxCrashHandler: Signal=11
Malloc Size=65535 LargeMemoryPoolOffset=196655
[2017.04.26-13.10.25:134][868]LogCrashTracker:

[2017.04.26-13.10.25:135][868]LogLinux: === Critical error: ===
Unhandled Exception: SIGSEGV: invalid attempt to read memory at address 0x0000000000000002

[2017.04.26-13.10.25:135][868]LogLinux: Fatal error!

[Callstack] 00 0x00007fd889be723f FLinuxPlatformStackWalk::CaptureStackBackTrace(unsigned long long, unsigned int, void)
[Callstack] 01 0x00007fd889acff65 FGenericPlatformStackWalk::StackWalkAndDump(char, unsigned long, int, void)
[Callstack] 02 0x00007fd889ba8621 FLinuxCrashContext::CaptureStackTrace()
[Callstack] 03 0x00007fd87fde61c0 CommonLinuxCrashHandler(FGenericCrashContext const&)
[Callstack] 04 0x00007fd889bac79a PlatformCrashHandler(int, siginfo_t, void)
[Callstack] 05 0x00007fd88f8a6390 /lib/x86_64-linux-gnu/libpthread.so.0(+0x11390) [0x7fd88f8a6390]
[Callstack] 06 0x00007fd889b4d008 FMallocBinned::Private::FreeInternal(FMallocBinned&, void)
[Callstack] 07 0x00007fd82155725a std::__1::basic_filebuf >::~basic_filebuf()
[Callstack] 08 0x00007fd8215453bd ASimModeWorldBase::~ASimModeWorldBase()
[Callstack] 09 0x00007fd889376eb6 IncrementalPurgeGarbage(bool, float)
[Callstack] 10 0x00007fd889379087 CollectGarbageInternal(EObjectFlags, bool)
[Callstack] 11 0x00007fd8893797d7 CollectGarbage(EObjectFlags, bool)
[Callstack] 12 0x00007fd8861f79f7 UGameInstance::InitializeForPlayInEditor(int, FGameInstancePIEParameters const&)
[Callstack] 13 0x00007fd8816524fd UEditorEngine::CreatePIEGameInstance(int, bool, bool, bool, bool, float)
[Callstack] 14 0x00007fd88164ec5a UEditorEngine::PlayInEditor(UWorld, bool)
[Callstack] 15 0x00007fd881642e80 UEditorEngine::StartQueuedPlayMapRequest()
[Callstack] 16 0x00007fd881141511 UEditorEngine::Tick(float, bool)
[Callstack] 17 0x00007fd88194e0d9 UUnrealEdEngine::Tick(float, bool)
[Callstack] 18 0x000000000041fffe FEngineLoop::Tick() [Runtime/Core/Public/Stats/Stats2.h, line 1686]
[Callstack] 19 0x000000000042480a GuardedMain(wchar_t const) [/home/vkk/Unreal/UnrealEngine/Engine/Source/Runtime/Launch/Private/Launch.cpp, line 166]
[Callstack] 20 0x00007fd87fde6f4b CommonLinuxMain(int, char, int ()(wchar_t const*))
[Callstack] 21 0x00007fd87f18e830 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0) [0x7fd87f18e830]
[Callstack] 22 0x0000000000415029 /home/vkk/Unreal/UnrealEngine/Engine/Binaries/Linux/UE4Editor(_start+0x29) [0x415029]
............................
............................
CrashDebugHelper (8)
[2017.04.26-12.48.05:453][ 0]LogModuleManager: Shutting down and abandoning module QoSReporter (6)
[2017.04.26-12.48.05:453][ 0]LogModuleManager: Shutting down and abandoning module Analytics (4)
[2017.04.26-12.48.05:453][ 0]LogModuleManager: Shutting down and abandoning module CoreUObject (2)
[2017.04.26-12.48.05:453][ 0]LogExit: Exiting.
Engine crash handling finished; re-raising signal 11 for the default handler. Good bye.

[VineeshKK]

@lovettchris, @AlexandreBorowczyk
Could you help on this?

[sytelus]

I suspect this is because of known issue rpclib. #lovettchris recently added json setting to turn rpclib off just so you can get past that.

[VineeshKK]

Thanks.. do you know the path to Json file.

[lovettchris]

~/Documents/AirSim/settings.json.

We are not sure any more that this is caused by rpclib. On windows the crash was caused by having wrong compile flags on mavlinkcom library, but since Linux build is cmake based, this is unlikely to be the same cause, but if the AirSim compile flags don't match what the unreal engine requires, then that could be an issue to look for.

[AlexandreBorowczyk]

I can confirm the problem is still present with the latest commits.

[Mayankm96]

Yes it still persistent. Would like to hear updates on this! @AlexandreBorowczyk , @lovettchris

Thank you!

[andre-nguyen]

Anyone else looking into this? Stepping through the code, the segfault just doesn't seem to make sense. After these lines here https://github.com/Microsoft/AirSim/blob/master/AirLib/include/controllers/Settings.hpp#L43 we get a segfault right before return.

std::__1::basic_ifstream<char, std::__1::char_traits<char> >::open (__mode=8, this=<optimized out>, __s=...) at ThirdParty/Linux/LibCxx/include/c++/v1/fstream:1150
1150            this->clear();
(gdb) 
43                  common_utils::FileSystem::openTextFile(path, s);
(gdb) 
44                  if (!s.fail()) {
(gdb) 
45                      s >> settings_.doc_;
(gdb) 
46                      settings_.load_success_ = true;
(gdb) 
50              }
(gdb) 

Thread 1 "UE4Editor" received signal SIGSEGV, Segmentation fault.
FMallocBinned::Private::FreeInternal (Allocator=..., Ptr=<optimized out>) at /home/inflatablequad/Documents/UnrealEngine/Engine/Source/Runtime/Core/Private/HAL/MallocBinned.cpp:442
442         if (Pool->TableIndex < Allocator.BinnedOSTableIndex)

And the back trace is

(gdb) bt
#0  FMallocBinned::Private::FreeInternal (Allocator=..., Ptr=<optimized out>) at /home/inflatablequad/Documents/UnrealEngine/Engine/Source/Runtime/Core/Private/HAL/MallocBinned.cpp:442
#1  0x00007fff7ae88eda in operator delete[] (Ptr=0x1d81fff) at /home/inflatablequad/Documents/Unreal Projects/MyProject/Plugins/AirSim/Source/AirSim.cpp:12
#2  std::__1::basic_filebuf<char, std::__1::char_traits<char> >::~basic_filebuf (this=0x7fffffffa6e0) at ThirdParty/Linux/LibCxx/include/c++/v1/fstream:372
#3  0x00007fff7ae809a9 in std::__1::basic_ifstream<char, std::__1::char_traits<char> >::~basic_ifstream (this=<optimized out>, vtt=<optimized out>) at ThirdParty/Linux/LibCxx/include/c++/v1/iosfwd:131
#4  std::__1::basic_ifstream<char, std::__1::char_traits<char> >::~basic_ifstream (this=<optimized out>) at ThirdParty/Linux/LibCxx/include/c++/v1/iosfwd:131
#5  msr::airlib::Settings::loadJSonFile (fileName=...) at /home/inflatablequad/Documents/Unreal Projects/MyProject/Plugins/AirSim/Source/AirLib/include/controllers/Settings.hpp:50
#6  0x00007fff7ae6e571 in ASimModeBase::initializeSettings (this=0x7fff6dadaf60) at /home/inflatablequad/Documents/Unreal Projects/MyProject/Plugins/AirSim/Source/SimMode/SimModeBase.cpp:41

So basically the segfault comes from the destructor of the ifstream?

[andre-nguyen]

After spending quite some time pondering this, I've decided to simply bypass the problem for now. If you disable functions pertaining to file IO for the settings you can get the quad to spawn.

If anyone wants to bypass things for now I pushed my hack here https://github.com/andre-nguyen/AirSim/tree/hardcode

Update: the latest code in my branch works with px4 sitl

[Mayankm96]

Thanks @andre-nguyen. I am able to spawn the quadrotor into the scene. :)

However, the UE4Editor seems to crash when I stop the simulation. :P

[2017.05.13-09.35.18:052][  0]LogUObjectArray: 146 objects as part of root set at end of initial load.
[2017.05.13-09.35.18:052][  0]LogUObjectAllocator: 24096 out of 0 bytes used by permanent object pool.
[2017.05.13-09.35.18:052][  0]LogUObjectArray: CloseDisregardForGC: 0/0 objects in disregard for GC pool
[2017.05.13-09.35.18:056][  0]LogTextLocalizationManager: No translations for ('') exist, falling back to 'en' for localization and internationalization data.
[2017.05.13-09.35.18:063][  0]CrashReportClientLog: CrashReportClientVersion=1.0
[2017.05.13-09.35.18:063][  0]CrashReportClientLog: CrashReportReceiver disabled
[2017.05.13-09.35.18:063][  0]CrashReportClientLog: DataRouterUrl: https://datarouter.ol.epicgames.com/datarouter/api/v1/public/data
[2017.05.13-09.35.18:107][  0]LogQoSReporter:Error: QoS API server is not configured, no QoS metrics will be reported.
[2017.05.13-09.35.18:107][  0]LogQoSReporter: QoSReporter initialized (InstanceId = '0004F22312050B0D0023006B8A3CA3C6', SystemId = '')
[2017.05.13-09.35.18:107][  0]LogQoSReporter: APIKey = 'CrashReportClient.Release'. APIServer = ''. AppVersion = 'UE4-CL-0'. AppEnvironment = 'Release'
[2017.05.13-09.35.18:107][  0]LogQoSReporter: QoSReporter has been configured without a valid deployment name.
[2017.05.13-09.35.18:107][  0]LogQoSReporter: QoSReporter has been configured without a valid deployment name.
[2017.05.13-09.35.18:107][  0]CrashReportClientLog: Initial state = Unknown UploadState value
[2017.05.13-09.35.18:107][  0]CrashReportClientLog: Initial state = Unknown UploadState value
[2017.05.13-09.35.18:112][  0]LogCrashDebugHelper: DepotName: //UE4/Release-4.15
[2017.05.13-09.35.18:112][  0]LogCrashDebugHelper: BuiltFromCL: 0
[2017.05.13-09.35.18:112][  0]LogCrashDebugHelper: EngineVersion: 4.15.2-0+++UE4+Release-4.15
[2017.05.13-09.35.18:112][  0]LogCrashDebugHelper: BuildVersion: ++UE4+Release-4.15-CL-0
[2017.05.13-09.35.18:112][  0]LogCrashDebugHelper:Warning: CrashDebugHelperConfig invalid
[2017.05.13-09.35.18:112][  0]LogCrashDebugHelper:Warning: PDB Cache disabled
[2017.05.13-09.35.19:172][  0]CrashReportClientLog: Got 5 pending files to upload from 'crashinfo-MyProject-pid-1737-679DB0170F050B0D00230302751D6153'
[2017.05.13-09.35.19:172][  0]CrashReportClientLog: State change from Ready to SendingFiles
[2017.05.13-09.35.19:172][  0]CrashReportClientLog: CompressAndSendData have 5 pending files
[2017.05.13-09.35.19:172][  0]CrashReportClientLog: CompressAndSendData compressing 1437 bytes ('/home/mittalm/Documents/UnrealProjects/MyProject/Saved/Crashes/crashinfo-MyProject-pid-1737-679DB0170F050B0D00230302751D6153/Diagnostics.txt')
[2017.05.13-09.35.19:173][  0]CrashReportClientLog: CompressAndSendData compressing 85 bytes ('/home/mittalm/Documents/UnrealProjects/MyProject/Saved/Crashes/crashinfo-MyProject-pid-1737-679DB0170F050B0D00230302751D6153/CrashReportClient.ini')
[2017.05.13-09.35.19:174][  0]CrashReportClientLog: CompressAndSendData compressing 4 bytes ('/home/mittalm/Documents/UnrealProjects/MyProject/Saved/Crashes/crashinfo-MyProject-pid-1737-679DB0170F050B0D00230302751D6153/minidump.dmp')
[2017.05.13-09.35.19:213][  0]CrashReportClientLog: CompressAndSendData compressing 56117 bytes ('/home/mittalm/Documents/UnrealProjects/MyProject/Saved/Crashes/crashinfo-MyProject-pid-1737-679DB0170F050B0D00230302751D6153/MyProject.log')
[2017.05.13-09.35.19:219][  0]CrashReportClientLog: CompressAndSendData compressing 1927 bytes ('/home/mittalm/Documents/UnrealProjects/MyProject/Saved/Crashes/crashinfo-MyProject-pid-1737-679DB0170F050B0D00230302751D6153/wermeta.xml')
[2017.05.13-09.35.19:256][  0]LogInit: Using libcurl 7.48.0-DEV
[2017.05.13-09.35.19:256][  0]LogInit:  - built for x86_64-unknown-linux-gnu
[2017.05.13-09.35.19:256][  0]LogInit:  - supports SSL with OpenSSL/1.0.2h
[2017.05.13-09.35.19:256][  0]LogInit:  - supports HTTP deflate (compression) using libz 1.2.8
[2017.05.13-09.35.19:256][  0]LogInit:  - other features:
[2017.05.13-09.35.19:256][  0]LogInit:      CURL_VERSION_SSL
[2017.05.13-09.35.19:256][  0]LogInit:      CURL_VERSION_LIBZ
[2017.05.13-09.35.19:257][  0]LogInit:      CURL_VERSION_IPV6
[2017.05.13-09.35.19:257][  0]LogInit:      CURL_VERSION_ASYNCHDNS
[2017.05.13-09.35.19:257][  0]LogInit:      CURL_VERSION_LARGEFILE
[2017.05.13-09.35.19:257][  0]LogInit:      CURL_VERSION_TLSAUTH_SRP
[2017.05.13-09.35.19:257][  0]LogInit:  Libcurl: checking if '/etc/pki/tls/certs/ca-bundle.crt' exists
[2017.05.13-09.35.19:258][  0]LogInit:  Libcurl: checking if '/etc/ssl/certs/ca-certificates.crt' exists
[2017.05.13-09.35.19:258][  0]LogInit:  CurlRequestOptions (configurable via config and command line):
[2017.05.13-09.35.19:258][  0]LogInit:  - bVerifyPeer = true  - Libcurl will verify peer certificate
[2017.05.13-09.35.19:258][  0]LogInit:  - bUseHttpProxy = false  - Libcurl will NOT use HTTP proxy
[2017.05.13-09.35.19:258][  0]LogInit:  - bDontReuseConnections = false  - Libcurl will reuse connections
[2017.05.13-09.35.19:258][  0]LogInit:  - CertBundlePath = /etc/ssl/certs/ca-certificates.crt  - Libcurl will set CURLOPT_CAINFO to it
[2017.05.13-09.35.19:259][  0]CrashReportClientLog: Sending HTTP request: https://datarouter.ol.epicgames.com/datarouter/api/v1/public/data?AppID=CrashReporter&AppVersion=4.15.2-0%2B%2B%2BUE4%2BRelease-4.15&AppEnvironment=Release&UploadType=crashreports&UserID=00021d2d2a05120800090373778f9030%7C%7C
[2017.05.13-09.35.20:095][  0]CrashReportClientLog: OnProcessRequestComplete(), State=SendingFiles bSucceeded=1
[2017.05.13-09.35.20:095][  0]CrashReportClientLog: State change from SendingFiles to SendingFiles
[2017.05.13-09.35.20:095][  0]CrashReportClientLog: All uploads done
[2017.05.13-09.35.20:095][  0]CrashReportClientLog: State change from SendingFiles to Finished
[2017.05.13-09.35.20:197][  0]LogLinux: FLinuxPlatformMisc::RequestExit(0)
[2017.05.13-09.35.20:197][  0]LogGenericPlatformMisc: FPlatformMisc::RequestExit(0)
[2017.05.13-09.35.20:197][  0]CrashReportClientLog: Final state (Receiver) = Finished
[2017.05.13-09.35.20:197][  0]CrashReportClientLog: Final state (Receiver) = Unknown UploadState value
[2017.05.13-09.35.20:197][  0]LogExit: Preparing to exit.
[2017.05.13-09.35.20:198][  0]LogObj: Freed 0b from 0 cluster array pools.
[2017.05.13-09.35.20:198][  0]LogExit: Object subsystem successfully closed.
[2017.05.13-09.35.20:199][  0]LogModuleManager: Shutting down and abandoning module HTTP (12)
[2017.05.13-09.35.20:209][  0]LogModuleManager: Shutting down and abandoning module SSL (11)
[2017.05.13-09.35.20:211][  0]LogModuleManager: Shutting down and abandoning module CrashDebugHelper (8)
[2017.05.13-09.35.20:211][  0]LogModuleManager: Shutting down and abandoning module QoSReporter (6)
[2017.05.13-09.35.20:211][  0]LogModuleManager: Shutting down and abandoning module Analytics (4)
[2017.05.13-09.35.20:211][  0]LogModuleManager: Shutting down and abandoning module CoreUObject (2)
[2017.05.13-09.35.20:212][  0]LogExit: Exiting.
Engine crash handling finished; re-raising signal 6 for the default handler. Good bye.

[andre-nguyen]

You're more than welcome to help investigate the cause of that one too!

[viniciusguigo]

Still have the same problem running your fork @andre-nguyen

[viniciusguigo]

Hello, @lovettchris, @AlexandreBorowczyk.

Any updates on this issue? Thank you!

[ShukuiZhang]

Same error as @viniciusguigo, I'm using Ubuntu 14.04, Clang 3.9.1, UE 4.15, any idea on this crash? Thanks! @lovettchris @AlexandreBorowczyk

[AlexandreBorowczyk]

Yup, I gave it a try with andre's bypass and latest commits and I'm getting a seg fault.

[AlexandreBorowczyk]

Update, got it to work once. Don't know why it worked but never did again.
Here is where the sig fault occurse

[viniciusguigo]

Is it on the same place as before? Or did something change after the first time it worked?
Did you mean that you spawned the quad?

Thanks!

[AlexandreBorowczyk]

On andre's hardcoded branch it worked once and then without any changes... didn't work. However it doesn't crash at the same place as on the master branch.

[isvogor-foi]

Any progress on this?

[deonblaauw]

I'm having the same problem. When I press play, it just crashes. I then checked out Andre's branch, and changed the "UseSerial" to "true" in the SimModeBase.cpp file (I'm trying to run HITL using the PixHawk). After doing this, no more crashes, the quad spawns and I think it's connected to the pixhawk.

I am getting this message however: "Vehicle controller cannot be started, please check your settings.jsonstd::exception std::exception Cannot start RpcLib ServerInvalid argument"

I noticed in the same file is an option to disable RPC, so I tried it, but got the same result :).

The hardcoded settings are:
`
//write some settings in new file otherwise the string "null" is written if all settigs are empty
enable_rpc = settings.setBool("RpcEnabled", false);
settings.setString("LocalHostIp", "127.0.0.1");

        //Settings rosflight_child;
        rosflight_child.setInt("RemoteControlID", 0);
        settings.setChild("RosFlight", rosflight_child);*/
        fpv_vehicle_name = "Pixhawk";
        Settings pixhawk_child;
        pixhawk_child.setString("LogViewerHostIp", "127.0.0.1");
        pixhawk_child.setInt("LogViewerPort", 14388);
        pixhawk_child.setInt("OffboardCompID", 1);
        pixhawk_child.setInt("OffboardSysID", 134);
        pixhawk_child.setString("QgcHostIp", "127.0.0.1");
        pixhawk_child.setInt("QgcPort", 14550);
        pixhawk_child.setInt("SerialBaudRate", 115200);
        pixhawk_child.setString("SerialPort", "*");
        pixhawk_child.setInt("SimCompID", 42);
        pixhawk_child.setInt("SimSysID", 142);
        pixhawk_child.setString("SitlIp", "127.0.0.1");
        pixhawk_child.setInt("SitlPort", 14556);
        pixhawk_child.setString("UdpIp", "127.0.0.1");
        pixhawk_child.setInt("UdpPort", 14560);
        pixhawk_child.setBool("UseSerial", true);
        pixhawk_child.setInt("VehicleCompID", 1);
        pixhawk_child.setInt("VehicleSysID", 135);
        settings.setChild("Pixhawk", pixhawk_child);

`

Does anyone have some additional insights here?

[isvogor-foi]

Is there any way to change any of these parameters and rebuild the plugin, without having to compile 900 files of the UE? Whenever I change something, UE stars rebuilding libUE4Editor-AirSim.so, which is around 900 files and takes a hour or so.

[AlexandreBorowczyk]

@isvogor-foi Full UE should only recompile if their is a change in it own files or if the compilation configuration of the plugin are not the same as when UE was compile. e.i EU compile in release and plugin being compile in debug.

@viniciusguigo @lovettchris I've just tried AirSim on another computer, it still crash as before. However on this new computer @andre-nguyen 's hardcoded bypass works for me. The only thing is that EU crash went I quit the simulation. Also the simulation is very laggy, the other default UE4 project test don´t have this issue.

[lovettchris]

Please check you have the latest bits, there was a bug in an uninitialized variable that might explain a lot of crashes.

[fshamshirdar]

I've tried the latest commit, it still crash on both Ubuntu 16.04 and Windows 10 (VS2015). But @andre-nguyen hardcoded branch works.

[fshamshirdar]

It seems this is caused by RpcLib, in andre's hardcoded branch, api_server_address is missing and I guess this line is the main issue which is here https://github.com/Microsoft/AirSim/blob/master/Unreal/Plugins/AirSim/Source/SimMode/SimModeBase.cpp#L48 after reading Json file:
api_server_address = settings.getString("LocalHostIp", "127.0.0.1");

By leaving this variable. it doesn't crash but then we don't have access to APIs anymore. Here in this line: https://github.com/Microsoft/AirSim/blob/master/AirLib/src/rpc/RpcLibServer.cpp#L29 I tried other constructor of rpc::server which only needs port value, but it still crash.

[sytelus]

We been busy fixing various issues in Linux build. Now we have new code out that should build and run on Ubuntu 16 with few known issues. Please see docs) for instructions.

[asimay]

any update? or solve or not? I met the same issue with new version 0.7, on ubuntu 16.04...

[2017.12.15-01.15.58:151][ 0]LogExit: Preparing to exit.
[2017.12.15-01.15.58:151][ 0]LogExit: Object subsystem successfully closed.
[2017.12.15-01.15.58:156][ 0]LogModuleManager: Shutting down and abandoning module HTTP (8)
[2017.12.15-01.15.58:163][ 0]LogModuleManager: Shutting down and abandoning module SSL (7)
[2017.12.15-01.15.58:164][ 0]LogModuleManager: Shutting down and abandoning module CrashDebugHelper (4)
[2017.12.15-01.15.58:164][ 0]LogModuleManager: Shutting down and abandoning module CoreUObject (2)
[2017.12.15-01.15.58:164][ 0]LogExit: Exiting.
Engine crash handling finished; re-raising signal 11 for the default handler. Good bye.
Segmentation fault (core dumped)