Conemu: Unable to launch Bash on Ubuntu on Windows on Windows Insider Build 15002

[asser-dk]

Versions

ConEmu build: 161206 x64
OS version: Windows 10 Pro x64 Insider Preview 15002.rs_prerelease.170102-1700
Used shell version (Far Manager, git-bash, cmd, powershell, cygwin, whatever): Bash on Ubuntu on Windows

Problem description

I updated my Windows Installation to Insider build 15002 this morning and now I'm unable to open bash with ConEmu. When I try to open bash I get the following error: Exception 0xC0000005 (Write x0000000001D26BEE) was occurred (ConEmu64.exe, PID=3944) ConEmu build 161206 64

I'm able to open Bash outside ConEmu just fine and I can open cmd, Powershell and git-bash (bundled with git from git-scm.com) in ConEmu just fine.

I tried to reinstall conemu and I tried to delete the conemu.xml file from AppDataRoaming but the issue persists.

Steps to reproduce

1. Update to Windows Insider build 15002
2. Open Bash on Ubuntu on Windows using ConEmu

Repro rate: 5/5

Actual results

An error popup appears (see screenshot "Capture").
When dismissing the popup ConEmu shuts down and Bash opens in a new non-ConEmu window (see screenshot "Capture2")

Expected results

Bash should open normally inside ConEmu

Additional files

Memory dump, settings, screenshots, logs

[asser-dk]

I forgot to add the file to the shared folder >.< fixed now I hope

[Maximus5]

Check the permissions? Download forbidden...

[asser-dk]

Try this link instead. https://drive.google.com/open?id=0B9uKIrTjhg2NcjE1Qm4yTF9IaVk

[Maximus5]

It works!

[Maximus5]

The code raises crash in the Windows kernel. @miniksa is that expected?

DWORD dwType, dwSize;
RegQueryValueEx(hKeyNames, L"Counters", 0, &dwType, NULL, &dwSize);

[Maximus5]

Try this test build
https://www.dropbox.com/s/vkp1yz14ataj4e8/ConEmu64.exe.170110.7z?dl=0

[asser-dk]

I got this new error instead:

Exception 0xC0000005 (Write x0000000001D26BFD) was occurred (ConEmu64.exe, PID=16536) ConEmu build 170110 64

logs and memory dump: https://drive.google.com/open?id=0B9uKIrTjhg2NWXFYMmJ3UkJVSG8

Just to be clear this wasn't me doing something wrong. I just replaced the ConEmu64.exe in my installation with the one provided in the zip. Was this correct?

[Maximus5]

Was this correct?

Yep.

Things are worse that I can imagine. The crash occurs at

~~~

advapi32.dll!PerflibciSetObjectsValidityState()
advapi32.dll!PerflibciEnsureCounterSetList()
advapi32.dll!PerflibciEnsurePerflibV2StringTable()
advapi32.dll!PerfGetNames()
advapi32.dll!_guard_dispatch_icall_nop()
KERNELBASE.dll!LocalBaseRegQueryValue()
KERNELBASE.dll!RegQueryValueExW()
~~~

That means, the Insider build 15002 is broken completely. ConEmu can't do anything with that.

[asser-dk]

Bummer. Would you know if it makes sense that I pass this bug along to the BashOnWindows team? And if so, do you have any information from the crash I could include in the issue?

[Maximus5]

The bug doesn't relate to BashOnWindows. It's native Windows API bug.

[Maximus5]

Perhaps it depends on BashOnWindows, but only MSDT devs may check that.

[miniksa]

I'm not able to reproduce this issue on a clean x64 VM of 15002.rs_prerelease.170102-1700.

I also couldn't find any records of bugs/issues logged against the performance counters code behind RegQueryValueExW nor do I see any recent changes to that code.

[Maximus5]

@asser-dk I see BitDefender in the loaded modules. I'm almost sure it's not a problem, but can you run Bash/ConEmu without it to ensure?

~
avcuf64.dll *C:\Program Files\Bitdefender\Endpoint Security\Signatures\AVC\AVC3_00565_030\avcuf64.dll 3.12.17122.6492 25.11.2016 14:46 000000006C0B0000-000000006C15E000
~

@miniksa Can we do anything to troubleshoot or report the problem? At the moment it looks like a race in advapi32.dll (PerflibciSetObjectsValidityState). Two crash dump are available.

~
advapi32.dll 10 10.00.15002.1001 13.09.1936 0:19 00007FF93D9D0000-00007FF93DA78000
~

~~~

advapi32.dll!PerflibciSetObjectsValidityState()
advapi32.dll!PerflibciEnsureCounterSetList() + 0xb7 bytes
advapi32.dll!PerflibciEnsurePerflibV2StringTable() + 0xd6 bytes
advapi32.dll!PerfGetNames() + 0x40a bytes
advapi32.dll!_guard_dispatch_icall_nop() + 0x5c0f bytes
KERNELBASE.dll!LocalBaseRegQueryValue() + 0x43f bytes
KERNELBASE.dll!RegQueryValueExW() + 0xf6 bytes
~~~

[miniksa]

I looked again at the dump and it's very strange.

0:006> uf advapi32!PerflibciSetObjectsValidityState
advapi32!PerflibciSetObjectsValidityState:
 2194 00007ff9`3d9d6d18 0000            add     byte ptr [rax],al
 2194 00007ff9`3d9d6d1a 50              push    rax
 2194 00007ff9`3d9d6d1b c3              ret

Having only 3 assembly instructions for this function (after looking at the source it is generated from) doesn't make any sense and makes me suspect something on @asser-dk's system has tampered with advapi32.

In comparison on my 15014 system, calling uf on the same function gives me 50+ assembly instructions, not 3. And it doesn't start with add byte ptr[rax], al, it starts with the more sensible retrieval of the stack variable with mov instructions to retrieve the single TRUE/FALSE parameter to this function.

0:007> uf advapi32!PerflibciSetObjectsValidityState
ADVAPI32!PerflibciSetObjectsValidityState:
 2194 00007fff`84ba6d18 488b0424        mov     rax,qword ptr [rsp]
 2194 00007fff`84ba6d1c 6448890424      mov     qword ptr fs:[rsp],rax
 2194 00007fff`84ba6d21 53              push    rbx
 2194 00007fff`84ba6d22 4883ec20        sub     rsp,20h
 2194 00007fff`84ba6d26 8ad9            mov     bl,cl
.... (more lines, I clipped them out) ....

Additionally, the exception itself...

0:006> .ecxr
rax=0000000001d26bee rbx=0000000000000000 rcx=0000000000000000
rdx=00007ff93d9e180c rsi=0000000000000000 rdi=0000000000001000
rip=00007ff93d9d6d18 rsp=0000003ffb1fe168 rbp=0000003ffb1fe1d0
 r8=0000003ffb1fe0c8  r9=0000003ffb1fe1d0 r10=0000000000000000
r11=0000003ffb1fe168 r12=0000000000000063 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000
iopl=0         nv up ei pl zr na po nc
cs=0033  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010246
advapi32!PerflibciSetObjectsValidityState:
00007ff9`3d9d6d18 0000            add     byte ptr [rax],al ds:00000000`01d26bee=??

The exception is a write access error to the memory at 1d26bee. But that's not where the heaps are with !heap and it's not inside any library or valid address space with !address. So if it's trying to write to invalid memory for no particularly good reason, that's why we hit an access violation.

But reading the source code for this function... there is no write operation that should be occurring immediately on entry to PerflibciSetObjectsValidityState. It should have grabbed its parameters off the stack, checked that the lock variable was not null (read access), and then jumped immediately to kernel32!WaitForSingleObject on the lock value it just checked.

So I further suspect something has tampered with advapi32.dll.

BitDefender could be a good suspect. AntiVirus applications commonly place hooks inside system DLLs and detour them. Perhaps this was a bad detouring of advapi32 or a hook/detour that can't compensate for some compiler/linker update that adjusted the layout of advapi32.dll.

So as of now, I don't really have anything to file unfortunately as it doesn't look like a race condition at all to me. It looks like some other piece of software on @asser-dk's system is tampering with system DLLs and that's not something we support. If it happens on a clean machine or another machine where we understand the full state of everything installed to reproduce it (so it can be proven that the system is at fault), I'm happy to file an internal bug and pass it along. @maximus5, you're not seeing this at all on your machines, correct?

[Maximus5]

Thanks for investigations! Unfortunately I have not yet updated any of my PC to 15002 build, so I can't say yes or no.
I believe you are right and the function was detoured.
ConEmu doesn't detour its own executable, so the problem might be in BitDefender.

[asser-dk]

@asser-dk I see BitDefender in the loaded modules. I'm almost sure it's not a problem, but can you run Bash/ConEmu without it to ensure?

Sorry for the late reply. I've uninstalled the insider update and I'm currently on insider build 14986 where everything seems to work fine. I had a bunch of other applications that failed as well so it could be possible that the installation of the update was somehow flawed on my machine.

[nwykes]

As another datapoint, bash is working for me fine in insider build 15007 with 161206 and 170118 on two different machines. No Bitdefender.

[t-anjan]

I can confirm that the problem is indeed Bitdefender.

I am on 15019 Insider build. I had the same problem as the OP. Reading the posts here, I disabled BitDefender. And it just worked. I am able to reproduce the error every single time with Bitdefender turned on.

So, should we be reporting this to Bitdefender?

I should probably try replacing Bitdefender with another antivirus and check if the issue persists.