Voodooi2c: kernel_task high CPU usage while trackpad is used

Created on 23 Mar 2020  路  45Comments  路  Source: VoodooI2C/VoodooI2C

Whenever i touch the track,kernel_task cpu usage jumps from 2% to 35+% and stays there as long as im touching the trackpad.I'm using GPIO interrrupt mode.Here is my logs of i2c:

Timestamp (process)[PID]
2020-03-23 23:50:57.215399+0600 localhost kernel[0]: (kernel) VoodooI2CPCIController::pci8086,a368 Starting I2C controller
2020-03-23 23:50:57.215408+0600 localhost kernel[0]: (kernel) VoodooI2CPCIController::pci8086,a369 Starting I2C controller
2020-03-23 23:50:57.215607+0600 localhost kernel[0]: (kernel) VoodooI2CPCIController::pci8086,a369 Set PCI power state D0
2020-03-23 23:50:57.215846+0600 localhost kernel[0]: (kernel) VoodooI2CPCIController::pci8086,a369 Publishing nub
2020-03-23 23:50:57.216088+0600 localhost kernel[0]: (kernel) VoodooI2CControllerDriver::pci8086,a369 Probing controller
2020-03-23 23:50:57.216198+0600 localhost kernel[0]: (kernel) VoodooI2CControllerDriver::pci8086,a369 Found valid Synopsys component, continuing with initialisation
2020-03-23 23:50:57.223265+0600 localhost kernel[0]: (kernel) VoodooI2CControllerDriver::pci8086,a369 Got bus configuration values
2020-03-23 23:50:57.226298+0600 localhost kernel[0]: (kernel) VoodooI2CPCIController::pci8086,a368 Set PCI power state D0
2020-03-23 23:50:57.227228+0600 localhost kernel[0]: (kernel) VoodooI2CPCIController::pci8086,a368 Publishing nub
2020-03-23 23:50:57.228335+0600 localhost kernel[0]: (kernel) VoodooI2CControllerDriver::pci8086,a368 Probing controller
2020-03-23 23:50:57.232307+0600 localhost kernel[0]: (kernel) VoodooI2CControllerDriver::pci8086,a369 Publishing device nubs
2020-03-23 23:50:57.232534+0600 localhost kernel[0]: (kernel) VoodooI2CControllerDriver::pci8086,a369 Found I2C device: SYNA2393
2020-03-23 23:50:57.237661+0600 localhost kernel[0]: (kernel) VoodooI2CDeviceNub::Got GPIO Controller! VoodooGPIOCannonLakeH
2020-03-23 23:50:57.259132+0600 localhost kernel[0]: (kernel) VoodooI2CControllerDriver::pci8086,a368 Found valid Synopsys component, continuing with initialisation
2020-03-23 23:50:57.266564+0600 localhost kernel[0]: (kernel) VoodooI2CControllerDriver::pci8086,a368 Got bus configuration values
2020-03-23 23:50:57.266640+0600 localhost kernel[0]: (kernel) VoodooI2CControllerDriver::pci8086,a368 Publishing device nubs
2020-03-23 23:51:13.244866+0600 localhost kernel[0]: (kernel) VoodooI2CPrecisionTouchpadHIDEventDriver::setProperties USBMouseStopsTrackpad = 0

CPU Usage:
Screen Shot 2020-03-24 at 12 06 02 AM

Most helpful comment

@tiger511 It is a device-specific bug that only occurs on some touchpads with a physical button. I will add an option in plist to disable the Force Touch emulation.

All 45 comments

Interesting note: If i put a finger on the trackpad and do nothing ,kernel_task stays at 35% till i remove my finger.

@tiger511 I'm experiencing the same problem!

Even if i disable trackpad,this still happens.WEIRD!

@tiger511 Please try this and see if it fixed
VoodooI2C.kext.zip
VoodooI2CHID.kext.zip

@tctien342 thanks, i'll try.But can you give me some back story about these kexts? where are they from and what has been changed?

@tiger511 all changes are in VoodooGPIO.cpp

  • Remove IODelay(25 * nInactiveCommunities); at line 797
  • In interruptOccurredGated block add IODelay(1000); before isInterruptBusy = false; like:
    nInactiveCommunities = (inactive < ncommunities)? inactive : ((UInt32)ncommunities - 1);
    IODelay(1000); // Sleep 1 miliseccond each handler => max 1000 times per sec => CPU reduced
    isInterruptBusy = false;

As changes above we will only have maximum 1000 report per seccond => fixed high CPU due to too many report from I2C

  • Change if (ret != kIOReturnSuccess) to if (ret != kIOReturnSuccess && ret != 3) at line 916 for fix InterruptOccurred:Failed on attemptAction() as I2C is busy

@lvs1974 in https://github.com/acidanthera/VoodooInput/pull/10 :

In interrupt mode, in idle mode I have quite low cpu usage (~0.5% for kernel_task). If I keep one finger touched - I see 33-40% cpu usage (the same for simple movement), If I keep two finger pressed - I see 66% cpu usage. I tried to comment out everything in VoodooI2CHIDDevice::interruptOccured, but it did not help. I have no clue what does actually load CPU.

@alexandred :

This is a bug that I believe I reduced down to either the I2C controller code (so somewhere in VooodooI2CControllerDriver iirc) or VoodooGPIO. We were unable to find the source of the issue inside there though. If you run the standard thread analysis tools in terminal you will see that the controller code is spinning up a ridiculously large amount of threads to deal with it鈥檚 interrupts (and thus a ridiculously large amount of threads on VoodooGPIO). I seem to recall now that devices that have ACPI controllers instead of PCI (so Haswell and Broadwell era machines) didn鈥檛 have this issue so it could indeed be related to using GPIO interrupts and would also explain why the issue doesn鈥檛 exist in polling mode (however I could be misremembering this)
Try having a looking at the threads - maybe you will spot something we missed. It鈥檚 indeed a very annoying and persistent bug and desperately needs fixing at some point.


just finally figure out the root cause.

The major CPU load in GPIO is due to the massive interrupts when there is finger contact.
However, it is caused by VoodooI2CHID, as it wrongly treats the level interrupts by VoodooGPIO as "clock tick" signals to non-blockingly read the I2C report, in the way like dealing with edge interrupts.

Some details

As we know the touchpad uses a level-triggered GPIO IRQ line to notify the host, which is kept asserted (low) when a touch report is ready to read.

image

Since it is level triggered,

Level triggered: as long as the IRQ line is asserted, you get an interrupt request. When you serve the interrupt and return, if the IRQ line is still asserted, you get the interrupt again immediately.

so we need to blockingly read I2C reports in the ISR until the IRQ line gets de-asserted, not to set up a read request and return immediately.

The implementation of IOInterruptEventSource will automatically disable interrupt during handler being called for level interrupt, to block the interrupt as intended, so no need to worry interrupts being stacked.

apple/darwin-xnu/iokit/Kernel/IOInterruptEventSource.cpp#L441-L473

void IOInterruptEventSource::disableInterruptOccurred
    (void */*refcon*/, IOService *prov, int source)
{
    bool trace = (gIOKitTrace & kIOTraceIntEventSource) ? true : false;

    prov->disableInterrupt(source); /* disable the interrupt */

    IOStatisticsInterrupt();
    producerCount++;

    ...

    signalWorkAvailable();

    ...
}


void IOInterruptEventSource::interruptOccurred
    (void *refcon, IOService *prov, int source)
{
    if (autoDisable && prov)
        disableInterruptOccurred(refcon, prov, source);
    else
        normalInterruptOccurred(refcon, prov, source);
}

The problem is, the current ISR handler implementation immediately returns, causing the IRQ line to keep firing massive interrupts.
alexandred/VoodooI2CHID/VoodooI2CHID/VoodooI2CHIDDevice.cpp#L236-L252

void VoodooI2CHIDDevice::interruptOccured(OSObject* owner, IOInterruptEventSource* src, int intCount) {
    if (read_in_progress)
        return;
    if (!awake)
        return;

    read_in_progress = true;


    thread_t new_thread;
    kern_return_t ret = kernel_thread_start(OSMemberFunctionCast(thread_continue_t, this, &VoodooI2CHIDDevice::getInputReport), this, &new_thread);
    if (ret != KERN_SUCCESS) {
        read_in_progress = false;
        IOLog("%s::%s Thread error while attempting to get input report\n", getName(), name);
    } else {
        thread_deallocate(new_thread);
    }
}

Btw, to be honest, raising a thread here is not a good idea, it messes up the sequential logic.
The blocking reading api->readI2C() in VoodooI2CHIDDevice::getInputReport() is gated to run in the workloop's thread, but the raised thread is not gonna be blocked at all, as intended to wait for the report.

Instead, it will be immediately terminated. The whole thing somehow works just because a new coming thread gets the result that is requested by the prior terminated thread! This might be the reason why it keeps spawning so many threads. Check out the lines where the request gets overridden in workloop.

https://github.com/alexandred/VoodooI2C/blob/2c89682d7e207386a84e98d5173c505855e25479/VoodooI2C/VoodooI2C/VoodooI2CController/VoodooI2CControllerDriver.cpp#L138-L155


Solution

A simple workaround to IOSleep a while in VoodooGPIO can help reduce the "clock" rate,
but I think it would be better to handle the level interrupt properly.
Plz check out the commit here and give a try. https://github.com/Goshin/VoodooI2CHID/commit/9d7a966f394e8f3b7bc26efdeebef38bd37a77c8

Hi Goshin,
i've compiled your VoodooI2CHID. I've an Intel i5-7200U but wasn't having a severe CPU problem. My CPU Kernel_task goes from 2% to around 13% when i place a finger on the touchpad even if i don't move it. Now with your latest commit the situation is improved. The CPU Kernel_Task goes from 2% to around 8% and stays there so there has been an improvement!
Let me know if you need me to try something else.

Mattia

@Goshin, I confirm - now it is far better!!!
Here are my results with your modified VoodooI2CHID:

  1. When touch pad is not used 0.3-0.4%
  2. Keep one finger on touch pad (move it or just keep): 7.0-7.2%
  3. Keep two fingers on touch pad (move them or just keep): 11.9-12.1%
  4. Keep three fingers on touch pad (move them or just keep): 16.9-17.2%

It is a big improvement, thank you for you findings!

@lvs1974 , can you upload your compiled version for me to try?
Thanks.

@Mateo1234454545, here it is:
Release.zip

@Goshin , can I ask you for a big favour ?
Could you compile your latest commit https://github.com/Goshin/VoodooI2CHID/commit/9d7a966f394e8f3b7bc26efdeebef38bd37a77c8
into this version , which is the best working for my toucpad so far (wothout the need of loading voodoimput) ?
Thank you in advance.
VoodooI2C.Mar.27.zip

@tmbt78 @lvs1974 It should have a close performance to that of polling mode. The bottleneck is now in the VoodooI2CHID.

@Mateo1234454545 sry I cannot build with the kext products. What's your problem with the latest master?

@Goshin , latest master has a problem with drag and drop (and two fingers scrolling) as described here :https://github.com/alexandred/VoodooI2C/issues/269

The major CPU load in GPIO is due to the massive interrupts when there is finger contact. However, it is caused by VoodooI2CHID, as it wrongly treats the level interrupts by VoodooGPIO as "clock tick" signals to non-blockingly read the I2C report, in the way like dealing with edge interrupts.

@Goshin i have figured it out long ago.I've removed the threading mechanism.Removing the thread didnt help much,but i don't know why they needed it thread in the first place.Also the gated functions are redundant. The "interruptOccured" functions work in the workloop which is already syncronized.However,throughout the code,i saw "Gated" functions being called from "interruptOccured" functions which adds complexity and serve no purpose.
Using the follwoing line reduces CPU load from 40% to 13% for me:

void VoodooGPIO::interruptOccurredGated() {
intel_gpio_irq_mask_unmask(28, true);
intel_gpio_irq_mask_unmask(28, false);

....

Also remove IODelay from "intel_gpio_community_irq_handler" function,it serves no purpose for me except hogging the CPU for no reason.

For me, VoodooGPIO is doing well in its job. There is no need to reduce the interrupts, which might bring the latency issue. The IODelay you mention is from https://github.com/alexandred/VoodooGPIO/commit/ec0b2d97acf0bdcfd41c903040bf5a0ff20f3b52, and can be reverted.
It does alleviate the load by slowing down interrupt rate in VoodooGPIO, but the essential to the problem is to properly handle the level interrupt in the ISR.

Several attemptAction will not have a big impact on performance.

VoodooGPIO is a generic GPIO controller, not just for VoodooI2C touchpad, so it needs to do some maintaining work for the attached IRQ lines.

@Mateo1234454545, here it is:
Release.zip

Any change workinh on High Sierra ?

@Goshin , your commits don;t have the problem I described. All working good on Catalina. Maybe you could push commit on voodooi2c master for fixing https://github.com/alexandred/VoodooI2C/issues/269

@Goshin as usual, please please create a PR and I'll merge it in.

@Goshin, thank you for your work!
I built new kexts from your sources, (VoodooI2CHID I did not replaced, it is with the latest fixes from branch opt-level-interrupt, VoodooInput is also with your latest fix).
Performance is really great (4.2% for 1 finger, 6.1% for 2 fingers, 7.6% for 3 fingers)!

But now I have quite old issue: dragging and text selection is not stable (when I have to take off one finger and put it again in a different point on touch pad, while another finger stay pressed).
I remember - we fixed it in VoodooInput.

It is very strange for me: VoodooI2CHID and VoodooInput were not changed - but I have got this issue again.
Probably I could made a mistake when checked out everything from different branches. Could you upload your current test version, please?
VoodooI2C with VoodooGPIO+Services inside, VoodooI2CHID and VoodooInput?

You might be not using the up-to-date VoodooInput. It's now embedded in VoodooI2C.kext/Contents/PlugIns/. It is a little tricky to rebuild the embedded VoodooInput now, you might need to rerun the bootstrap script.

@Goshin, is it OK to build it from acidanthera master?

Build VoodooInput from master, place the kext file in VoodooI2C/Dependencies/VoodooInput/Debug and VoodooI2C/Dependencies/VoodooInput/Release, and build VoodooI2C.

Archive.zip
Only VoodooI2C.kext and VoodooI2CHID.kext are needed.

@Goshin, sorry to say it, but I've got the same issue described before.
VoodooInput - from acidanthera master and VoodooI2C kexsts from Archive.zip.
Video: https://drive.google.com/file/d/1eUonsSsyK7IAA7THdxTvLL3xrNWs63-B/view?usp=sharing

@lvs1974 weird, try to revert the change in VoodooI2C and see if it helps?

@Goshin, sorry to say it, but I've got the same issue described before.
VoodooInput - from acidanthera master and VoodooI2C kexsts from Archive.zip.
Video: https://drive.google.com/file/d/1eUonsSsyK7IAA7THdxTvLL3xrNWs63-B/view?usp=sharing

Try to revert this commit, https://github.com/alexandred/VoodooI2C/commit/b709d0fb7ed5ce031758d9a4da159e397a01eefd only.

@Goshin, It worked!!!
Thank you!
Just reverted this one commit!

Some performance improvement in GPIO and I2C bus.
Goshin/VoodooGPIO@17c13f0

@lvs1974 @tmbt78 give a try at your convenience. 馃憤

I just tried your Archieve.zip above, And year it improves a lot.CPU usage is near 6% now.Only problem is when a device has touchscreen , it tries to cache touchscreen pin even though it's disabled in bios.So the touchpad doesnt work altogther.I had to remove "pci8086,a368" from Info.plist in VoodooI2C for it to not discover my Touchscreen I2C and now the touchpad works fine.

And by the way,i don't have the selection problem mentioned by lvs1974

Some performance improvement in GPIO and I2C bus.
[Goshin/VoodooGPIO@17c13f0](Goshin/VoodooGPIO@17c13f0
Goshin@b7e869b
@lvs1974 @tmbt78 give a try at your convenience. 馃憤

I just tried your Archieve.zip above, And year it improves a lot.CPU usage is near 6% now.Only problem is when a device has touchscreen , it tries to cache touchscreen pin even though it's disabled in bios.So the touchpad doesnt work altogther.I had to remove "pci8086,a368" from Info.plist in VoodooI2C for it to not discover my Touchscreen I2C and now the touchpad works fine.

Weird, VoodooGPIO should work as before. It only caches pin which is registered for interrupt. Can you provide more details? Are they using the same pin? Could it be caused by other commits?

Timestamp (process)[PID] 2020-04-15 02:41:25.975538+0600 localhost kernel[0]: (VoodooGPIO) VoodooGPIOCannonLakeH::Loading GPIO Data for CannonLake-H 2020-04-15 02:41:25.985178+0600 localhost kernel[0]: (VoodooGPIO) VoodooGPIOCannonLakeH::VoodooGPIO Init! 2020-04-15 02:41:25.985180+0600 localhost kernel[0]: (VoodooGPIO) VoodooGPIOCannonLakeH::VoodooGPIO Initializing Community 0 2020-04-15 02:41:25.985414+0600 localhost kernel[0]: (VoodooGPIO) VoodooGPIOCannonLakeH::VoodooGPIO Initializing Community 1 2020-04-15 02:41:25.985520+0600 localhost kernel[0]: (VoodooGPIO) VoodooGPIOCannonLakeH::VoodooGPIO Initializing Community 2 2020-04-15 02:41:25.985634+0600 localhost kernel[0]: (VoodooGPIO) VoodooGPIOCannonLakeH::VoodooGPIO Initializing Community 3 2020-04-15 02:41:25.985921+0600 localhost kernel[0]: (VoodooGPIO) VoodooGPIOCannonLakeH::GPIO Controller is already awake! Not reinitializing. 2020-04-15 02:41:26.172911+0600 localhost kernel[0]: (VoodooI2C) VoodooI2CPCIController::pci8086,a368 Starting I2C controller 2020-04-15 02:41:26.174427+0600 localhost kernel[0]: (VoodooI2C) VoodooI2CPCIController::pci8086,a368 Set PCI power state D0 2020-04-15 02:41:26.176008+0600 localhost kernel[0]: (VoodooI2C) VoodooI2CPCIController::pci8086,a368 Publishing nub 2020-04-15 02:41:26.178763+0600 localhost kernel[0]: (VoodooI2C) VoodooI2CPCIController::pci8086,a369 Starting I2C controller 2020-04-15 02:41:26.188282+0600 localhost kernel[0]: (VoodooI2C) VoodooI2CControllerDriver::pci8086,a368 Probing controller 2020-04-15 02:41:26.188710+0600 localhost kernel[0]: (VoodooI2C) VoodooI2CControllerDriver::pci8086,a368 Found valid Synopsys component, continuing with initialisation 2020-04-15 02:41:26.189106+0600 localhost kernel[0]: (VoodooI2C) VoodooI2CControllerDriver::pci8086,a368 Got bus configuration values 2020-04-15 02:41:26.189833+0600 localhost kernel[0]: (VoodooI2C) VoodooI2CControllerDriver::pci8086,a368 Publishing device nubs 2020-04-15 02:41:26.190492+0600 localhost kernel[0]: (VoodooI2C) VoodooI2CControllerDriver::pci8086,a368 Found I2C device: WCOM488F 2020-04-15 02:41:26.191744+0600 localhost kernel[0]: (VoodooI2C) VoodooI2CControllerDriver::VoodooI2CDeviceNub Warning: Incompatible APIC interrupt pin (0x3b > 0x2f) and no GPIO interrupts found; if your chosen satellite implements polling then VoodooI2CDeviceNub will run in polling mode. 2020-04-15 02:41:26.196671+0600 localhost kernel[0]: (VoodooI2C) VoodooI2CPCIController::pci8086,a369 Set PCI power state D0 2020-04-15 02:41:26.197955+0600 localhost kernel[0]: (VoodooI2C) VoodooI2CPCIController::pci8086,a369 Publishing nub 2020-04-15 02:41:26.198872+0600 localhost kernel[0]: (VoodooI2C) VoodooI2CControllerDriver::pci8086,a369 Probing controller 2020-04-15 02:41:26.199471+0600 localhost kernel[0]: (VoodooI2C) VoodooI2CControllerDriver::pci8086,a369 Found valid Synopsys component, continuing with initialisation 2020-04-15 02:41:26.200464+0600 localhost kernel[0]: (VoodooI2C) VoodooI2CControllerDriver::pci8086,a369 Got bus configuration values 2020-04-15 02:41:26.200666+0600 localhost kernel[0]: (VoodooI2C) VoodooI2CControllerDriver::pci8086,a369 Publishing device nubs 2020-04-15 02:41:26.201658+0600 localhost kernel[0]: (VoodooI2C) VoodooI2CControllerDriver::pci8086,a369 Found I2C device: SYNA2393 2020-04-15 02:41:26.204433+0600 localhost kernel[0]: (VoodooI2CHID) VoodooI2CHIDDevice::WCOM488F Warning: Could not get interrupt event source, using polling instead 2020-04-15 02:41:26.207025+0600 localhost kernel[0]: (VoodooI2C) VoodooI2CDeviceNub::Got GPIO Controller! VoodooGPIOCannonLakeH 2020-04-15 02:42:42.466200+0600 localhost kernel[0]: (VoodooI2CHID) VoodooI2CPrecisionTouchpadHIDEventDriver::setProperties USBMouseStopsTrackpad = 0

I'm pretty sure this is related to pin caching commit.I have my touchscreen disabled from BIOS by the way.

And kernel_task is 101% 馃サ

@tiger511

I'm pretty sure this is related to pin caching commit.I have my touchscreen disabled from BIOS by the way.

Try to revert the commit. The commit might ignore interrupts from the unregistered pin. Btw, why could it be able to attach if it is disabled?

To me, it seems to be a DSDT related problem.

(VoodooI2CHID) VoodooI2CHIDDevice::WCOM488F Warning: Could not get interrupt event source, using polling instead

(VoodooI2C) VoodooI2CControllerDriver::VoodooI2CDeviceNub Warning: Incompatible APIC interrupt pin (0x3b > 0x2f) and no GPIO interrupts found; if your chosen satellite implements polling then VoodooI2CDeviceNub will run in polling mode.

To me, it seems to be a DSDT related problem.

It was working all fine for about a year.DSDT is not changed.I'll debug it tomorrow and let you know whats causing the issue.I'm currently having problem building VoodooI2C from source.Kept getting this strange error in xcode: /Users/v/VoodooI2C/build/VoodooI2C/Build/Products/Release/VoodooI2C.kext: code object is not signed at all In subcomponent: /Users/v/VoodooI2C/build/VoodooI2C/Build/Products/Release/VoodooI2C.kext/Contents/PlugIns/VoodooInput.kext

i've VoodooInput-Release in Dependencies/VoodooInput/Release folder and VoodooInput-Debug in Dependencies/VoodooInput/Debug folder .

@Goshin I tried your Archive.zip version and indeed i'm having less CPU usage! Great Job. I even didn't have to revert the commit you suggested (I had the same problem lvs1974 was having with the drag and drop before VoodooInput).

So thanks again !

UPDATE :
After some minutes i realize i've the same problem with the drag and drop as lvs1974 is having. I will try to revert that commit and let you know.

UPDATE 2 :
Yes now it's working again using the lvs1974 release without the b709d0f commit.

@Goshin, It worked!!!
Thank you!
Just reverted this one commit!

Could you upload it here?
I'm getting erros trying to compile it.
Thanks in advance.

@Mateo1234454545:
Release.zip

@lvs1974 , thank you very much.
Working very well on Catalina.

Maybe we can do a HS version?

UPDATE :
After some minutes i realize i've the same problem with the drag and drop as lvs1974 is having. I will try to revert that commit and let you know.

Can you give me a screenshot of your trackpad settings from Preferences? Are you using click or tap with two fingers for secondary click?

@tiger511, here are my settings:
Screenshot 2020-04-15 at 16 34 44
Screenshot 2020-04-15 at 16 34 56
Screenshot 2020-04-15 at 16 35 03

Screen Shot 2020-04-15 at 8 43 43 PM

With this settings i;m not having the drag n drop issue.I didn't have to revert any commit.

@tiger511 It is a device-specific bug that only occurs on some touchpads with a physical button. I will add an option in plist to disable the Force Touch emulation.

@tiger511 @kprinssu @Goshin Can we close this one (Regarding CPU usage, I saw some unrelated discussion...) after the 2.4.1 release? :)

Edit: We should also port HID changes to other satellites, let's keep this one open?

There are some performance improvements for GPIO and I2C controller that have not been merged. They maybe need further testing and review?
https://github.com/alexandred/VoodooI2C/issues/259#issuecomment-613545186

@tiger511 reported an issue might be related, but I am still waiting for the details.

@Goshin which issue are you referring to?
@ben9923 Yes this is mostly solved but i believe it can further be improved.

@tiger511 https://github.com/alexandred/VoodooI2C/issues/259#issuecomment-613659043 Have you figured out the cause of the problem?

Please use 2.4.1 as it contains fixes for high CPU usage, thanks @Goshin. If any issues still persist please create a new issue and let's tackle it there.

Was this page helpful?
0 / 5 - 0 ratings

Related issues

Hunter9888x picture Hunter9888x  路  7Comments

packyan picture packyan  路  6Comments

williambj1 picture williambj1  路  11Comments

onejay09 picture onejay09  路  6Comments

anonymous-writer picture anonymous-writer  路  6Comments