Cocotb: Distribute wheels on PyPI

[ktbarrett]

We can now distribute cocotb as a wheel, except for FLI support (well maybe we can, need to talk to Mentor about that). I suggest we provide the following matrix of wheels at first:

• OS
  
  
  
  • Linux (manylinux1 supports C++11)
  
  
  • Mac OS X
  
  
  • Windows-MinGW
  
  
• Arch
  
  
  
  • x86_64
  
  
• Python version
  
  
  
  • CPython 3.X cp3
  
  

These packages should be built and uploaded in CI on release candidate and final releases. May need to investigate distributing with Github to work around the issue of Pypi not supporting nightlies.

[imphil]

I'm looking into that now as part of the release upload CI. Seems to be not super straight-forward, though.

[ktbarrett]

Yeah, it's not. For manylinux1 distributions (which support C++11), you need to build using RHEL6 image, there are official docker images that are typically used. 'auditwheel' can check manylinux1 support. Windows and Mac wheels are not standardized. Extension modules are built to only work with particular version of python, so we need potentially multiple builds per arch. And of course there is the issue of FLI support.

There is cibuildwheel and python-wheels-manylinux-build which should help.

I was thinking it would be useful to provide wheels for platforms that are more difficult to support (Windows) and/or very common platforms like standard CPython 3.5-3.9 on Linux. And for now leave the other platforms for source distribution only. And I think we can safely support only x86_64 only for now.

[imphil]

Yeah, cibuildwheel should do most of the trick. However, we really need to sort out the FLI stuff. If we provide wheels and users want to use a Mentor simulator, what do they do? If we don't provide a "recompile on first use" option, users would need to re-install without wheels? That doesn't feel very easy to understand from a user's perspective, so I'd argue they'd be better off with no wheels at all.

[ktbarrett]

I know the FLI and Questa headers are not licensed in a way we can distribute, but I'm not sure about how they feel about distributing binaries built from that source. It might be possible to build FLI in private CI.

If we don't provide a "recompile on first use" option, users would need to re-install without wheels?

Users currently have to do that if they install the first time without Questa in the path. You have to force reinstall with the wheel cache off or it will simply reinstall the cached wheel without FLI support again. So we are already suffering from that problem in a way. Our current solution is broken IMO.

Personally, I think the best way forward is to separate out the FLI into an extension (pip install cocotb[fli]) and have the extension __require__ the FLI headers to be picked up. That way no faulty wheel without FLI support can be made, and a built wheel can be reused without FLI being on the path during installation.

[imphil]

Making it a separate extension might do the trick if we cannot reach anybody on the Mentor side. I'll try to do that, but for the 1.5 release all of that feels too risky, so let's continue distributing without wheels for now, and pick this discussion up after we have clarity what we can and cannot do with mti.h

[umarcor]

And I think we can safely support only x86_64 only for now.

GHDL and VUnit work on ARM, either RPi, Zynq (Pynq) or smartphones/tables (Android/Termux). Although not first priority, I think it would be very desirable to have cocotb available on ARM (at least aarch64) targets.

[ktbarrett]

Making wheels and supporting the environment in the first place are different efforts, you might be getting ahead of yourself 😄.