Opencv-python: All development and new releases blocked: migrating from Travis to some other CI provider

[skvark]

Background: https://www.jeffgeerling.com/blog/2020/travis-cis-new-pricing-plan-threw-wrench-my-open-source-works

It's not possible to use Travis to build the Linux and macOS wheels due to changes in Travis pricing.

To do:

• [x] Select a new CI provider
• [x] Check if it is possible to use multibuild with the new CI provider
• [x] Migrate the Travis side of the build toolchain to the new CI provider

Related issues:

https://github.com/skvark/opencv-python/issues/421

[skvark]

Going see this one through before starting any migration process: https://github.com/skvark/opencv-python/issues/366

[skvark]

It seems that Travis is ignoring also much larger projects so it doesn't look like I could use Travis in the future considering that I run this project mostly alone and I would need most likely millions of "Travis credits" per year to keep the builds running.

• https://github.com/numpy/numpy/issues/17824
• https://github.com/scipy/scipy/issues/13109

You can find a lot more similar issues via Github search in other repositories. Github Actions, Azure Pipelines and CircleCI are probably worth taking a look at as alternatives.

[skvark]

Current situation is that Travis removed for some reason all the credits I already had in place (https://github.com/skvark/opencv-python/issues/366#issuecomment-739256591). Due to this, I cannot run any Linux or macOS builds.

I won't be doing any development effort towards this project during this month. There will be no new releases, PRs will not be reviewed etc. If there is some relevant issues / comments, I might answer to them. I'll check in January 2021 if it is possible to continue the project via alternative CI solutions.

[skvark]

It seems we have a final confirmation that Travis has stopped all OSS credit allocations: https://twitter.com/james_hilliard/status/1336081776691843072

[johnthagen]

From my experience the two most popular alternatives are:

• GitHub Actions
  
  
  
  • For public repositories, unlimited credits (For private repositories: This provides 2,000 free minutes per month)
  
  
  • https://docs.github.com/en/free-pro-team@latest/github/setting-up-and-managing-billing-and-payments-on-github/about-billing-for-github-actions
  
  
• CircleCI
  
  
  
  • This provide 2,500 free "credits" per month, but only 1 concurrent job can run at a time
  
  
  • https://circleci.com/pricing/
  
  

Edit: Per https://github.com/skvark/opencv-python/issues/366#issuecomment-740529069 a close reading of the GitHub actions docs does make it seem that public repos are not counted against the limit.

[skvark]

I guess Github Actions is the best option since there are no limits for open source projects. The only issue I see is that ARM builds cannot be done currently, but ARM runners should be available during Q1 2021: https://github.com/github/roadmap/issues/95.

[hannesa2]

but ARM runners should be available during Q1 2021: github/roadmap#95.

It #95 is now closed. But to be honest it's not 100% clear for me. Does it mean the integration to ARM build can be done now ?

[skvark]

It seems that the Github roadmap was modified and ARM support was not implemented (check the roadmap issue edit log) so the ARM issue remains for future builds.

There are already opencv-python wheels for arm64 for the latest release (4.5.1.48) in PyPI (see: https://github.com/opencv/opencv-python/releases/tag/48). Those were built in Travis when that was still possible.

[hannesa2]

There are already opencv-python wheels for arm64 for the latest release (4.5.1.48) in PyPI

Sorry I failed with this https://github.com/apple/tensorflow_macos/issues/67#issuecomment-803062153 (I'm not an experienced python dev)

Is there a better way ?

[skvark]

There are no ARM wheels for the latest macOS devices. arm64 wheels are currently provided only for Linux. Please see: https://github.com/opencv/opencv-python/issues/429

[hannesa2]

I did a look on current .travis.yml Puhh that much variables, eg, what is this ?
- SDIST=0 here

If there would be the build reduced to a single script (eg here), without variable handling in CI script, a jump from one CI-system to an other CI-system would be much more easy ! (A jump to Github action will be not the last one in the future)

But I'm not able to build such a script, I've too less knowledge what's to do on openCV here. -sorry-

[skvark]

The build matrix needs various variables and they need to be defined somewhere. Usually, the place is the CI system config file (SDIST means source distribution which is by default false since it needs to be created just once for each package variant).

Please note that there's no way to build ARM wheels for macOS currently because build infrastructure and tooling lags behind. Similarly, NumPy is a hard dependency for the OpenCV builds, and also they are missing pre-built macOS ARM wheels. Apple Silicon is such a new platform that it takes some time to get the whole software ecosystem in sync with it.

[johnthagen]

It looks like OpenCV 4.5.2 was just released: https://github.com/opencv/opencv/releases/tag/4.5.2

[asmorkalov]

@johnthagen Yes, Alalek pushed tags to the repository and published binaries. @sergregory Is working on Python packages right now.

[sergregory]

470 changes the build infrastructure from Travis & Appveyour to Github Actions. However, with this transition we miss:

• aarch64 packages
• x86 packages for Linux
  Personally, I think the latter ones are of minor importance and we can stop providing them. What about aarch64 - I'm working on the possible options now.

[johnthagen]

For what it's worth (single user), I think x86 (I presume you mean 32-bit vs x86-64) is fine to drop. In my mind having a health CI system is much more valuable. If people really need x86, they can still build from sdist?

For example, you can't even download official Ubuntu 32-bit ISOs any longer.

[johnthagen]

Now that 4.5.2 has been successfully released, should this issue be closed? https://pypi.org/project/opencv-python/4.5.2.52/

[asmorkalov]

Finally migrated to Github Actions.