Appcenter: Better support for monorepos

[yoshikischmitz]

Describe the solution you'd like
I would like to be able to control how AppCenter clones my git repo. My app lives inside of a monorepo which is pretty big. It would be nice if AppCenter had a way to configure the git cloning process to use something like a partial fetch.

Describe alternatives you've considered
Alternatives would be changing how our repo works. But I think there are enough monorepo users out there for AppCenter to consider this.

[yoshikischmitz]

Hi, would love to hear from the appcenter team if this is something they're interested in adding!

[patniko]

Definitely interested. Just a matter of getting to it. We do monitor this repo heavily to see where people are heavily requesting certain features.

[alexanderkrum]

Yeah +1

[lhrolim]

+1

[ericyip]

I'm trying to use app center to build react-native.
How can I set the entry file in monorepo build?

[nilofer]

@ericyip you cannot specify the file path for builds today. Noted as something that would be important as part of this feature request. Thanks!

[iamchathu]

We are using monorepo with Lerna and have structured the project with main App with sub packages with shared code and native dependancies. We need a way to specify with package should AppCenter build as main project and use Yarn workspaces or Lerna to install dependancies.

[yoshikischmitz]

As part of this, another issue I noticed is in the package dropdown, the current selected package shows up as just package.json. When you have many package.json files, this is not super helpful. To disambiguate between packages, it would be much more helpful to either show the _path_ of the package file, or perhaps even better(as it shows you in the dropdown selection), or the _name_ of the package(as specified in the name field).

[putuyoga]

as monorepo become mainstream, this feature would benefit a lot of people out there.

[dluc]

Hopefully the team will also fix the way a repository is scanned to detect React Native apps. As described in https://github.com/microsoft/appcenter/issues/928 if a RN app is in a subfolder, AppCenter doesn't detect it. Apparently also the "4 layers" search for package.json is not working as expected (see comments).

[brandonpearcy]

+1 for AppCenter to support monorepo projects. We were using AppCenter to deploy our react-native project, but have had to move away from it recently after switching over to a monorepo w/ yarn workspaces. We miss you AppCenter! 😢

Our monorepo project structure is as follows:

/projectRoot
    /node_modules (yarn installs most modules here)
    /packages
        /app
            /node_modules (appcenter build incorrectly assume RN modules will be here)
        /shared
        /web

[zeevl]

I was able to get this to work with our RN monorepo (using yarn workspaces) by doing the following:

• configure build to use package.json in RN sub project
• configure that RN package.json to nohoist it's packages:

  "workspaces": {
    "nohoist": [
      "**"
    ]
  }

• create a script in that sub project called appcenter-post-clone.sh that does a yarn install in the root project directory
• create an empty yarn.lock in the RN sub project to trick appcenter into using yarn

We also had to install a more recent version of node in our post-clone script so rn & others could install correctly. This is what appcenter-post-clone.sh looks like for us:

#!/usr/bin/env bash

set -ex

brew uninstall node@6
NODE_VERSION="12.13.0"
curl "https://nodejs.org/dist/v${NODE_VERSION}/node-v${NODE_VERSION}.pkg" > "$HOME/Downloads/node-installer.pkg"
sudo installer -store -pkg "$HOME/Downloads/node-installer.pkg" -target "/"

# run yarn twice, ignoring errors from the first one
# https://github.com/yarnpkg/yarn/issues/6988
yarn --cwd ../../ install || true
yarn --cwd ../../ install 

[brandonpearcy]

Thank you! That got me closer, but I'm still seeing errors. I wonder what's different between our configurations.

I added nohoist to packages/app/package.json
Moved react-native.config.js from project root to packages/app
Confirmed that react-native config output all looks reasonable.
Confirmed that react-native run-ios and run-android both build correctly locally.
Added your appcenter-post-clone.sh script to packages/app, and verified that it is running successfully in AppCenter build

For my iOS build, I have AppCenter configured as follows:

• use packages/app/package.json
• XCode version 11.2.1
• Node version: 10.x
• Use legacy build system: off

Here is where the build fails. The build script is looking for index.js in the projects root directory, instead of packages/app/index.js

##[section]Starting: Generate source map
==============================================================================
Task         : Command line
Description  : Run a command line script using Bash on Linux and macOS and cmd.exe on Windows
Version      : 2.151.2
Author       : Microsoft Corporation
Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/command-line
==============================================================================
Generating script.
========================== Starting Command Output ===========================
[command]/bin/bash --noprofile --norc /Users/runner/runners/2.160.1/work/_temp/c6837c87-5b14-443b-9a75-e1e5c8ac0a5b.sh
Found index.js for ReactNative index.
warn The following packages use deprecated "rnpm" config that will stop working from next release:
  - react-native-code-push: https://microsoft.github.io/code-push
  - rn-fetch-blob: https://npmjs.com/package/rn-fetch-blob
Please notify their maintainers about it. You can find more details at https://github.com/react-native-community/cli/blob/master/docs/configuration.md#migration-guide.
warning: the transform cache was reset.
error The resource `/Users/runner/runners/2.160.1/work/1/s/index.js` was not found. Run CLI with --verbose flag for more details.
Error: The resource `/Users/runner/runners/2.160.1/work/1/s/index.js` was not found.
    at fs.realpath.err (/Users/runner/runners/2.160.1/work/1/s/packages/app/node_modules/metro/src/IncrementalBundler.js:157:26)
    at gotStat (fs.js:1600:21)
    at FSReqWrap.oncomplete (fs.js:153:21)
##[error]Bash exited with code '1'.

[agusvazquez]

Hello!

I'm having the same issue as @brandonpearcy
Did you guys find a workarround for this?

Thanks!

[vitalyiegorov]

Hello, also huge + for implementing monorepo support.

[agusvazquez]

Hello guys! Got it working by generating the main.js file manually and then pushing the file.

yarn workspace app-name build:ios|android to generate the bundle file

Remember to add this to the scripts part of package.json

"build:android": "react-native bundle --entry-file=./packages/mobile/index.js --bundle-output=./android/CodePush/index.android.bundle --assets-dest ./android/CodePush/ --dev=false --platform=android",
"build:ios": "react-native bundle --entry-file=./packages/mobile/index.js --bundle-output=./ios/CodePush/main.jsbundle --assets-dest ./ios/CodePush/ --dev=false --platform=ios",

When the folder is created, just do a release

appcenter codepush release -c ./packages/mobile/ios/CodePush -t **VERSION** -a **PROJECT** -d **ENVIRONMENT**

or

appcenter codepush release -c ./packages/mobile/android/CodePush -t **VERSION** -a **PROJECT** -d **ENVIRONMENT**

[vitalyiegorov]

Using RN 0.61.5 tried @zeevl solution, it does not work with following error:

* Where:
Script '/Users/runner/runners/2.163.1/work/1/s/node_modules/@react-native-community/cli-platform-android/native_modules.gradle' line: 206

* What went wrong:
A problem occurred evaluating script.
> React Native CLI failed to determine Android project configuration. This is likely due to misconfiguration. Config output:
  [root:/Users/runner/runners/2.163.1/work/1/s, reactNativePath:/Users/runner/runners/2.163.1/work/1/s/node_modules/react-native, dependencies:[@react-native-community/masked-view:...

I guess the problem is that reactNativePath is resolved to the root node_modules folder

@zeevl Maybe you have a clue?

[agusvazquez]

Hey @vitalyiegorov try generating the bundle js manually and uploading it to code push like I explained on my previus post.

That should work and there is no need for black magic on the project.

[vitalyiegorov]

@agusvazquez Thanks for the solution! I will try to integrate it to speed up our builds.

Your solution is viable but it does not fully fit our development process, as our project is in the MVP stage we continue adding new native modules, thus we need to rebuild our APK and CodePush deployment that you have suggested will only partly solve our needs as we would love to dedicate build process for iOS/Android to Appcenter.

Anyway, we need someone help from @microsoft @patniko team to investigate and improve monorepo support as it is getting more and more popular, maybe some things need to be also fixed inside react-native-cli for running RNCLI inside the subfolders, latest RN 0.61.5 release states that monorepo support has been improved which means that the RN team dedicating its resources for it.

[agusvazquez]

@vitalyiegorov if your project has a lot of native modules, then Code Push makes no sense because you will always need to update the native code, therefore you will need to send another APK / IPA for approval to Google / iTunes.

Unless you want to do quick JS only fixes.

The solution I gave is for just doing a code push changing only javascript code. If you need to change native code of course you will need to recompile everything.

[vitalyiegorov]

@agusvazquez Again thank you very much for your advice, but this issue, in general, is not about publishing a new CodePush version, it is related to Appcenter monorepo support which is currently not working, we should be able to build monorepo projects.

[pontusab]

I guess we need to have a config for --entryFile for react native cli build command on AppCenter. Choosing package.json in the subdirectory of the monorepo is not enough.

[GoMino]

to fix @brandonpearcy issue and trick app-center when using a monorepo, a quick hack is it to create a index.js file at the root of the monorepo and require the index of your react-native project:

require("./packages/mobile");

[Maushundb]

@GoMino I tried that and it seemed to fix the index not found error folks have been describing, but now its telling me error Unable to resolve module lib/graphQl/ApolloClient from packages/mobile/src/App.js: lib/graphQl/ApolloClient could not be found within the project., which is the first non-node_modules import I have in my app.

I have a hunch it's trying to resolve imports relative to the root, rather than the package/mobile directory because of require("./packages/mobile");. Any other way to get appcenter to find the right index?

Setting the build setting to use packages/mobile/package.json and changes build:ios to --entry-file=./packages/mobile/index.js didn't seem to fix it.

[zeevl]

Hey all, just wanted to pipe in -- we're successfully building both iOS and android apps from a monorepo on appcenter, using the normal appcenter build tools. Keys for me were:

1. this comment,

2. Pinning the rn cli to 3.0.0 by adding the following to the root package.json:

  "resolutions": {
    "@react-native-community/cli-platform-android": "3.0.0"
  }

due to https://github.com/microsoft/appcenter/issues/1518 and https://github.com/react-native-community/cli/pull/852

I'm not sure I'll be able to provide any support other than that, as so many of the issues in this thread are not issues I encountered. But, I wanted to post this to let you know, it _is_ possible.

[Maushundb]

@zeevl Did you change your build settings to use the root package.json or the sub-folder package.json?

[zeevl]

@Maushundb Build settings are configured to use the mobile app's subfolder's package.json (packages/mobile/package.json).

We also nohoist all of the RN project's modules:

  // in packages/mobile/package.json

  "workspaces": {
    "nohoist": [
      "**"
    ]
  }

[Maushundb]

Yeah I have the "nohoist": [ "**" ] in root/package.json, which seems to do the same thing as putting it in all your packages.

Last q @zeevl - where are your appcenter-pre-build.sh etc located? Root or package? Even if I set the build settings to use the subfolder, appcenter can't seem to find the scripts unless they're in the root.

[zeevl]

I only have appcenter-post-clone.sh, and that's in packages/mobile. It's definitely running with each build.

[Maushundb]

Ok got it working - for posterity, here's the steps we had to do -

1. Put nohoist: [**] in root/package.json.
2. Change the build settings on your branch to use the sub-folder package.json, then hit save and build. this was the key thing I missed:

When App Center detects build scripts for the first time, or whenever you make changes to the location of scripts or, for iOS projects, where you store CocoaPods, you must click the Save or Save & Build button in the build configuration to apply the changes. When you do this, App Center performs an analysis to index your repository tree and updates the build definition.

1. Put a dummy yarn.lock in the package dir

That's why it wasn't finding my appcenter-pre-whatever scripts when I moved them out of the root. Everything else (the require in index, changing the build:ios, the whole appcenter-post-clonescript etc) was unnecessary in our case.

[GoMino]

The problem for me is: since we migrated our react-native repo to a monorepo, we are not able to save or save and build anymore because we cannot select the buildVariant for Android or the build scheme for iOS. The dropdown selector is empty in both cases. How are you able to hit save and build @Maushundb

[Maushundb]

Idk how your appcenter is setup, but we have a master branches which all PR's are tested against, as well as feature branches. Hitting save and build worked on the feature branch since it had a new commit pushed with the new package structure. On master though, we had to merge the PR first (with failed tests) to master, then once the commit was pushed were we able to change the build settings on master.

[yannickvidal]

@GoMino I had the same issue and it seems like that App Center doesn't like to build.gradle too far down, even though the docs say otherwise. I solved the problem by moving by RN application from ./packages/mobile to ./mobile and modifying ./package.json accordingly

[fesaza]

Hey All, I got it worked for lerna monorepo

My folder structure

/packages
    /common
    /web
    /mobile

My mobile/package.json has something like:

   ...
  "dependencies": {
     "@codenull/common": "0.1.1", //my common package
  ....

I've added a pot-clone script with the following (Basically it installs dependencies first with lerna)

#!/usr/bin/env bash

# move to root folder
cd .. && cd ..

# Install dependencies using lerna
npm run bootstrap

# move to mobile and fix local dependencies (this is a custom js fn)
cd packages/mobile && node appCenterPostClone.js

# run cocoapods
cd ios && pod install

Without appCenterPostClone.js appcenter fails since npm i command during build process is not able to "download" my @codenull/common package.

The solution implemented in my appcenterPostClone file was just change my mobile/package.json. Change "@codenull/common": "0.1.1" to "@codenull/common": "file":"../common"

This is the code for appCenterPostClone.js in case this works for you

const fs = require('fs');

const appCenterPostClone = () => {
  console.log('running');
  fs.readFile('package.json', 'utf8', function(err, data) {
    if (err) {
      return console.log(err);
    }
    const result = data.replace(
      '"@codenull/common": "^0.1.1"',
      '"@codenull/common": "file:../common"',
    );

    fs.writeFile('package.json', result, 'utf8', function(errWritting) {
      if (errWritting) return console.log(errWritting);
    });
  });
};

appCenterPostClone();

[purplecandle]

I made monorepo using Yarn workspaces. My folder structure:

/packages
    /common -- api's, models, etc. 
    /web -- React web app
    /native -- React Native mob apps

I encountered a few problems. This is how I've fixed them.

1. "We couldn’t find any Android/iOS projects in your branch" - silly me, I forgot to add the Pods folder in gitignore. Pods folder contains over 5000 files. For some reason, Appcenter couldn't handle my repo with the Pods folder.
2. Added postclone script. I think it is self-describing (thanks fesaza). Don't forget to hit Save and Build button when you add any build script for first time (thanks Brandon)

#!/usr/bin/env bash

# move to the root folder
cd .. && cd ..

# install dependencies
yarn

# compile js files in common folder
cd packages/common && yarn build

# move to native folder and fix local dependencies
cd .. && cd native && node scripts/appcenter-postclone.js

# install pods
cd ios && pod install

[younes200]

@nilofer any update on this issue ? monorepo is widely used and hacky script isn’t a long terme devops solutions.

[nilofer]

@elamalani to provide an update

[younes200]

Any update on this issue @elamalani please ?

Problem is due to this line which look for react-native-cli on local folder : https://github.com/microsoft/appcenter-cli/blob/v1.1.9/src/commands/codepush/lib/react-native-utils.ts#L212

@jokester This could just run @react-native-community/cli in the current directory instead of calling node_modules/react-native/local-cli/cli.js which run react-native/local-cli/cli.js anyway.

[jacob-israel-turner]

@elamalani Anything I can do to help with this? We use AppCenter and CodePush extensively - we are presently moving to use a mono repo and are running into issues building in AppCenter. I'm currently trying to hack something together, but it would be comforting to know if the AppCenter team is planning to improve the experience around mono repos.

[cheunjm]

@elamalani Would love to get some updates on this.
As @younes200 suggests, we just need a way to call react-native-cli from the root of the folder rather than from the local one

[younes200]

The workaround I found is manually coping the cli.js from root to local folder using the appcenter-pre-build.sh.
It's ugly but it works. Still waiting a response from Appcenter team : @elamalani @nileliao

  mkdir -p packages/mobile/node_modules/react-native/local-cli/
  cp node_modules/react-native/local-cli/cli.js packages/mobile/node_modules/react-native/local-cli/cli.js 

[elamalani]

Hey everyone, sorry for my late reply on this thread. App Center team has decided to prioritize improvements in reliability and performance for the service through mid-2021. This is mainly because we have accumulated technical debt in the product that needs to be addressed now. Adding new features in the product will be significantly reduced and unfortunately, we don't plan to support monorepos at the moment.

[jacob-israel-turner]

Bummer - sad to hear @elamalani. Will you let us know next year if this makes it into the roadmap?

@younes200 thanks for that pre-build script - it looks like that's getting me across the finish line to get this working in our monorepo.

@elamalani would it be possible to at least provide a small document or something around getting AppCenter working for monorepos? There have been some great suggestions in this issue. Even referencing this issue for those looking through the official docs would be helpful.

[edmofro]

Building off the very helpful answer by @fesaza, here's a tweaked script for anyone who needs to handle multiple local dependencies (note that all of our local deps are prefixed @tupaia/)

const fs = require('fs');

const fixLocalDepsForAppcenter = () => {
  console.log(
    'Fixing local dependencies for appcenter (see https://github.com/microsoft/appcenter/issues/278)',
  );
  fs.readFile('package.json', 'utf8', (err, data) => {
    if (err) {
      console.log(err);
      return;
    }

    const result = data.replace(/"@tupaia\/([^"]*)": "[^"]*"/g, '"@tupaia/$1": "file:../$1"');

    fs.writeFile('package.json', result, 'utf8', writeError => {
      if (writeError) {
        console.log(writeError);
      }
    });
  });
};

fixLocalDepsForAppcenter();

[wmaca]

It seems people are confusing the problems RN already has on monorepos (such as hoisting) with what App Center should do.

The main problem with App Center on our usage is that it checks for yarn.lock inside the app's package folder (which was configured on the Build Configuration UI).

As it cannot be found, it uses npm install and everything goes south.

But that is not the only problem. As it is being called inside your app's directory (e.g. packages/app), it may cause problems according to your setup.

The approach I used requires using Yarn2, and is comprised of two parts:

Part 1 - Install on the root and tricking App Center to use yarn

appcenter-post-clone.sh

#!/usr/bin/env bash
echo "Running post-clone script..."

cd .. && cd ..
yarn install 

cd packages/app

# Trick App Center into using yarn
touch yarn.lock

Part 2 - Avoid using the fake yarn.lock and unsupported arguments from App Center

When App Center runs its npm/yarn install script, now that we have a fake yarn.lock inside in packages/app, it will use yarn.

As we are using Yarn2, it will look for the .yarn/releases/yarn-berry.cjs when calling yarn on the command line.

This call must be hijacked, as App Center is mistakenly using --list and --network-timeout=600000 which are not supported by Yarn2.

Also, we created and empty yarn.lock, which yarn will see as a problem. We need to remove it before yarn actually tries to install anything.

Create a yarn-berry.cjs versions that calls the _yearn-berry.cjs (which is the original file that was renamed):

#!/usr/bin/env node
const fs = require("fs");

const isRunningOnAppCenter = process.env["APPCENTER_BUILD_ID"] !== undefined;

if (process.cwd().includes("/packages/app") && isRunningOnAppCenter) {
    console.log("Removing `yarn.lock` added to trick App center...");

    try {
        fs.unlinkSync("yarn.lock");
    } catch (err) {
        if (err.code !== "ENOENT") {
            throw err;
        }

        console.log(`File (${err.path}) was already removed.`);
    }
}

if (process.argv[2] === "list") {
    console.log("List not supported");
    process.exit(0);
}

if (process.argv[2] === "install" && process.argv[3] === "--network-timeout=600000") {
    console.log("Popping network timeout yarn flag since its not supported");
    process.argv.pop();
}

module.exports = require("./_yarn-berry.cjs");

This second part comes from here: https://github.com/microsoft/appcenter/issues/2134#issuecomment-790823974

[FranFara]

Any update on this issue @elamalani ?