Rescript-compiler: Pinned packages support

[BlueHotDog]

https://github.com/BlueHotDog/reason_monorepo

Monorepo support via common tooling(specifically in the above repo Lerna) is broken.
Monorepo support for JS projects is usually being done by using symlinks to support cross-project dependencies - That's the most basic and common workflow.
Another possible monorepo approach is by using Yarn workspaces, which utilizes symlinking even more by hoisting all dependencies to the "root".

Currently, bucklescript is unable to handle any kind of monorepo setup, which, after trying to bang my head, is impossible to solve and leads to a very weird workflow for any ambitious, project(e.g a modern SPA + Nodejs server in Reason).

The example project above illustrates all the problems, but just to summarize what i've encountered:

• Bucklescript randomly spits "duplicate dependency" warning, when two packages depend on the same package. e.g: package A depends on ReFetch(v1 for example) and package B depends on ReFetch too(same version), and package B depends on package A - will result in this warning, which is incorrect.
• Bucklescript will randomly spit an error that bsconfig.json is broken(it's not) when trying to compile package B.
• Bucklescript will randomly spit a: incorrect assumption about interface(same case as the first scenario)
• Watch is just broken completely for the above reasons
• When compiling to Es6 even more errors happen(paths etc) :(

Links to previous discussions that died out and might be related:
https://github.com/BuckleScript/bucklescript/issues/3521
https://github.com/BuckleScript/bucklescript/pull/4047

As there's no roadmap for reasonml. any type of communication regarding stale issue would really help.

[jchavarri]

Another issue that makes the monorepo experience challenging is #4084. An example use case that we've seen is the generation of .ml and .mli files from .atd files. Because there are BuckleScript libraries in the monorepo that depend on .atd files, and on have dependents as well, we have to run atdgen "off-bsb" with Dune, but this means that bsb doesn't know anything about how these targets are generated, so one has to run scripts for every change to the source .atd files.

[TheSpyder]

Other references to monorepo requests, so we can centralise the discussion

1950

2750

3528

[TheSpyder]

I have forked the monorepo @BlueHotDog posted and rewritten it to show the style of monorepo I would like to use. I've written up all the things this is designed to achieve, and why it doesn't currently work, in the readme:
https://github.com/TheSpyder/reason_monorepo

This structure holds a lot of potential, I very nearly used it for my production project, but the watch-rebuild process is _just_ slow enough to be annoying and the lack of compiler warnings really killed it for us.

[BlueHotDog]

@TheSpyder that's awesome! thank you(hopefully BS gods will hear this)
One requirement that you didnt list which is needed for me is the ability to run each project separately:
i'm developing a chrome extension + front-end and need to be able to run bsb+webpack + etc from two different packages side by side.

The way i'm doing it now, e.g by having a single top level bs-config makes this impossible(you have to run a single build, so when working on the flow i've to:

• work on the extension. close it.
• open the front-end. and vice-versa
  )
  If BuckleScript emphasis is performance, i would argue this isnt performant way to write code :)

[TheSpyder]

@BlueHotDog you can still run each project separately if you want, even in my example 😁

Little-known fact about npm module resolution, it also supports searching for a _parent_ package.json and node_modules folder. This is what makes yarn workspaces work so well.

reason_monorepo spyder$ yarn
<stuff here>
reason_monorepo spyder$ yarn build
<build output>
reason_monorepo spyder$ cd packages/a
a spyder$ bsb
[18/18] Building src/FetchedDogPictures/FetchedDogPictures-A.cmj

packages/b also works, but that's where I have the warning-as-error configured.

In fact, my setup might work for you _today_ if you never use a top-level bsb.

[harigopal]

I actually work on a monorepo that uses Lerna + Yarn workspaces - Pupilfirst.

It's not a _traditional_ workspaces-based monorepo, in that there is a top-level application, with other packages _inside_ it; they're not all side-by-side like in the example @TheSpyder created.

And it works... most of the time.

Occasionally, the bsb compilation will start failing, complaining about the presence of duplicate .re files (files with the same name). At that point, one of the packages (app/javascript/packages/pf-icon) will have a lib/ocaml folder inside it that contains dupes of some Reason files. Deleting that folder gets the compilation going again... until seemingly at random, the lib/ocaml folder shows up again.

It's a bit annoying, but it happens rarely enough that it doesn't get in the way of work, and it was the only way I could set this up as a monorepo.

I haven't a clue _why_ this happens - I haven't been able to replicate it reliably, but it probably has something to do with all packages (including _root_) having their own bsconfig.json files - and two of the packages depend on another -pf-icon... and it's always pf-icon that gets the weird lib/ocaml folder with the dupe files inside it.

[TheSpyder]

@harigopal your top-level bsconfig.json also loads all the packages as source, which is the technique I ended up using in production, unfortunately that means the packages can't leverage namespace: true.

My guess about your lib/ocaml problem is the language server occasionally realises there is another bsconfig.json and compiles that folder.

[BlueHotDog]

Seems that on BS 8 the situation is better and now i can work with Yarn Workspaces.(was this fixed? couldnt see anything in any communications nor was it on the roadmap.. but.. yeah)

[TheSpyder]

I'm not sure what changed, but I don't believe it has been fixed. I'm still struggling along without access to namespace:true.

If you don't want to keep this open, I'll open a new one 🤔

[BlueHotDog]

Yup, my mistake, wasn't fixed.
@bobzhang is this still planned for the next releases as documented in the roadmap?

[BlueHotDog]

@bobzhang any update on this? is this planning to be getting some attention? unfortunately if this is a no-go i can't use Reasonml and need to move code back to typescript i would just love to know before i make any drastic changes.

[bobzhang]

Hi, sorry for your bad experience.
It’s on our roadmap for this half so we will have a look . But I am not
sure if we can deliver it in a timely manner to match your expectation.

Danni Friedland [e-mail]于2020年8月10日 周一下午9:55写道：

@bobzhang https://github.com/bobzhang any update on this? is this
planning to be getting some attention? unfortunately if this is a no-go i
can't use Reasonml and need to move code back to typescript i would just
love to know before i make any drastic changes.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/BuckleScript/bucklescript/issues/4361#issuecomment-671368746,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAFWMK5MDLJJJQ2FO35UJRLR7735XANCNFSM4M4ORL5A
.

>

Regards
-- Hongbo Zhang

[BlueHotDog]

Hopefully, if i could get some visibility into what's planned to ship(and a ballpark of when) i can make an educated decision with the team.

[bobzhang]

I will think about it and get back to u later this week

Danni Friedland [e-mail]于2020年8月10日 周一下午10:53写道：

Hopefully, if i could get some visibility into what's planned to ship(and
a ballpark of when) i can make an educated decision with the team.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/BuckleScript/bucklescript/issues/4361#issuecomment-671403912,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAFWMKYXVEBK5PFXOODG5J3SAACWFANCNFSM4M4ORL5A
.

>

Regards
-- Hongbo Zhang

[bobzhang]

Hi Danni, sorry for being late, I checked my calendar that we'll start the
improvement over build system in general in mid September, this is an issue
we will prioritize.

On Mon, Aug 10, 2020 at 10:53 PM Danni Friedland [e-mail]
wrote:

Hopefully, if i could get some visibility into what's planned to ship(and
a ballpark of when) i can make an educated decision with the team.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/BuckleScript/bucklescript/issues/4361#issuecomment-671403912,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAFWMKYXVEBK5PFXOODG5J3SAACWFANCNFSM4M4ORL5A
.

--
Regards
-- Hongbo Zhang

[BlueHotDog]

Hey Bob,
That'll work. please feel free to reach out for input/tests any assistance. happy to spend time on making this as best as we can make it.

Thanks!

[TheSpyder]

I'm also happy to help with thoughts on how changes will impact my attempts to develop in a monorepo

[itayadler]

Hey guys, any news on this issue?
I've been working heavily these past few weeks on a ReScript monorepo, and I experience pains with setting up a consistent watch flow, between 2 packages (app, common), it keeps breaking whenever I change something in the common package, and then I need to run a clean build on both packages in order to get it working

[bobzhang]

Hi, we are working on the improvement of build system currently, will get
back to you if we need more feedback

On Mon, Sep 14, 2020 at 4:05 AM Itay Adler [e-mail] wrote:

Hey guys, any news on this issue?
I've been working heavily these past few weeks on a ReScript monorepo, and
I experience pains with setting up a consistent watch flow, between 2
packages (app, common), it keeps breaking whenever I change something in
the common package, and then I need to run a clean build on both packages
in order to get it working

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/rescript-lang/rescript-compiler/issues/4361#issuecomment-691718354,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAFWMKZU2OMQTI23OLF2KSLSFUQZHANCNFSM4M4ORL5A
.

--
Regards
-- Hongbo Zhang

[itayadler]

sweet, ty 🙇 would love to help anyway I can

[BlueHotDog]

Hey, i see there's a lot of progress on 8.3, and i was eager to try and see if we get good Monorepo support.
Not sure if this release is intended to fix these issue, but it seems like the problems are still there - Is 8.3 intends to address better support for monorepo?

[bobzhang]

Hi, it is not in this release. As you can see, we are working on build
system stuff in this release, we will keep working on build system related
stuff in next release

On Wed, Sep 23, 2020 at 11:25 PM Danni Friedland [e-mail]
wrote:

Hey, i see there's a lot of progress on 8.3, and i was eager to try and
see if we get good Monorepo support.
Not sure if this release is intended to fix these issue, but it seems like
the problems are still there - Is 8.3 intends to address better support for
monorepo?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/rescript-lang/rescript-compiler/issues/4361#issuecomment-697539010,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAFWMK7YSPSGSMURLYQ57ODSHIHPHANCNFSM4M4ORL5A
.

--
Regards
-- Hongbo Zhang

[itayadler]

@bobzhang exciting stuff about 8.3! myself and the company I work for really love ReScript and this issue is affecting us on a daily basis, as we need to workaround it each time it occurs.
If there was a Patreon for the project myself and the company would be the first to signup to help and prioritise this issue somehow.

[TheSpyder]

If there was a Patreon for the project

As far as I understand they're all facebook employees, I don't think they need patreon support 😂

[fakenickels]

Really eager to see this happening! We have a pretty big monorepo here and we suffer in a daily basis with the problems pointed above. If there is any way we can help testing you can count with us @bobzhang. Thanks for your effort

[Freddy03h]

I'm transitioning to a monorepo too, to share code between the react-native and the react-native-web apps essentially.

For now I launch bsb watch in each packages I needed, with the same package-specs (module: es6, in-source: true), It work while working on implementation detail (example : doing styles in packages/design-system while working on the package/react-native-app), but doesn't if I change parameters of a componentd for example.

[fakenickels]

That's exactly it @Freddy03h. Also, even in a setup like this the problem seems to be the same https://github.com/fhammerschmidt/bucklescript-monorepo

[BlueHotDog]

@bobzhang @Hongbo-bb any update on this?
I see a lot of work being done on PPX, is there any progress on this issue, or is PPX a prerequisite?(Or just more important which is fine, i just would love some visibility and expectation setting)

[Freddy03h]

In fact, bsb create the .bsb.js files for all the bs-dependencies in the node_modules (including monorepo packages) at launch time (of course otherwise it wouldn't work). But bsb doesn't watch for changes in the dependencies like webpack or metro do (you can monkey patch a js file in node_modules and changes will be watched).

[bobzhang]

Hi I am starting to look at this issue, shall we have a video chat on the pain points?
Note some pain points are independent which will be delayed later:

• watcher, the built in node.js watcher is not great, I plan to make it user pluggable so they can use their own watcher
• duplicated packages installed, this is the issue comes with package manager, it can not be solved in the near future

[Khady]

How do we get someone from Ahrefs to participate to the video chat?

[TheSpyder]

I'm keen to participate, but I'm in Australia which usually means chats like this are at 3am or similar.

[Coobaha]

Keen to participate too, Helsinki, we do have monorepo setup at Wolt

Also another pain point worth to mention but probably very hard to solve: transitive dependencies for published packages. When there is another RE(S) dependency - we can't just ship JS output of rescript-compiler... We need to rebuild all top level packages on consumer side too, because node_modules are usually shipped as is (re code)

[BlueHotDog]

I'm excited like a groom on his wedding day.
@bobzhang - Maybe some hours and we can vote or something? seems like a lot of interested people(Nice opportunity for the reasonml community to join in)

[fakenickels]

I'm keen to participate as well

[bobzhang]

I live in +8 timezone. 9am - 10pm would work for me. What would be the best time to cover most people? Do you have experience in setting up team meetings. The zoom free versions look good to me (the limits are 100 people, 40 minutes which look fine.)

To make the meeting efficient, it would be nice that if you can share some reproducible build failures.

Note there's one issue that should be solved immediately, the current way of bsb -make-world walking directory in a non-deterministic way (depends on the OS behavior)

[TheSpyder]

You have a lot of cross-over with me in +10, although I would not be available after about 8pm your time.

The example use case I posted earlier in this thread is still my ideal setup:
https://github.com/rescript-lang/rescript-compiler/issues/4361#issuecomment-629780898 and the example repo at https://github.com/TheSpyder/reason_monorepo

For the last 12 months I have been working around the inability to achieve this use case by adding all project source directories to the top-level bsconfig.json, flattening the monorepo into a single project. This has caused significant overheads in namespace management.

[jchavarri]

I noticed that both examples above (1, 2) use Lerna+Yarn, so here's another repro that just uses Yarn workspaces to show the problem with watcher: https://github.com/jchavarri/bsb-smallest-monorepo-example.

At Ahrefs, our workaround as well has been to flatten the monorepo into single bsb "project" (i.e. 1 bsconfig.json). As @TheSpyder mentioned, this solved the issues related to monorepo (watcher, generators not being transitive which broke atdgen, etc) but complexity moved to other places (single namespace leads to a lot of conflicts, tooling for auto-generation of bsconfig.json files for "per-project" local development).

I'm in EU time (CET), but have flexibility to wake up early or stay late if needed. @bobzhang I would suggest that you set a time as the organizer, and people try to accommodate. With folks in Asia, America and Europe, it'll be challenging to find a time that suits everyone :)

Also, would it be interesting to record the call for those that are not able to join? It seems Zoom allows it on free plans.

[bobzhang]

Hi all I made a post here https://forum.rescript-lang.org/t/rescript-office-hours-3/582

[TheSpyder]

I noticed that both examples above (1, 2) use Lerna+Yarn

Oh I forgot I left that in there. I don't use lerna anymore, I don't think we ever leveraged it for the ReasonML project I'm working on before it was removed.

[leeor]

I am using a monorepo based on yarn workspaces.

Given a monorepo with packages A, B, C, when A depends on both B & C, and B depends on C, when working on module A there are two main pains in my experience:

1. You can either build the specific module you're in or all dependencies, but not just dependencies from the local repo. Since the watcher doesn't trigger for dependencies and I did not want to flatten the repo (namespacing), I am left with having to run a bsb -make-world - which leads to the next issue.
2. Very often, changes to C will require a complete rebuild of A because it will be in conflict with interfaces in B.

Ideally, we will have the option to run bsb in the monorepo root and it will build (or watch) everything while maintaining dependencies across modules so that cleaning will not be required very often for small changes.

I can describe these scenarios in more detail if needed.

[ixzzd]

Very often, changes to C will require a complete rebuild of A because it will be in conflict with interfaces in B.

@leeor I had the same issue, but can't make small repro. Maybe you know how to reproduce it? You are talking about staled build issue, right?

[bobzhang]

please chime in on this thread if you have concerns on pinned packages support
https://forum.rescript-lang.org/t/call-for-testing-bs-platform-8-3-3-dev-1-better-make-world-performance/616/5

[BlueHotDog]

Writing here to make sure issue is well documented and easy to find for newcomers:
"I have read all feedback available on the mono-repo support.

Let me explain the original design and why it works in current way.

Note the name is a bit misleading, mono-repo structure is simple and well supported, here users want to support develop multiple packages in separate repos while achieve the similar experience to monorepo.

what does -make-world do

-make-world will first resolves each package recursively and build them independently, this has to be done whether there are changes to the package or not, the changes are not tracked by the build system, that’s one of the reasons why it is slow

The building is on a different mode from toplevel projects (dependency mode):

The warning/errors are mostly ignored, this is designed with the mindset that if your dependency
has a warn-error, it should not get in your way.
Custom rules are not run
The motivation is that your dependency should not impose his build dependency to its consumers.
There’s an installation process after finishing building the package. So for a package a, the user wants to hide some interface from its consumer, he can choose not to install that cmi file. This abstraction is implemented in the installation. Installation is basically copying files which can not be as fast as we hope

What we can improve on current situation?
We can fix the dependency mode as long as we have user input that some packages are pinned packages that it should go to toplevel mode instead of dependency mode.

We could bypass the installation for pinned packages. That means for pinned packages, every interface is exposed (minor semantics difference from unpinned packages)

Watcher

Our built-in watcher is not great due to the limitation of cross platform and minimal dependencies. We want to offer the help to make it easier to integrate 3rd party watchers e.g, watchman

Merge multiple pinned packages into a monorepo?

In theory, the build system could scan multiple pinned packages and treat it as a monorepo, this is a lot of work. However much work we have done here, the semantics would not be the same as multiple packages since the generated js files are path sensitive, even if we make sure all relative paths are still calculated correctly, it is hard to preserve the semantics with bindings, The benefit is of course performance.

Suggestions

If you truly own these packages, I would encourage you adopt a real monorepo setup, then you don’t have these problems at all. For pinned packages, we will discuss how we can improve here.

Bugs

There are some other minor bugs about path resolving which should be fixed later."

[BlueHotDog]

Bug fixes when dependencies changed
There are several obvious bugs when you change your dependencies but will be fixed, hope this alone would make your life easier:

Installation

the installation was not always triggered for some reasons,
This was correct if you don’t change the depency, but seems more people
will change the deps, we are going to fix this.
Scan the directory in a deterministic way && Lock the deps in the bs-depenencies
Previously, the scanning order depends on the OS specific behavior, and re-do a make-world,
will rescan, we will fix this non-deterministic behavior, and lock the bs-dependencies once
it’s scanned, this probably contributes to your inconsistent assumption error
Toplevel mode and dependency mode
The two modes are slightly different, for example, dependency mode does not run generators, it makes sense. However, we will apply toplevel mode to pinned packages too so that it generators are always re-run when the source changes

Watcher issues

We plan to make the build system 3rd party friendly so that you can query which directories to watch and integrate with your favorite watcher.

Other issues

Concurrent build
Suppose both A and B depends on C. -make-world A and -make-world B could potentially trigger a concurrent build on C. We are not sure to allow concurrent build in the same directory, but at least we
will lock the build directory and spits out a meaningful error.
Other high level changes to the build system

Proposed UI Changes: subcommands mode

The current command line flags are a bit messy and some flags are in conflict for different state. We plan to follow other tools, e.g:

npx bsb clean
npx bsb build
npx bsb format
npx bsb install
Build system bugs are hard to reproduce, if you can contribute some reproducible failure examples, that would be much appreciated! – Hongbo

[BlueHotDog]

Is this fixed? what's the status? i see 8.3.3 released without any details.
Some sporadic discussion on the forum, but i can't figure out if this is fixed or what's the status.

[TheSpyder]

Looking at the git history, the 8.3.3 release was cut at least a couple of days ago, two recent PRs mentioning pinned packages were merged today.

[Hongbo-bb]

8.3.3 was announced on the forum with a detailed list of changes.
This issue is a work in progress

On Wed, Nov 18, 2020 at 2:25 AM Danni Friedland [e-mail]
wrote:

Is this fixed? what's the status? i see 8.3.3 released without any details.
Some sporadic discussion on the forum, but i can't figure out if this is
fixed or what's the status.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/rescript-lang/rescript-compiler/issues/4361#issuecomment-729115959,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADLRMSXNMXUOXZMKNBNDKZ3SQK5Y5ANCNFSM4M4ORL5A
.

--
-- Regards, Hongbo

[bobzhang]

Hi, we made a release of 8.4.0-dev.3 which should have most issues mentioned here solved.

Besides some bug fixes, two major fixes here:

First, dependency changes will trigger the rebuild of its descendants.

Second, we introduced a concept of pinned-dependencies, an example usage is here so that packages are classified as three categories:

• main package
  
  
  
  • warnings reported
  
  
  • warn-error respected
  
  
  • build dev dependency
  
  
  • run custom rules
  
  
  • package-specs like es6/commonjs overrides its all dependencies
  
  
• pinned packages
  
  
  
  • warnings reported
  
  
  • warn-error respected
  
  
  • build dev dependency
  
  
  • run custom rules
  
  
• dependency package
  
  
  
  • warnings, warn-error ignored
  
  
  • ignore dev directories
  
  
  • ignore custom generator rules
  
  

Caveats:

• Watcher for pinned packages are not set up yet
• User should always run bsb -make-world in the main package

Tests and bug fixes are welcome, we will mark this issue solved if there is no major drawbacks in the current design

[fakenickels]

Thanks for the update @bobzhang!
I'll be trying it out with ReForm for the next few days.

I just started testing it out here:
https://github.com/Astrocoders/reform/pull/192

First things I noticed:

• The pinned deps changes are only picked up by the compiler if I keep running -make-world. So that means the watcher change is yet to come right?
• If a pinned dep has a source dir with the type dev the compiler gives an error saying it can't find the bs-dev-dependencies of the pinned dep (Jest in this case). Is that an intended behavior?

[TheSpyder]

I stopped using bsb -w recently when I realised it runs via JS fs.watch(). I've been using a little wrapper to watchexec that works quite well, and it sounds like it will be perfect for pinned packages. I'll give the dev release a go soon.

[fakenickels]

I was using redemon but I was looking for dropping the extra dependencies

On Thu, 26 Nov 2020 at 09:32 Andrew Herron [e-mail] wrote:

I stopped using bsb -w recently when I realised it runs via JS fs.watch().
I've been using a little wrapper to watchexec that works quite well, and
it sounds like it will be perfect for pinned packages. I'll give the dev
release a go soon.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/rescript-lang/rescript-compiler/issues/4361#issuecomment-734272021,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAJZJADZOZ5PYHVKPQR6ULLSRZDENANCNFSM4M4ORL5A
.

>

Cheers,
Gabriel Rubens

[BlueHotDog]

I'm having a weird issue while trying to build on netlify..

2:30:48 PM: ────────────────────────────────────────────────────────────────
2:30:48 PM:   1. Build command from Netlify app                             
2:30:48 PM: ────────────────────────────────────────────────────────────────
2:30:48 PM: ​
2:30:48 PM: $ yarn build
2:30:48 PM: yarn run v1.22.10
2:30:48 PM: $ NODE_ENV=production yarn re:cleanbuild && NODE_ENV=production webpack --mode production
2:30:49 PM: $ bsb -make-world -clean-world
2:30:49 PM: bsb: error: loading 'build.ninja': No such file or directory
2:30:49 PM: Failed
2:30:49 PM: Dependency on reason-react
2:30:49 PM: bsb: error: build.ninja:2: expected '=', got ':'
2:30:49 PM: g_finger :=
2:30:49 PM:          ^ near here
2:30:49 PM: Failure: /opt/build/repo/node_modules/bs-platform/linux/ninja.exe
2:30:49 PM: Location: /opt/build/repo/node_modules/reason-react/lib/bs
2:30:49 PM: error Command failed with exit code 2.
2:30:49 PM: info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
2:30:49 PM: error Command failed with exit code 2.
2:30:49 PM: info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
2:30:49 PM: ​
2:30:49 PM: ────────────────────────────────────────────────────────────────
2:30:49 PM:   "build.command" failed                                        
2:30:49 PM: ────────────────────────────────────────────────────────────────
2:30:49 PM: ​
2:30:49 PM:   Error message
2:30:49 PM:   Command failed with exit code 2: yarn build
2:30:49 PM: ​
2:30:49 PM:   Error location
2:30:49 PM:   In Build command from Netlify app:
2:30:49 PM:   yarn build
2:30:49 PM: ​
2:30:49 PM:   Resolved config
2:30:49 PM:   build:
2:30:49 PM:     base: /opt/build/repo/packages/demo
2:30:49 PM:     command: yarn build
2:30:49 PM:     commandOrigin: ui
2:30:49 PM:     environment:
2:30:49 PM:       - REVIEW_ID
2:30:49 PM:       - YARN_VERSION
2:30:49 PM:     publish: /opt/build/repo/packages/demo/build
2:30:49 PM:   redirects:
2:30:49 PM:     - from: /demo/assets/*

[Coobaha]

Here is an example where package with pinned dev dependency is not building https://github.com/Coobaha/bsb-smallest-monorepo-example

try to run yarn build and try to build packages/app on its own (yarn build:app is doing this)

More on this report is here https://forum.rescript-lang.org/t/call-for-testing-bs-platform-8-4-0-dev-2/661/19

├── app                                  # depends on my-lib, dev depends on my-shared-dev-lib, pinned deps: my-shared-dev-lib, my-lib
├── my-lib                               # dev depends on my-shared-dev-lib, pinned deps: my-shared-dev-lib
└── my-shared-dev-lib

calling build via lerna (yarn build which just builds every package in order that is resolved by lerna) works just fine, but if you will try to build packages/app via yarn build or root alias for this yarn build:app compiler will throw:

bsb: [1/6] dev/DevDemo.ast
bsb: [2/6] src/MyLib.ast
bsb: [3/6] dev/DevDemo.d
bsb: [4/6] src/MyLib.d
bsb: [5/6] dev/DevDemo.cmj
FAILED: dev/DevDemo.cmj

  We've found a bug for you!
  /Projects/bsb-smallest-monorepo-example/packages/my-lib/dev/DevDemo.re:1:1-12

  1 │ DevLib.print("Hello, BuckleScript and Reason!");

  The module or file DevLib can't be found.
  - If it's a third-party dependency:
    - Did you list it in bsconfig.json?
    - Did you run `bsb` instead of `bsb -make-world`
      (latter builds third-parties)?
  - Did you include the file's directory in bsconfig.json?

Notice that i am build app but dependency that is missing in bsc is from my-lib

[bobzhang]

@BlueHotDog reading the log, it seems that you are mixing the old version of the bsb with the latest

[bobzhang]

@TheSpyder the watcher is decoupled with the build. You are recommended to use your own if you have a better candidate on your platform

[Hongbo-bb]

another thing is that -clean-world -make-world instead of -make-world -clean-world

On Thu, Nov 26, 2020 at 8:48 PM Danni Friedland [e-mail]
wrote:

I'm having a weird issue while trying to build on netlify..

2:30:48 PM: ────────────────────────────────────────────────────────────────
2:30:48 PM: 1. Build command from Netlify app
2:30:48 PM: ────────────────────────────────────────────────────────────────
2:30:48 PM: ​
2:30:48 PM: $ yarn build
2:30:48 PM: yarn run v1.22.10
2:30:48 PM: $ NODE_ENV=production yarn re:cleanbuild && NODE_ENV=production webpack --mode production
2:30:49 PM: $ bsb -make-world -clean-world
2:30:49 PM: bsb: error: loading 'build.ninja': No such file or directory
2:30:49 PM: Failed
2:30:49 PM: Dependency on reason-react
2:30:49 PM: bsb: error: build.ninja:2: expected '=', got ':'
2:30:49 PM: g_finger :=
2:30:49 PM: ^ near here
2:30:49 PM: Failure: /opt/build/repo/node_modules/bs-platform/linux/ninja.exe
2:30:49 PM: Location: /opt/build/repo/node_modules/reason-react/lib/bs
2:30:49 PM: error Command failed with exit code 2.
2:30:49 PM: info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
2:30:49 PM: error Command failed with exit code 2.
2:30:49 PM: info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
2:30:49 PM: ​
2:30:49 PM: ────────────────────────────────────────────────────────────────
2:30:49 PM: "build.command" failed
2:30:49 PM: ────────────────────────────────────────────────────────────────
2:30:49 PM: ​
2:30:49 PM: Error message
2:30:49 PM: Command failed with exit code 2: yarn build
2:30:49 PM: ​
2:30:49 PM: Error location
2:30:49 PM: In Build command from Netlify app:
2:30:49 PM: yarn build
2:30:49 PM: ​
2:30:49 PM: Resolved config
2:30:49 PM: build:
2:30:49 PM: base: /opt/build/repo/packages/demo
2:30:49 PM: command: yarn build
2:30:49 PM: commandOrigin: ui
2:30:49 PM: environment:
2:30:49 PM: - REVIEW_ID
2:30:49 PM: - YARN_VERSION
2:30:49 PM: publish: /opt/build/repo/packages/demo/build
2:30:49 PM: redirects:
2:30:49 PM: - from: /demo/assets/*

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/rescript-lang/rescript-compiler/issues/4361#issuecomment-734279661,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADLRMSV6OQBZKAYWONEWW2TSRZFCXANCNFSM4M4ORL5A
.

--
-- Regards, Hongbo

[Hongbo-bb]

The thing is that we treat packages in three categories which are handled
slightly differently:

• main-package:
  It decides which module format is chosen, es6 or commonjs?
  It respects warn-error, dev directories are built, picks pinned packages.
  Its generator rules are respected
• pinned-package:
  It is similar to main-package except that it does not decide which module
  format
• normal-dependency-package
  warn-error are ignored, dev tools are not run

For a build, we have to decide which module format to be chosen, if we have
multiple main entry points, there would be no single source of truth.

On Thu, Nov 26, 2020 at 9:29 PM Alexander Ryzhikov [e-mail]
wrote:

Here is an example where package with pinned dev dependency is not
building https://github.com/Coobaha/bsb-smallest-monorepo-example

try to run yarn build and try to build packages/app on its own (yarn
build:app is doing this)

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/rescript-lang/rescript-compiler/issues/4361#issuecomment-734298841,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADLRMSVVBWS7T5S4XC7WQB3SRZJ2HANCNFSM4M4ORL5A
.

--
-- Regards, Hongbo

[Coobaha]

For a build, we have to decide which module format to be chosen, if we have
multiple main entry points, there would be no single source of truth.

Yes this is exactly how I see the main entry bsconfig too! Please check this https://github.com/Coobaha/bsb-smallest-monorepo-example and try to build packages/app as an entry point/root build, it will raise mentioned error for transitive pinned dev dependency..

[Hongbo-bb]

it works for me, see the attachment

On Fri, Nov 27, 2020 at 11:20 AM Alexander Ryzhikov <
[email protected]> wrote:

For a build, we have to decide which module format to be chosen, if we have
multiple main entry points, there would be no single source of truth.

Yes this is exactly how I see the main entry bsconfig too! Please check
this https://github.com/Coobaha/bsb-smallest-monorepo-example and try to
build packages/app as an entry point/root build, it will raise mentioned
error for transitive pinned dev dependency..

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/rescript-lang/rescript-compiler/issues/4361#issuecomment-734625154,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADLRMSWWQVA46WSEPDKOASLSR4LI3ANCNFSM4M4ORL5A
.

--
-- Regards, Hongbo

[Coobaha]

Interesting.. Might be related to the difference between fs, as I am using macos? I will add ubuntu github build

[Coobaha]

@Hongbo-bb https://github.com/Coobaha/bsb-smallest-monorepo-example/runs/1461940833?check_suite_focus=true

but build via lerna passes (job title is wrong) 😨
https://github.com/Coobaha/bsb-smallest-monorepo-example/runs/1461940796?check_suite_focus=true

UPD:
Linux also fails to build https://github.com/Coobaha/bsb-smallest-monorepo-example/runs/1461964839?check_suite_focus=true

[TheSpyder]

@TheSpyder the watcher is decoupled with the build. You are recommended to use your own if you have a better candidate on your platform

This wasn't clear before, but the chat around custom watchers and pinned packages made me look deeper at what bsb -w actually does, and I'm not happy with it anymore 😂

I'll post some details on the forum soon about what switched to, it is cross platform. Using all native tooling means that for single-file incremental recompiles bsb can be done before I've released cmd+s on my keyboard. It's made an already fast feedback loop practically instant.

[TheSpyder]

I have spent the day trial-converting the big project I'm working on to pinned packages (using 8.4.0-dev.4). This was quite an effort, it's been running with a single flat bsconfig.json since September 2019 so we had some accidental circular package dependencies.

I am quite happy so far, I'm able to use public in sources specifications to only export the API modules from my big packages. This will later become namespace:true but I don't want to rename anything just yet.

My feedback:

• Whether by accident or design, it's really nice that dev dependencies are able to disconnect what would normally be a circular dependency. I have a stdlib package, and the tests in that package depend on a jsdom binding package that itself depends on stdlib. This ended up working perfectly.
• The design decision (probably made years ago) to not auto-include transitive dependencies is annoying. My testing framework is based around bs-mocha, and I now have to specify bs-mocha as a dev dependency everywhere (after migrating I have 10 bsconfig.json files, plus one in the root).
• Related to the above, I'm confused about where I need to specify pinned-dependencies. Is it only at the entry point? Or in every bsconfig.json? I've been doing it in every file (which is why it's so annoying) but now I'm wondering if that was necessary.
• Before I started migrating to pinned packages, bsb -make-world was completing more or less instantly. Now, even with zero changes, it seems to rebuilding all pinned packages every time it runs. This might be related to not specifying pinned-packages correctly; with 11 config files it's hard to keep track of things. Certainly if one package is accidentally triggering stdlib to rebuild (and it is the first package built) it would trigger a rebuild of _everything_.

[TheSpyder]

I'm confused about where I need to specify pinned-dependencies. Is it only at the entry point?

Looks like yes. I removed all of the pinned-dependencies from anywhere but the entry point and it behaves the same. This definitely needs to be mentioned in the docs once this feature is released, imo.

[Coobaha]

Related to the above, I'm confused about where I need to specify pinned-dependencies. Is it only at the entry point? Or in every bsconfig.json? I've been doing it in every file (which is why it's so annoying) but now I'm wondering if that was necessary.

Yeah it is annoying, I wrote a script for our monorepo that scans all monorepo packages:

1. Adds all monorepo packages from package.json as pinned dependency
2. Checks bsconfig for missing RE dependencies from package.json and warns
3. Checks package.json for missing deps, dev deps, pinned-deps from bsconfig

Maybe bsb can support this out of the box at some point?

[BlueHotDog]

@BlueHotDog reading the log, it seems that you are mixing the old version of the bsb with the latest

hmm, just triple verified, using the latest, the failure happens on github actions/netlify/vercel so very hard to debug.
But i also added "bs-platform": "^8.4.0-dev.4" to the monorepo resolution, and it still fails :(

[bobzhang]

@fakenickels

If a pinned dep has a source dir with the type dev the compiler gives an error saying it can't find the bs-dev-dependencies of the pinned dep (Jest in this case). Is that an intended behavior?

For pinned package, the type dev directory will get compiled, and bs-dev-dependencies needs to be installed

[bobzhang]

@Coobaha I tried again, it works for me:

bsb-smallest-monorepo-example$yarn build
yarn run v1.19.2
$ lerna run build --concurrency 1 --stream
lerna notice cli v3.22.1
lerna info versioning independent
lerna info Executing command in 3 packages: "yarn run build"
@myorg/my-shared-dev-lib: $ bsb -make-world
@myorg/my-shared-dev-lib: Dependency Finished
@myorg/my-shared-dev-lib: bsb: [1/3] src/DevLib.ast
@myorg/my-shared-dev-lib: bsb: [2/3] src/DevLib.d
@myorg/my-shared-dev-lib: bsb: [3/3] src/DevLib.cmj
@myorg/my-lib: $ bsb -make-world
@myorg/my-lib: Dependency pinned on @myorg/my-shared-dev-lib
@myorg/my-lib: Duplicated package: bs-platform /Users/hongbozhang/git/bsb-smallest-monorepo-example/packages/my-lib/node_modules/bs-platform (chosen) vs /Users/hongbozhang/git/bsb-smallest-monorepo-example/packages/my-lib/node_modules/@myorg/my-shared-dev-lib/node_modules/bs-platform in /Users/hongbozhang/git/bsb-smallest-monorepo-example/packages/my-lib/node_modules/@myorg/my-shared-dev-lib 
@myorg/my-lib: bsb: [1/3] src/DevLib.ast
@myorg/my-lib: bsb: [2/3] src/DevLib.d
@myorg/my-lib: bsb: [3/3] src/DevLib.cmj
@myorg/my-lib: bsb: [1/1] DevLib.cmt
@myorg/my-lib: Dependency Finished
@myorg/my-lib: bsb: [1/6] dev/DevDemo.ast
@myorg/my-lib: bsb: [2/6] src/MyLib.ast
@myorg/my-lib: bsb: [3/6] dev/DevDemo.d
@myorg/my-lib: bsb: [4/6] src/MyLib.d
@myorg/my-lib: bsb: [5/6] dev/DevDemo.cmj
@myorg/my-lib: bsb: [6/6] src/MyLib.cmj
@myorg/app: $ bsb -make-world
@myorg/app: Dependency pinned on @myorg/my-lib
@myorg/app: bsb: [1/6] dev/DevDemo.ast
@myorg/app: bsb: [2/6] src/MyLib.ast
@myorg/app: bsb: [3/6] dev/DevDemo.d
@myorg/app: bsb: [4/6] src/MyLib.d
@myorg/app: bsb: [5/6] dev/DevDemo.cmj
@myorg/app: bsb: [6/6] src/MyLib.cmj
@myorg/app: bsb: [1/2] DevDemo.cmt
@myorg/app: bsb: [2/2] MyLib.cmt
@myorg/app: Dependency pinned on @myorg/my-shared-dev-lib
@myorg/app: Duplicated package: bs-platform /Users/hongbozhang/git/bsb-smallest-monorepo-example/packages/app/node_modules/@myorg/my-lib/node_modules/bs-platform (chosen) vs /Users/hongbozhang/git/bsb-smallest-monorepo-example/packages/app/node_modules/@myorg/my-shared-dev-lib/node_modules/bs-platform in /Users/hongbozhang/git/bsb-smallest-monorepo-example/packages/app/node_modules/@myorg/my-shared-dev-lib 
@myorg/app: bsb: [1/3] src/DevLib.ast
@myorg/app: bsb: [2/3] src/DevLib.d
@myorg/app: bsb: [3/3] src/DevLib.cmj
@myorg/app: bsb: [1/1] DevLib.cmt
@myorg/app: Dependency Finished
@myorg/app: bsb: [1/2] dev/DevDemo-App.cmj
@myorg/app: bsb: [2/2] src/Demo-App.cmj
lerna success run Ran npm script 'build' in 3 packages in 1.7s:
lerna success - @myorg/app
lerna success - @myorg/my-lib
lerna success - @myorg/my-shared-dev-lib
✨  Done in 2.54s.
bsb-smallest-monorepo-example$yarn build:app
yarn run v1.19.2
$ cd packages/app && yarn build
$ bsb -make-world
Dependency pinned on @myorg/my-lib
bsb: [2/2] dev/DevDemo.cmj
bsb: [2/2] DevDemo.cmt
Dependency pinned on @myorg/my-shared-dev-lib
Dependency Finished
bsb: [2/2] src/Demo-App.cmj
✨  Done in 0.65s.
bsb-smallest-monorepo-example$

[bobzhang]

@TheSpyder

Whether by accident or design, it's really nice that dev dependencies are able to disconnect what would normally be a circular dependency.

It is intentional, we want to cut dependency as much as we can.

The design decision (probably made years ago) to not auto-include transitive dependencies is annoying

This is intentional, I don't see any package manager take transitive dependencies into account, I think this is also
modular, users have to list its real dependencies.

Related to the above, I'm confused about where I need to specify pinned-dependencies. Is it only at the entry point?

Yes, I will document it once released.

Before I started migrating to pinned packages, bsb -make-world was completing more or less instantly. Now, even with zero changes, it seems to rebuilding all pinned packages every time it runs

If you want great performance, I would suggest you stay with true monorepo. If you split it into multiple packages, the build system always need check the status of different packages, it takes some time.
If your dependency changes, it will trigger a lot of rebuild

[bobzhang]

@BlueHotDog anyway it could be reproduced? The error message does show that you are using an old version of ninja

[bobzhang]

Hi all, I think most concerns are resolved on master.
Since this issue gets very lengthy I will close this issue, there will be some bugs, you can file it in a new issue.

[Coobaha]

@Coobaha I tried again, it works for me:

bsb-smallest-monorepo-example$yarn build
yarn run v1.19.2
$ lerna run build --concurrency 1 --stream
lerna notice cli v3.22.1
lerna info versioning independent
lerna info Executing command in 3 packages: "yarn run build"
@myorg/my-shared-dev-lib: $ bsb -make-world
@myorg/my-shared-dev-lib: Dependency Finished
@myorg/my-shared-dev-lib: bsb: [1/3] src/DevLib.ast
@myorg/my-shared-dev-lib: bsb: [2/3] src/DevLib.d
@myorg/my-shared-dev-lib: bsb: [3/3] src/DevLib.cmj
@myorg/my-lib: $ bsb -make-world
@myorg/my-lib: Dependency pinned on @myorg/my-shared-dev-lib
@myorg/my-lib: Duplicated package: bs-platform /Users/hongbozhang/git/bsb-smallest-monorepo-example/packages/my-lib/node_modules/bs-platform (chosen) vs /Users/hongbozhang/git/bsb-smallest-monorepo-example/packages/my-lib/node_modules/@myorg/my-shared-dev-lib/node_modules/bs-platform in /Users/hongbozhang/git/bsb-smallest-monorepo-example/packages/my-lib/node_modules/@myorg/my-shared-dev-lib 
@myorg/my-lib: bsb: [1/3] src/DevLib.ast
@myorg/my-lib: bsb: [2/3] src/DevLib.d
@myorg/my-lib: bsb: [3/3] src/DevLib.cmj
@myorg/my-lib: bsb: [1/1] DevLib.cmt
@myorg/my-lib: Dependency Finished
@myorg/my-lib: bsb: [1/6] dev/DevDemo.ast
@myorg/my-lib: bsb: [2/6] src/MyLib.ast
@myorg/my-lib: bsb: [3/6] dev/DevDemo.d
@myorg/my-lib: bsb: [4/6] src/MyLib.d
@myorg/my-lib: bsb: [5/6] dev/DevDemo.cmj
@myorg/my-lib: bsb: [6/6] src/MyLib.cmj
@myorg/app: $ bsb -make-world
@myorg/app: Dependency pinned on @myorg/my-lib
@myorg/app: bsb: [1/6] dev/DevDemo.ast
@myorg/app: bsb: [2/6] src/MyLib.ast
@myorg/app: bsb: [3/6] dev/DevDemo.d
@myorg/app: bsb: [4/6] src/MyLib.d
@myorg/app: bsb: [5/6] dev/DevDemo.cmj
@myorg/app: bsb: [6/6] src/MyLib.cmj
@myorg/app: bsb: [1/2] DevDemo.cmt
@myorg/app: bsb: [2/2] MyLib.cmt
@myorg/app: Dependency pinned on @myorg/my-shared-dev-lib
@myorg/app: Duplicated package: bs-platform /Users/hongbozhang/git/bsb-smallest-monorepo-example/packages/app/node_modules/@myorg/my-lib/node_modules/bs-platform (chosen) vs /Users/hongbozhang/git/bsb-smallest-monorepo-example/packages/app/node_modules/@myorg/my-shared-dev-lib/node_modules/bs-platform in /Users/hongbozhang/git/bsb-smallest-monorepo-example/packages/app/node_modules/@myorg/my-shared-dev-lib 
@myorg/app: bsb: [1/3] src/DevLib.ast
@myorg/app: bsb: [2/3] src/DevLib.d
@myorg/app: bsb: [3/3] src/DevLib.cmj
@myorg/app: bsb: [1/1] DevLib.cmt
@myorg/app: Dependency Finished
@myorg/app: bsb: [1/2] dev/DevDemo-App.cmj
@myorg/app: bsb: [2/2] src/Demo-App.cmj
lerna success run Ran npm script 'build' in 3 packages in 1.7s:
lerna success - @myorg/app
lerna success - @myorg/my-lib
lerna success - @myorg/my-shared-dev-lib
✨  Done in 2.54s.
bsb-smallest-monorepo-example$yarn build:app
yarn run v1.19.2
$ cd packages/app && yarn build
$ bsb -make-world
Dependency pinned on @myorg/my-lib
bsb: [2/2] dev/DevDemo.cmj
bsb: [2/2] DevDemo.cmt
Dependency pinned on @myorg/my-shared-dev-lib
Dependency Finished
bsb: [2/2] src/Demo-App.cmj
✨  Done in 0.65s.
bsb-smallest-monorepo-example$

@bobzhang could you also try yarn clean && yarn build:app? While your local builds are passing please check github builds here https://github.com/Coobaha/bsb-smallest-monorepo-example/actions/runs/386395351

[TheSpyder]

If you want great performance, I would suggest you stay with true monorepo. If you split it into multiple packages, the build system always need check the status of different packages, it takes some time.

The performance isn't a problem if nothing builds. It's the incorrect rebuilding that is a problem.

If your dependency changes, it will trigger a lot of rebuild

Even with no changes it triggers a lot of rebuild. This doesn't happen without pinned packages. I'll see if updating my example repo reproduces the problem and file a new issue.

[ixzzd]

I'm having a weird issue while trying to build on netlify..

@BlueHotDog
I have the same error building app on linux.

@bobzhang Looks like linux ninja binary is outdated, is it possible?

[Hongbo-bb]

Indeed, it seems ninja binary was not updated on Linux..
Egor Chemokhonenko [e-mail]于2020年11月30日 周一下午7:16写道：

I'm having a weird issue while trying to build on netlify..

@BlueHotDog https://github.com/BlueHotDog
I have the same error building app on linux.

@bobzhang https://github.com/bobzhang Looks like linux ninja binary is
outdated, is it possible?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/rescript-lang/rescript-compiler/issues/4361#issuecomment-735722905,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADLRMSXSDJQDFU5KOLI4DDTSSN5H5ANCNFSM4M4ORL5A
.

>

-- Regards, Hongbo

[BlueHotDog]

@Hongbo-bb could you please ping when i can test again?

[Hongbo-bb]

sure thing

On Tue, Dec 1, 2020 at 1:15 AM Danni Friedland [e-mail]
wrote:

@Hongbo-bb https://github.com/Hongbo-bb could you please ping when i
can test again?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/rescript-lang/rescript-compiler/issues/4361#issuecomment-735921688,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADLRMSRWXV5TKZFHAVDJMRDSSPHMNANCNFSM4M4ORL5A
.

--
-- Regards, Hongbo

[Hongbo-bb]

Hi, thanks for reporting this issue, should be fixed here:
https://github.com/rescript-lang/rescript-compiler/pull/4854

On Mon, Nov 30, 2020 at 2:31 PM Alexander Ryzhikov [e-mail]
wrote:

@Coobaha https://github.com/Coobaha I tried again, it works for me:

bsb-smallest-monorepo-example$yarn build

yarn run v1.19.2

$ lerna run build --concurrency 1 --stream

lerna notice cli v3.22.1

lerna info versioning independent

lerna info Executing command in 3 packages: "yarn run build"

@myorg/my-shared-dev-lib: $ bsb -make-world

@myorg/my-shared-dev-lib: Dependency Finished

@myorg/my-shared-dev-lib: bsb: [1/3] src/DevLib.ast

@myorg/my-shared-dev-lib: bsb: [2/3] src/DevLib.d

@myorg/my-shared-dev-lib: bsb: [3/3] src/DevLib.cmj

@myorg/my-lib: $ bsb -make-world

@myorg/my-lib: Dependency pinned on @myorg/my-shared-dev-lib

@myorg/my-lib: Duplicated package: bs-platform /Users/hongbozhang/git/bsb-smallest-monorepo-example/packages/my-lib/node_modules/bs-platform (chosen) vs /Users/hongbozhang/git/bsb-smallest-monorepo-example/packages/my-lib/node_modules/@myorg/my-shared-dev-lib/node_modules/bs-platform in /Users/hongbozhang/git/bsb-smallest-monorepo-example/packages/my-lib/node_modules/@myorg/my-shared-dev-lib

@myorg/my-lib: bsb: [1/3] src/DevLib.ast

@myorg/my-lib: bsb: [2/3] src/DevLib.d

@myorg/my-lib: bsb: [3/3] src/DevLib.cmj

@myorg/my-lib: bsb: [1/1] DevLib.cmt

@myorg/my-lib: Dependency Finished

@myorg/my-lib: bsb: [1/6] dev/DevDemo.ast

@myorg/my-lib: bsb: [2/6] src/MyLib.ast

@myorg/my-lib: bsb: [3/6] dev/DevDemo.d

@myorg/my-lib: bsb: [4/6] src/MyLib.d

@myorg/my-lib: bsb: [5/6] dev/DevDemo.cmj

@myorg/my-lib: bsb: [6/6] src/MyLib.cmj

@myorg/app: $ bsb -make-world

@myorg/app: Dependency pinned on @myorg/my-lib

@myorg/app: bsb: [1/6] dev/DevDemo.ast

@myorg/app: bsb: [2/6] src/MyLib.ast

@myorg/app: bsb: [3/6] dev/DevDemo.d

@myorg/app: bsb: [4/6] src/MyLib.d

@myorg/app: bsb: [5/6] dev/DevDemo.cmj

@myorg/app: bsb: [6/6] src/MyLib.cmj

@myorg/app: bsb: [1/2] DevDemo.cmt

@myorg/app: bsb: [2/2] MyLib.cmt

@myorg/app: Dependency pinned on @myorg/my-shared-dev-lib

@myorg/app: Duplicated package: bs-platform /Users/hongbozhang/git/bsb-smallest-monorepo-example/packages/app/node_modules/@myorg/my-lib/node_modules/bs-platform (chosen) vs /Users/hongbozhang/git/bsb-smallest-monorepo-example/packages/app/node_modules/@myorg/my-shared-dev-lib/node_modules/bs-platform in /Users/hongbozhang/git/bsb-smallest-monorepo-example/packages/app/node_modules/@myorg/my-shared-dev-lib

@myorg/app: bsb: [1/3] src/DevLib.ast

@myorg/app: bsb: [2/3] src/DevLib.d

@myorg/app: bsb: [3/3] src/DevLib.cmj

@myorg/app: bsb: [1/1] DevLib.cmt

@myorg/app: Dependency Finished

@myorg/app: bsb: [1/2] dev/DevDemo-App.cmj

@myorg/app: bsb: [2/2] src/Demo-App.cmj

lerna success run Ran npm script 'build' in 3 packages in 1.7s:

lerna success - @myorg/app

lerna success - @myorg/my-lib

lerna success - @myorg/my-shared-dev-lib

✨ Done in 2.54s.

bsb-smallest-monorepo-example$yarn build:app

yarn run v1.19.2

$ cd packages/app && yarn build

$ bsb -make-world

Dependency pinned on @myorg/my-lib

bsb: [2/2] dev/DevDemo.cmj

bsb: [2/2] DevDemo.cmt

Dependency pinned on @myorg/my-shared-dev-lib

Dependency Finished

bsb: [2/2] src/Demo-App.cmj

✨ Done in 0.65s.

bsb-smallest-monorepo-example$

@bobzhang https://github.com/bobzhang could you also try yarn clean &&
yarn build:app? While your local builds are passing please check github
builds here
https://github.com/Coobaha/bsb-smallest-monorepo-example/actions/runs/386395351

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/rescript-lang/rescript-compiler/issues/4361#issuecomment-735580001,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADLRMSQVKQKMVPLRTQNJXA3SSM32DANCNFSM4M4ORL5A
.

--
-- Regards, Hongbo

[Hongbo-bb]

Hi, it would be helpful if you can make a reproducible repo, I am planning
to cut a release early next week, thanks

On Mon, Nov 30, 2020 at 2:31 PM Andrew Herron [e-mail]
wrote:

If you want great performance, I would suggest you stay with true
monorepo. If you split it into multiple packages, the build system always
need check the status of different packages, it takes some time.

The performance isn't a problem if nothing builds. It's the incorrect
rebuilding that is a problem.

If your dependency changes, it will trigger a lot of rebuild

Even with no changes it triggers a lot of rebuild. This doesn't happen
without pinned packages. I'll see if updating my example repo reproduces
the problem and file a new issue.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/rescript-lang/rescript-compiler/issues/4361#issuecomment-735580027,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADLRMSQPQO3NCLALS6HMEJLSSM32LANCNFSM4M4ORL5A
.

--
-- Regards, Hongbo

[TheSpyder]

Hi, it would be helpful if you can make a reproducible repo, I am planning to cut a release early next week, thanks
…

yes, sorry, I've been busy but I'll give it a go tonight or tomorrow

[TheSpyder]

I couldn't make my example reason monorepo reproduce the issue so I'll try making it more complex. In the meantime I switched back to my main repo and I was wrong about something:

Whether by accident or design, it's really nice that dev dependencies are able to disconnect what would normally be a circular dependency.

It is intentional, we want to cut dependency as much as we can.

I must have put my repository into a weird state to make that compile. The problem is that it's still a circular dependency:

• bsb tries to build both regular and dev sources of library 1 at the same time
• the tests of library 1 depend on library 2
• library 2 depends on library 1

So unless the dev sources can be compiled in a separate pass I'll need to unwind those dependency links. On a clean repo it's giving me a module not found error compiling the tests of library 1, which is a bit misleading but oh well.

[TheSpyder]

Looks like the weird state I was seeing where no changes caused a full rebuild were related to the cycle that wasn't really a cycle. With a clean checkout and no fake cycles it's fine. Sorry about that.

It's still unusable for large monorepo projects due to this thing you said earlier

If your dependency changes, it will trigger a lot of rebuild

But I'm going to log a new issue for that as it is a separate discussion. Pinned packages seem to be working as designed.

[Hongbo-bb]

Hi, I made a release 8.4.0 under dev tag, it should work on
Linux/Windows/Mac

On Tue, Dec 1, 2020 at 1:15 AM Danni Friedland [e-mail]
wrote:

@Hongbo-bb https://github.com/Hongbo-bb could you please ping when i
can test again?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/rescript-lang/rescript-compiler/issues/4361#issuecomment-735921688,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADLRMSRWXV5TKZFHAVDJMRDSSPHMNANCNFSM4M4ORL5A
.

--
-- Regards, Hongbo

[Coobaha]

@Hongbo-bb amazing! it seems to pass, i've noticed that linux dist seems to be incorrectly packed? https://github.com/Coobaha/bsb-smallest-monorepo-example/runs/1505808007?check_suite_focus=true

Build passes on mac and uses correct bsb :)

[BlueHotDog]

Doesnt work :(

$ bsb -clean-world -make-world
--
11:52:11.055 | bsb: error: loading 'build.ninja': No such file or directory
11:52:11.056 | Failed
11:52:11.060 | bs-platform version mismatch Running bsb 8.4.0-dev.1 (/vercel/workpath0/node_modules/bs-platform) vs vendored 8.4.0 (/vercel/workpath0/node_modules/bs-platform)
11:52:11.076 | error Command failed with exit code 2.
11:52:11.076 | info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
11:52:11.096 | error Command failed with exit code 2.
11:52:11.096 | info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
11:52:11.105 | Error: Command "yarn build" exited with 2
11:52:13.238 | Done with "package.json"

[Hongbo-bb]

Indeed, it seems I made a mistake for publishing on linux.
Should be good now -- 8.4.1

On Sun, Dec 6, 2020 at 5:44 PM Alexander Ryzhikov [e-mail]
wrote:

@Hongbo-bb https://github.com/Hongbo-bb amazing! it seems to pass, i've
noticed that linux dist seems to be incorrectly packed?
https://github.com/Coobaha/bsb-smallest-monorepo-example/runs/1505808007?check_suite_focus=true

Build passes on mac and uses correct bsb :)

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/rescript-lang/rescript-compiler/issues/4361#issuecomment-739479013,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADLRMSTIAYKWTL2E23YXVTLSTNHAXANCNFSM4M4ORL5A
.

--
-- Regards, Hongbo

[BlueHotDog]

Just tried again :(
I'm a bit lost between all the versions, but seems to not work still:
Still getting this weird error.

19:51:04.978 | [4/4] Building fresh packages...
-- | --
19:51:16.633 | Done in 112.46s.
19:51:17.028 | yarn run v1.22.4
19:51:17.058 | $ NODE_ENV=production yarn re:cleanbuild && NODE_ENV=production yarn js:build
19:51:17.264 | $ bsb -clean-world -make-world
19:51:17.336 | bsb: error: loading 'build.ninja': No such file or directory
19:51:17.336 | Failed
19:51:17.341 | bs-platform version mismatch Running bsb 8.4.0-dev.1 (/vercel/workpath0/node_modules/bs-platform) vs vendored 8.4.0 (/vercel/workpath0/node_modules/bs-platform)
19:51:17.352 | error Command failed with exit code 2.
19:51:17.352 | info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
19:51:17.369 | error Command failed with exit code 2.
19:51:17.369 | info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
19:51:17.377 | Error: Command "yarn build " exited with 2
19:51:19.526 | Done with "package.json"

[Hongbo-bb]

can you double check that if your bs-platform is indeed 8.4.1?

On Mon, Dec 7, 2020 at 1:55 AM Danni Friedland [e-mail]
wrote:

Just tried again :(
I'm a bit lost between all the versions, but seems to not work still:
Still getting this weird error.

19:51:04.978 | [4/4] Building fresh packages...
-- | --
19:51:16.633 | Done in 112.46s.
19:51:17.028 | yarn run v1.22.4
19:51:17.058 | $ NODE_ENV=production yarn re:cleanbuild && NODE_ENV=production yarn js:build
19:51:17.264 | $ bsb -clean-world -make-world
19:51:17.336 | bsb: error: loading 'build.ninja': No such file or directory
19:51:17.336 | Failed
19:51:17.341 | bs-platform version mismatch Running bsb 8.4.0-dev.1 (/vercel/workpath0/node_modules/bs-platform) vs vendored 8.4.0 (/vercel/workpath0/node_modules/bs-platform)
19:51:17.352 | error Command failed with exit code 2.
19:51:17.352 | info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
19:51:17.369 | error Command failed with exit code 2.
19:51:17.369 | info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
19:51:17.377 | Error: Command "yarn build " exited with 2
19:51:19.526 | Done with "package.json"

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/rescript-lang/rescript-compiler/issues/4361#issuecomment-739538789,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADLRMSWKJDS4VK6N66J27ITSTPASLANCNFSM4M4ORL5A
.

--
-- Regards, Hongbo

[BlueHotDog]

i did, also made sure yarn.lock has 8.4.1.
it fails the same way on chromatic/vercel

[Hongbo-bb]

The log is weird, what's the output of npx bsb -v? it should say 8.4.1
on your platform

On Mon, Dec 7, 2020 at 3:27 PM Danni Friedland [e-mail]
wrote:

i did, also made sure yarn.lock has 8.4.1.
it fails the same way on chromatic/vercel

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/rescript-lang/rescript-compiler/issues/4361#issuecomment-739727366,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADLRMSSNDE3IVXYBSWNT64TSTR7UZANCNFSM4M4ORL5A
.

--
-- Regards, Hongbo

[BlueHotDog]

Yeah, super weird. and it works locally on my OSX. the problem is, i dont have shell access to any of these platforms. pinging you on Discord to maybe give you some other way to look.

[Coobaha]

It seems to be passing on Linux, before it was failing with the same error as you have @BlueHotDog
https://github.com/Coobaha/bsb-smallest-monorepo-example/actions/runs/403993626

Can you reproduce this error with my repo and vercel build?

[BlueHotDog]

It seems to be passing on Linux, before it was failing with the same error as you have @BlueHotDog
https://github.com/Coobaha/bsb-smallest-monorepo-example/actions/runs/403993626

Can you reproduce this error with my repo and vercel build?

Just tried, and seems to work. well. no idea what is happening.. and no idea what this error means :(
The thing is that it also fails on Chromatic and they dont share the same cache/build pipeline

[TheSpyder]

But I'm going to log a new issue for that as it is a separate discussion

I won't have time to log the issue until tomorrow, probably, but I have added pinned packages support to my example monorepo (it's probably a good way to show how pinned packages work, actually) and there's a blurb in the readme about why it isn't _quite_ good enough for large projects.

[Hongbo-bb]

Changing ExampleStyles.re in b causes package c to rebuild even though c does
not use it.
Changing ATest.re - which is a development source file and therefore
cannot have any impact on the exported code - triggers a complete rebuild
of packages b and c.

This could be fixed but I think it still won't match with the expectation
of true monorepo -- when you do change the interface type, it would trigger
the whole project rebuild.
In that case, the benefit of such minor improvement is not that high?

On Tue, Dec 8, 2020 at 4:56 AM Andrew Herron [e-mail]
wrote:

But I'm going to log a new issue for that as it is a separate discussion

I won't have time to log the issue until tomorrow, probably, but I have
added pinned packages support to my example monorepo
https://github.com/TheSpyder/reason_monorepo (it's probably a good way
to show how pinned packages work, actually) and there's a blurb in the
readme about why it isn't quite good enough for large projects.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/rescript-lang/rescript-compiler/issues/4361#issuecomment-740175050,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADLRMSRWWQEVTNDSBNGZHPLSTU6PZANCNFSM4M4ORL5A
.

--
-- Regards, Hongbo

[Hongbo-bb]

@Andrew Herron [e-mail] I made an experimental PR in #4863
to support more fine-grained dependency tracking. This is the best that we
can achieve given we treat the package as a black box.
It works best when you write interface file for your packages

On Tue, Dec 8, 2020 at 11:53 AM Hongbo Zhang [e-mail] wrote:

Changing ExampleStyles.re in b causes package c to rebuild even though c does
not use it.
Changing ATest.re - which is a development source file and therefore
cannot have any impact on the exported code - triggers a complete rebuild
of packages b and c.

This could be fixed but I think it still won't match with the expectation
of true monorepo -- when you do change the interface type, it would trigger
the whole project rebuild.
In that case, the benefit of such minor improvement is not that high?

On Tue, Dec 8, 2020 at 4:56 AM Andrew Herron [e-mail]
wrote:

But I'm going to log a new issue for that as it is a separate discussion

I won't have time to log the issue until tomorrow, probably, but I have
added pinned packages support to my example monorepo
https://github.com/TheSpyder/reason_monorepo (it's probably a good way
to show how pinned packages work, actually) and there's a blurb in the
readme about why it isn't quite good enough for large projects.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/rescript-lang/rescript-compiler/issues/4361#issuecomment-740175050,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADLRMSRWWQEVTNDSBNGZHPLSTU6PZANCNFSM4M4ORL5A
.

--
-- Regards, Hongbo

--
-- Regards, Hongbo

[TheSpyder]

Thanks! I look forward to testing it, I do usually have more interfaces it was just an example

[Hongbo-bb]

Hi @Andrew Herron [e-mail] I published a version
8.4.2-dev.1 for Mac, let me know if it works for you

On Tue, Dec 8, 2020 at 3:13 PM Andrew Herron [e-mail]
wrote:

Thanks! I look forward to testing it, I do usually have more interfaces it
was just an example

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/rescript-lang/rescript-compiler/issues/4361#issuecomment-740430668,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADLRMSVEVMTNIABWORTXDULSTXGZBANCNFSM4M4ORL5A
.

--
-- Regards, Hongbo

[TheSpyder]

I published a version 8.4.2-dev.1 for Mac, let me know if it works for you

That's beautiful, thank you! 😁

So the change means that saving an implementation will no longer trigger dependent rebuilds, but saving an interface still does? I tested by swapping back and forth between pinned packages and a single flat bsconfig.json, then saving one of my .rei files (no changes but it causes the file to be built).

The single namespace is able to avoid dependent modules rebuilding in this scenario when the interface contents remains the same. As a pinned package, that same change causes all dependent modules to rebuild; presumably because the build system is operating in new contexts for dependencies and only checks the file timestamp not whether the interface actually changed.

This means it's still slower to develop on than the single namespace but it's vastly improved from 8.4.1 and I think that tips the balance such that team will overlook the speed difference to get the improved monorepo experience.

[TheSpyder]

I have updated my example monorepo with 8.4.2-dev.1 and added interface files to every pinned dependency. The readme now lists just one example that triggers a forest of rebuilds.

[Hongbo-bb]

Thanks for checking the new release. When the package's interface changes,
it will trigger a rebuild, the current situation is the best we can get in
theory when we treat the package as a black box.

On Thu, Dec 10, 2020 at 3:42 AM Andrew Herron [e-mail]
wrote:

I published a version 8.4.2-dev.1 for Mac, let me know if it works for you

That's beautiful, thank you! 😁

So the change means that saving an implementation will no longer trigger
dependent rebuilds, but saving an interface still does? I tested by
swapping back and forth between pinned packages and a single flat
bsconfig.json, then saving one of my .rei files (no changes but it causes
the file to be built).

The single namespace is able to avoid dependent modules rebuilding in this
scenario when the interface contents remains the same. As a pinned package,
that same change causes all dependent modules to rebuild; presumably
because the build system is operating in new contexts for dependencies and
only checks build date not whether the interface actually changed.

This means it's still slower to develop on than the single namespace but
it's vastly improved from 8.4.1 and I think that tips the balance such that
team will overlook the speed difference to get the improved monorepo
experience.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/rescript-lang/rescript-compiler/issues/4361#issuecomment-742004318,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADLRMSUEYFVLDL2TNLNOT3TST7HI7ANCNFSM4M4ORL5A
.

--
-- Regards, Hongbo

[TheSpyder]

Yeah I can understand that’s the current design. It would be nice if pinned packages weren’t a black box but that would be hard with features like namespace: true (and it can be improved later if you find a way to do it).

[Freddy03h]

Hi! I read the blogpost but I don't understand how it work. I put one of my workspace bs-dependencies in pinned-dependencies in my bsconfig file with updated (8.4.2) bs-platform inside all my packages.
But … the behavior seems the same? Nothing change when I work on a file in the pinned-dep.
I still can launch a other bsb -make-world -w on the other yarn workspace package. It's still work for inner change like style, but still can't for a fonction definition for example.

I read that watcher is an other topic that isn't yet ready I think.

How can I benefit of the advantages of a monorepo with yarn workspace? Working on files in different packages without switching script run, clean and relaunch every time, just like if we work on separate repositories…

I should miss something I don't understand, or maybe it's not yet ready? (because of watcher)

[TheSpyder]

@Freddy03h The test monorepo I used to provide feedback is a good example of how to run with pinned packages. I'll update it to the 8.4 release soon.
https://github.com/TheSpyder/reason_monorepo

[Freddy03h]

@TheSpyder Thank's for the answer, I finally got time to test it.
My monorepo is different because I don't have source code at the root. But anyway, I tested to add a bsconfig at the root and references all packages as bs-dependencies and pinned-dependencies on it. (my difference is that sources target an empty directory).
The compile work fine, but it's still the same probleme: I can't work on the packages files because it's not watched…

pinned-dependencies seems to be a step for the monorepo use case, but for now, without the watcher part, it's impossible to have a normal workflow on a monorepo with rescript.

[TheSpyder]

Watching is the easiest part, I had actually stopped using bsb -w even before pinned packages. I discovered it uses nodejs watch not a native implementation. There are many good native watch tools, I'm using watchexec. Here's a quick example, similar to the one I use in my example repo:

watchexec -w modules -i lib -e re,rei npx bsb

This does lose the timing printout for each compile but I'm planning to add a little shell script to do that later (and then I'll post the whole thing on the forum).

[ryyppy]

@TheSpyder we wanted to create a dedicated doc page just for monorepo support (see related issue)... might be worth adding your findings on a reliable watch setup in userland there as well.

[TheSpyder]

Will do, sorry I thought I would have it done by now but life has been a bit busy.