I have found two problems tha seem specific to Libertinus when used in LuaLaTeX in math mode :
A minimal set of example is in the enclosed archive
LibertinusPbms.zip. (Sorry for this, the Github Thought police accepts neither pdf nor tex files...).
These problems make the Libertinus family currently unsuitable as a drop-in replacement of Latin Modern.
Libertinus never been intended to be a drop-in replacement of Latin Modern. That being said, indeed we lack calligraphic and several other alphabets. This should be fixed, but I don鈥檛 have time to draw a Libertinus-compatible calligraphic alphabet and need to find a suitable existing one (suitable in design and license).
I don鈥檛 get what is the second problem is, can you please elaborate more?
@EmmanuelCharpentier PDF files _are_ accepted as attachments, but not all browsers (looking at you Firefox) properly identify them in headers on POST requests. Tex files can be attached by adding .txt extensions.
Libertinus never been intended to be a drop-in replacement of Latin Modern.
I wrote a bit too fast. Deciding to use Libertinus in place of Latin Modern entails, of course, a lot of other changes, such as the adaptation of page layout.
What I meant is that you also have to change the font specification in the text's preamble : \setmathfont{Libertinus Math} is clearly not enough, whereas \setmathfont{Latin Modern Math} is...
One of the points of OTF format is that, by stating \setmathfont{XXX}, you accept the font's author decisions about typography with his/her typographic decisions embodied in the font's tables themselves. In contrast, the "classic" way uses a "support package" possibly wrote by someone else, and possibly inconsistent with the font's design.
For the (usually non-typograph) user of the font, the \setmathfont mechanism at least guarantees the _consistency_ of the typographic decisions. Not every one is DEK...
That being said, indeed we lack calligraphic and several other alphabets
This should be advertised (and the correction of this situation should also...).
I don鈥檛 get what is the second problem is, can you please elaborate more?
In Eq. (8), the variable-sized delimiters chosen by LuaLaTeX for denoting the LHS vector are (about) as "rounded" as those picked up for the RHS, whereas, the delimiters of Eq. (10)'s LHS are much "flatter" than RHS's. The effect is surprising.
HTH,
One of the points of OTF format is that, by stating
\setmathfont{XXX}, you accept the font's author decisions about typography with his/her typographic decisions embodied in the font's tables themselves.
The ultimate control on the typography is always to the typographer (user) despite what many type designers like to believe, because no one size fits them all and one鈥檚 usage can always has different requirement than other鈥檚.
the delimiters of Eq. (10)'s LHS are much "flatter" than RHS's. The effect is surprising.
OK, I get it now. That needs to be fixed.
That being said, indeed we lack calligraphic and several other alphabets. This should be fixed, but I don鈥檛 have time to draw a Libertinus-compatible calligraphic alphabet and need to find a suitable existing one (suitable in design and license).
Out of curiosity, what licenses are suitable? Most of the OpenType math fonts I find (like the TeX Gyre fonts) are under variations of the LaTeX Project Public License, as are most non-OpenType math fonts in the TeX distribution.
I tend to need calligraphic characters a lot, so I've been tinkering with selecting ones that look good with Libertinus Math; if I know which fonts are options, I'm happy to make some comparison documents to make the decision easier.
Libertinus fonts are distributed under Open Font License (OFL), so any fonts to be combined with them will need to be under the same license or a compatible license (which generally means non-copyleft licenses like MIT or ASL).
Okay, so are LPPL-licensed fonts okay? It's not copyleft, and requires only that derived works identify the changes made to them and provide a reference to the original work (see conditions 6 and 10).
In either case, a quick comparison is here; XITS and Asana are the only two OFL contestants:

I might put together something more detailed based on my current work later.
I鈥檓 not a lawyer and I鈥檓 not familiar with LPPL so I can not say if it is compatible with OFL or not. The requirements looks complicated to me and I鈥檓 not sure how one can even comply with them in the context of distributing a font. I鈥檇 rather keep things simple.
@EmmanuelCharpentier This doesn't solve your issue, but I in case you weren't aware of how to bandage over the issue, I thought I might share this.
I don't mind either the XITS Math or the Latin Modern Calligraphic fonts with Libertinus. They aren't perfect, but the italic slant and weight are a fairly decent fit---at least for the few characters I use. (I mostly use L and H.)
To use other fonts for only calligraphic characters, you may specify:
\setmathfont[range={\mathcal, \mathbfcal}]{Latin Modern Math}
which will load the calligraphic in normal and bold from Latin Modern Math.
If you notice spacing issues, keep in mind there is a quirk in the unicode-math package which causes it to use spacings from the last font from which anything was loaded. To fix it---after you have loaded all of your other math fonts, reload libertinus with an empty range:
\setmathfont[range={}]{Libertinus Math} % Reload with empty range for correct spacing
Most helpful comment
@EmmanuelCharpentier This doesn't solve your issue, but I in case you weren't aware of how to bandage over the issue, I thought I might share this.
I don't mind either the XITS Math or the Latin Modern Calligraphic fonts with Libertinus. They aren't perfect, but the italic slant and weight are a fairly decent fit---at least for the few characters I use. (I mostly use L and H.)
To use other fonts for only calligraphic characters, you may specify:
which will load the calligraphic in normal and bold from Latin Modern Math.
If you notice spacing issues, keep in mind there is a quirk in the
unicode-mathpackage which causes it to use spacings from the last font from which anything was loaded. To fix it---after you have loaded all of your other math fonts, reload libertinus with an empty range: