Mathjs: Custom user literals support

[ovk]

It appears like MathJS does not support adding custom literals. Neither it supports built-in literal for hexadecimal numbers.

In some applications, it would be extremely useful to have hexadecimal (i.e. 0xDEADBEEF) or binary (i.e. 0b11011100) literals. Or, more generically, to be able to register custom literal handler (similarly to how it's done it in C++11 and above). Since MathJS already has bit manipulation operators, this would allow to easily work with bitmasks.

[josdejong]

Thanks for the suggestion. That would be a great addition indeed.

[Nekomajin42]

How much work would it be to handle binary and hexadecimal numbers in the existing functions?

[josdejong]

There are two different approaches possible:

1 Use regular numbers and/or BigNumbers internally (like the JavaScript console)

• Support inputs like 0xFD11 in the expression parser, turning this input into a number or BigNumber.
• Create functions convert numbers to hex/oct/bin notation, like math.hex and math.bin.

2 Create new data types like Hex, Bin, Oct.

• Implement support for these data types in the expression parser: 0xFD11 will create a Hex.
• Implement support for the new data types in all functions. Lot of work
• Create conversion functions from numbers to these new data types.
• Output of Hex, Bin, Oct is easy via the toString methods of these classes.

For example an expression like 0xFD11 >> 2 will return 16196 with solution 1, and will return 3F44 with solution 2 (it remembers that it was a hexadecimal value). I think the ideal solution is solution 2, but this is more work than solution 1. It's not _hard_ but its more work extending all functions and writing unit tests for it.

[firepick1]

I'd prefer to separate formatting concerns from actual values. Specialized objects such as Hex introduce headaches such as "how do you define equality between Hex("0x10") and 16?" In addition, the formatting of a number is highly contextual. Consider: 1000, "One Thousand", "1,000.00", "1e3", etc. In general, it might be easier to simply introduce a HexFormatter, and OctalFormatter, etc. Such formatters would bear the responsibility of converting to/from doubles, rationals and BigNumbers and various string representations.

[ovk]

I agree with @firepick1.

While the 2nd approach is somewhat more user friendly, with everything working out of the box, I think conceptually hex/bin/oct should be just literals, and not types. Of course it would be nice to write something like 0xA0 | 0x3 and get 0xA3 as a result, but that will introduce a lot of ambiguity in more complex expressions. For example, what should be the result of (0xF123 | 0b1010) << 8? Should it be hexadecimal number, or binary, or decimal?

I think having dedicated formatting functions is not a big deal. If you want to get a hex number as a result in the example above, just write something like hex((0xF123 | 0b1010) < 8)

[firepick1]

Note that 0xF123 << N introduces a new issue. Word size. Those of us dealing with bits also have to deal with those pesky nuances such as 32-bitness vs. 64-bitness vs. 8-bitness. Perhaps the Hex/Oct/Bin use cases might play happily in a MachineWord and MachineWordFormatter world.

[ovk]

In general, there's a lot of nuances if you think about machine word size, signedness, endianness, overflows (especially signed overflows), type casting etc. But I think 4-byte unsigned numbers is a reasonable default for MathJS to cover probably 90% of the cases when you simply need basic operations on bitmasks. I doubt someone would want to do something like fast inverse square root in MathJS.

[josdejong]

Thanks guys, I think you're right and we shouldn't overcomplicate things. Good argument to keep formatting and literals separated from calculation. We indeed will have to introduce a configuration option for word size too, I hadn't thought of that yet. That shouldn't be a big deal though.

Going with option 1 is way simpler. Let's go for it :)

Anyone interested in picking this up?

[hyzhak]

ok, I have tried:

math.eval('255 == 0x0000ff')

and got:

Uncaught Error: Undefined symbol x0000ff

it seems this feature is not supported yet. Am I correct? I'm looking for support of hexadecimal because I'm getting a raw expression from a user and it could have hexadecimal numbers. So support of hexadecimal would be very helpful :)

[ovk]

As far as I know, this issue is not addressed yet.

For my project that is based on MathJS I just used some pre-processing to "support" hex and binary literals. It is ugly workaround, but it works: example

[hyzhak]

@ovk big thanks for your support!

[clnhlzmn]

I would love to take a crack at option 1:

1 Use regular numbers and/or BigNumbers internally (like the JavaScript console)

* Support inputs like `0xFD11` in the expression parser, turning this input into a number or BigNumber.

* Create functions convert numbers to hex/oct/bin notation, like `math.hex` and `math.bin`.

Does anyone have any tips on good places to start? I'm guessing the function getToken in parse.js is where this would have to begin?

[josdejong]

Thanks Colin 👍 . You should indeed familiarize yourself with the code in parse.js. It's probably a combination of extending getToken and adding a new parse step (similar to parseNumber and parseDoubleQuotesString etc).

[clnhlzmn]

Thanks Colin 👍 . You should indeed familiarize yourself with the code in parse.js. It's probably a combination of extending getToken and adding a new parse step (similar to parseNumber and parseDoubleQuotesString etc).

@josdejong what do you think about this?

[josdejong]

For reference, see #1968

[josdejong]

Published now in v7.3.0 🎉