Mbed-os: strtok_r() does not return valid tokens in recurrence calls

[cederom]

Hello world,
I have a parser function, that sometimes needs to parse itself / recursively with a string of commands. I am using strtok() to split commands separated by spaces. I have noticed that strtok() is non-reentrant function.. but strtok_r() should work with recursion. Unfortunately results are the same for both functions. Inside the "deeper" function strtok_r() does not return the tokens but the whole string (holding the correct context as additionally showed below).

As you can see on the picture - the first run (top function that later calls itself with a different parameters) properly splits space delimeted elements, but the second run (insode recursion) does not split them corretly, full string is returned (indtead only first element) while the context pointer holds the resf of the string to process..

Am I missing anything?

[ciarmcom]

Internal Jira reference: https://jira.arm.com/browse/MBOCUSTRIA-1196

[kjbracey-arm]

strtok_r would be being provided by the C library of whichever toolchain you're using - it's not part of Mbed OS itself.

My first guess looking at that is that you've passed your s100 etc as a string literal to strtok_r. It's in ROM, and the system is just ignoring strtok's attempts to write \0 into it, rather than generating faults. The string must be modifiable. (C++ would diagnose this at compile time by string literals being const char[]; C lets you get away with it because string literals are char[] for pre-C90 compatibility, but writing to them is undefined behaviour).

Mbed OS 5.12 tries to set up the memory map so that ROM writes generate faults, but that isn't operational on all platforms.

[cederom]

Whoah tanks @kjbracey-arm you are the master of the universe!!! Thanks for those amazing hints:

1. MBED uses build host libraries also for C functions!
2. strtok() and strtok_r() cannot work on immutable ROM arrays as they need to put \0 while parsing! This is why first string (variable) did work and second (const char[]) did not. Compiler indeed warned me about that but you know compilers are stupid not me :-) :-) :-)
3. 5.12 upwards will fault on ROM write, which is really nice! Would that call mbed_die()? I have some neat stuff right there :-)

[kjbracey-arm]

Glad that was it. Don't ignore your warnings!

Personally I find it annoying that so many real errors are treated as warnings, so overlookable. Passing const char[] to char * is fundamentally invalid; the compiler is required to issue a "diagnostic". I would much rather that was an error - it shouldn't be the same severity as "unused variable".

Yes, ROM write attempts should end up hitting mbed_die. The target needs to have a Memory Protection Unit, and also to have either standard memory layout or target-custom setup code.

However, some targets with MPU disable the protection because they have non-standard layout and haven't provided custom setup code.

[cederom]

const char[] on the other hand is very useful when you want to store an immutable string or bytes array and just read it out.. this data goes to a ROM/Flash region saving some RAM.. unless anything tries to write into it like strotk() (man page does not mention that crucial information!!).. fault on ROM write will reveal that problem. Thank you again @kjbracey-arm!! :-)

[kjbracey-arm]

man page does not mention that crucial information!!

The fact that the first parameter to strtok is char * not const char *is effectively telling you it's going to write to it.

All standard C++ library functions are const-correct, in that they always take const char * if they don't write to the buffer, so you always get an error when attempting to pass a C++ string literal to a function that can write to it.