Let's start a discussion on routines for string handling and manipulation. The thread over at j3-fortran already collected some ideas:
The discussion also mentioned the proposed iso_varying_string module, which was supposed to include some string routines. I found three distinct implementations of this module:
iso_varying_string proposal; the module dates back to 1998)I also found the following Fortran libraries targeting string handling:
sub, gsub, split, join, and conversion on concatenation. WIP thoughIt is likely that several of the tools in the list of popular Fortran projects also contain some tools for working with strings. Given the numerous implementations it seems like this is one of the things where the absence of the standard "... led to everybody re-inventing the wheel and to an unnecessary diversity in the most fundamental classes" to borrow the quote of B. Stroustrup in a retrospective of the C++ language.
For comparison here are some links to descriptions of string handling functions in other programming languages:
Obviously, for now we should not aim to cover the full set of features available in other languages. Since the scope is quite big, it might be useful to break this issue into smaller issues for distinct operations (numeric converions, comparisons, finding the occurence of string in a larger string, joining and splitting, regular expressions).
My suggestion would be to start with some of the easy functions like capitalize, count, endswith, startswith, upper, lower, and the conversion routines from numeric types to strings and vice-versa.
This is a great summary! I feel like most of what we need has already been done in these projects (and others), so mainly we need to just gather it all together. Some important things to decide:
string).s.lower()) or functional (lower(s))?Should we base it on the ISO_VARYING_STRING module? If so, the class is VARYING_STRING and the procedures are functional (lower(s)).
Should we utilize intrinsic function names? like real(string, [kind, [status]]) and have it stop if the conversion fails and no status variable is provided?
functional-fortran implements several functions on strings:
Further, these functions (and their corresponding operators) are compatible with character strings:
complement,empty,head,init,intersection,insert,last,reverse,set,sort,split,tail, andunion.
(Caution: split in functional-fortran is not quite what's been discussed at j3-fortran repo. It merely splits the string in two and returns the first or second part)
Thanks for this initiative and listing the current landscape. I think we definitely want stdlib to have good string support.
(For conversion from real/integer numbers to strings, I implemented a function str to be used like this: https://github.com/certik/fortran-utils/blob/b43bd24cd421509a5bc6d3b9c3eeae8ce856ed88/tests/strings/test_str.f90, implemented here and here, so one can do things like "Number i = " // str(i) // ".".)
@ivan-pi do you want to go ahead and create a table of the basic subroutines and let's brainstorm how they should be named, to be consistent with other languages and/or the above various string implementations if possible. And also if they should be functions or subroutines and what arguments to accept.
@jacobwilliams is right about raising the question how to represent the string. We should start with that.
I would recommend (as usual) to have a lowest level API that operates on the standard Fortran (allocatable where appropriate) character. Then, have a higher level API that operates on a string type, and simply calls the lower level API. Regarding a name, see #26, it seems most people agree that the convention to name derived type is to append _t, so it would be string_t.
That way people can use these low level API routines right away. For example in my codes I do not need to modify any data structures and can start using it. The higher level string_t API can then be used by codes that choose to refactor them, or in new codes. If the syntax is not as nice, some people might opt for the lower level API anyway.
I would vote that the low level API be based on functions, pure and elemental where possible and appropriate. I would stick with the Fortran convention of optional status parameters where there is the possibility of things going wrong, and if one is not provided and something goes wrong it crashes. I have tended to use that convention in any routines that go from a string to some intrinsic like:
if (present(status)) then
read(string, *) result
else
read(string, *, iostat=status) result
end if
Honestly, I thought the ISO_VARYING_STRING standard did a great job of covering all of the intrinsic functions available for character(len=*) variables, and extending IO to work with that type (put, get, put_line). Aside from the strange interface for split I think it's a great starting point.
My plan was to go through the libraries above and create a table of the most commonly available routines in the next days.
I agree we should consider both low-level routines which work directly on strings of type character(len=*) and a high-level string_t type.
The book Fortran Tools for VAX/VMS and MS-DOS by Jones & Crabtree contains a description of a Fortran string-handling library. Interestingly, they decided to use null-terminated strings like in C, meaning they needed to build a separate set of functions from the intrinsic ones (concatenation operator // and length function). They later used these tools to develop a compiler for a subset of the Fortran language itself! Their conclusion about strings was:
Fortran is often maligned for its lack of facilities for character-oriented processing. ... The apparent deficiency of Fortran for string manipulation is primarily because of the methods traditionally used rather than because of a shortcoming of the language itself. The main shortcoming of Fortran for string handling is the lack of a standard library of routines for often-needed functions. As Fortran programmers we are faced with a choice: we either invest the up-front effort required to create our own standard library or we live with the continuing effort of hacking together a solution each time we are presented with similar problems.
Where is the latest ISO_VARYING_STRING implementation? Most links are dead by now. The only version I was able to find so far is this one: http://fortrangis.sourceforge.net/doc/iso__varying__string_8F90_source.html.
Where is the latest ISO_VARYING_STRING implementation? Most links are dead by now. The only version I was able to find so far is this one: http://fortrangis.sourceforge.net/doc/iso__varying__string_8F90_source.html.
I have linked three distinct implementations in the top post. The links from the gfortran compiler pages are dead as well as the link in Modern Fortran Explained by MCR.
Edit: An informal description of the iso_varying_string module for Varying Length Character Strings in Fortran can be found at: http://numat.net/fortran/is1539-2-99.html
@ivan-pi thanks. I like your plan. It looks like the iso_varying_string is in the "high-level" API category, as it operates on a VARYING_STRING derived type. Our low-level API would be similar, but operating directly on character(len=*).
Building the low-level API on character(len=*) variables will be problematic for some operations, since they can't be resized. The high-level API will need to call routines that operate on character(len=:),allocatable variables. So you may end up with two slightly different routines in some cases. So are there really three APIs?
character(len=*)character(len=:),allocatablestring_t or whatever we call itThat seems complicated to me... but it would cover all the bases...
I think there are two possible APIs here: intrinsic and derived-type one.
For the intrinsic API, character(len=*) works well for input strings. If the function will return a string of known size, you return a charecter(len=something) string. If unknown, you return an allocated character(len=:), allocatable string. User doesn't need to know which one it is.
I also see the intrinsic one as the starting point. Higher-level (derived type) implementation is likely to use the intrinsic API internally.
My understanding, and somebody correct me if I don't have this quite right, is that the ISO_VARYING_STRING standard was created before character(len=:), allocatable (around 2001 I think?), but then when character(len=:), allocatable was added to the standard, it was supposed to function like variable length strings, and so the former was mostly abandoned. However, I have found most compilers to be buggy with their implementation. Memory leaks when used as the return from a function, failure to properly reallocate on assignment, false-positive warnings about accessing uninitialized memory, etc.
If allocatable character actually worked we wouldn't a new derived type for strings. You would just use the intrinsic type and move on. But I think as written in the standard, it probably will never truly work properly in all cases (especially as in read statements, since other allocatable arrays don't and aren't supposed to).
If there is a new type for strings, I don't think a lower level library or API should be exposed, and it should probably not be based on allocatable characters.
As I mentioned above, you can use this trick to return character(len=N) strings from functions. The downside is that the string operation gets executed twice --- once to compute the length in the pure procedure, and second time to actually return it. So we probably don't want to do it that way. What I was thinking is to do what @milancurcic suggested: use character(len=*) as well as character(len=N) where we can, and use character(len=:),allocatable to avoid doing the operation twice as I described above. And that's the low level API. Below I provide two examples: https://github.com/fortran-lang/stdlib/issues/69#issuecomment-570399250 and https://github.com/fortran-lang/stdlib/issues/69#issuecomment-570403926, to show one one would decide whether to expose character(len=*) or character(len=:), allocatable.
As @everythingfunctional mentioned, for example GFortran used to have huge problems with allocatable strings and leaked memory. The latest version has improved a lot. Given that this is standard Fortran, and stdlib is a standard library, I think it is ok if we depend on the standard, and if there are compiler bugs, we'll try to workaround them and ensure they are reported. Regarding read statements, see #14 that would handle that. I think we should at least try to create a consistent low level API, not give up without even trying. If it truly cannot be done, only then we'll have to do what you propose, and only expose the string_t type and report the bugs to compilers (and keep the list somewhere) and propose improvements to the language itself, so that it can be done in the future.
I thought you could only use intrinsic procedures in variable declaration statements. Learned something new. That's a neat trick, but like you said, not particularly efficient.
Let's discuss a simple example: upcase.
Here is an implementation:
function upcase(s) result(t)
! Returns string 's' in uppercase
character(*), intent(in) :: s
character(len(s)) :: t
integer :: i, diff
t = s; diff = ichar('A')-ichar('a')
do i = 1, len(t)
if (ichar(t(i:i)) >= ichar('a') .and. ichar(t(i:i)) <= ichar('z')) then
! if lowercase, make uppercase
t(i:i) = char(ichar(t(i:i)) + diff)
end if
end do
end function
When the user wants to use it, he could do this:
character(*), parameter :: s = "Some string"
character(:), allocatable :: a
print *, s
allocate(character(len(s)) :: a)
a = upcase(s)
print *, a
which prints:
Some string
SOME STRING
The main disadvantage of this approach is that the user needs to know the size ahead of time. In this case he knows --- it's the same size as the original string. Although modern gfortran has reallocatable LHS turned on, so then just this works:
character(*), parameter :: s = "Some string"
character(:), allocatable :: a
print *, s
a = upcase(s)
print *, a
So I think that would work for upcase.
Here is the implementation using character(:), allocatable
function upcase(s) result(t)
! Returns string 's' in uppercase
character(*), intent(in) :: s
character(:), allocatable :: t
integer :: i, diff
t = s; diff = ichar('A')-ichar('a')
do i = 1, len(t)
if (ichar(t(i:i)) >= ichar('a') .and. ichar(t(i:i)) <= ichar('z')) then
! if lowercase, make uppercase
t(i:i) = char(ichar(t(i:i)) + diff)
end if
end do
end function
It's still used like this:
character(*), parameter :: s = "Some string"
character(:), allocatable :: a
print *, s
a = upcase(s)
print *, a
But since this as an extra allocation inside upcase, I would think that in this case, the character(*) version is better.
Now let's discuss integer to string conversion, the two implementations:
pure integer function str_int_len(i) result(sz)
! Returns the length of the string representation of 'i'
integer, intent(in) :: i
integer, parameter :: MAX_STR = 100
character(MAX_STR) :: s
! If 's' is too short (MAX_STR too small), Fortran will abort with:
! "Fortran runtime error: End of record"
write(s, '(i0)') i
sz = len_trim(s)
end function
pure function str_int(i) result(s)
! Converts integer "i" to string
integer, intent(in) :: i
character(len=str_int_len(i)) :: s
write(s, '(i0)') i
end function
And usage:
character(:), allocatable :: a
a = str_int(12345)
print *, a, len(a)
which prints:
12345 5
pure function str_int(i) result(s)
! Converts integer "i" to string
integer, intent(in) :: i
integer, parameter :: MAX_STR = 100
character(MAX_STR) :: tmp
character(:), allocatable :: s
! If 'tmp' is too short (MAX_STR too small), Fortran will abort with:
! "Fortran runtime error: End of record"
write(tmp, '(i0)') i
s = trim(tmp)
end function
And usage:
character(:), allocatable :: a
a = str_int(12345)
print *, a, len(a)
which prints:
12345 5
Unlike in the upcase (see previous comment), here the character(*) version is converting twice, so it is inefficient. The character(:), allocatable version just converts once, and so that would be the preferable API.
(Note: if we implement our own integer to string conversion algorithm, then we avoid the ugly MAX_STR thing and the need to call trim. The above implementation was reused from my codes, where I just use the Fortran intrinsic conversion as part of write so that I save code.)
Here is my proposal for the low level API:
character(*) where possible and efficient (see the previous two comments for examples how to decide)character(:), allocatable otherwiseUnfortunately some compilers might leak memory or segfault when such strings are used in derived types. Ultimately, long term, the compilers must be fixed. That's why I think the above proposal is a good one for the long term. In the short term, if we want to provide strings to users that actually work in all today's compilers, it might be that the only way is to create a string_t type not based on allocatable strings, in which case one could still use the low level API with the workarounds 1., 2., and 3., but make a copy of the result into the derived type string_t that is internally represented differently, so that today's compilers do not leak memory. That would be less efficient than providing a separate string implementation, but it's only a short term issue anyway, until compilers catch up. (Alternatively we can have an efficient duplicate implementation based on the internal string_t representation directly if we want better performance until compilers catch up.)
Specifically for the case of integer to string conversion, you could also dynamically allocate a buffer for each integer kind and then trim the result into an allocatable character string:
function integer_to_string2(i) result(res)
character(len=:),allocatable :: res
integer, intent(in) :: i
character(len=range(i)+2) :: tmp
write(tmp,'(i0)') i
res = trim(tmp)
end function
If we want to avoid internal I/O this function becomes something like
function integer_to_string1(ival) result(str)
integer, intent(in) :: ival
character(len=:), allocatable :: str
integer, parameter :: ibuffer_len = range(ival)+2
character(len=ibuffer_len) :: buffer
integer :: i, sign, n
if (ival == 0) then
str = '0'
return
end if
sign = 1
if (ival < 0) sign = -1
n = abs(ival)
buffer = ""
i = ibuffer_len
do while (n > 0)
buffer(i:i) = char(mod(n,10) + ichar('0'))
n = n/10
i = i - 1
end do
if (sign == -1) then
buffer(i:i) = '-'
i = i - 1
end if
str = buffer(i+1:ibuffer_len)
end function
For processing floating point values the functions are much more difficult to develop compared to those using internal read and write statements.
I did some keyword searchs in the list of popular Fortran projects. It seems that most projects use their own set of character conversion and string handling routines for stuff like reading input values from files, parsing command line options, defining settings, etc..
Here are the results of my search of some of the top projects:
| Project | # of "string" | # of "character" | # of Fortran files |
| --- | ---:| ---:| ---:|
| ElmerFEM | 248 | 1319 | 2076 |
| WRF | 306 | 966 | 1668 |
| fds | 16 | 28 | 41 |
| quantum-Espresso | 66 | 472 | 1516 |
| fluidity | 38 | 279 | 747 |
| json-fortran | 26 | 47 | 49 |
| fortranlib | 11 | 18 | 38 |
| Nek5000 | 54 | 204 | 336 |
| cp2k | 439 | 1043 | 1132 |
| nastran-95 | 85 | 551 | 1838 |
| specfem3d | 186 | 404 | 765 |
| nwchem | 323 | 2768 | 17214 |
| gtk-fortran | 59 | 77 | 92 |
| cfl3d | 14 | 216 | 397 |
| shtools | 2 | 20 | 113 |
| arpack-ng | 1 | 259 | 332 |
The second and third column measure the number of Fortran files that contain the keywords string or character, respectively. This includes both command statements and comments so it may be a bit misleading.
In one of the codebases I even found this comment:
! String parsing in Fortran
! is such a pain
! it's unreal
The purpose of these functions is to return of copy of a character string ( either character(len=*) or a derived string type) with the case converted . The common variants are uppercase, lowercase, and titlecase.
The libraries cited in the first post contain the following function prototypes:
! functional
function str_upper(str)
function str_lower(str)
function str_swapcase(str)
pure function ucase(input)
pure function lcase(input)
function str_lowercase(str)
function str_uppercase(str)
subroutine str_convert_to_lowercase(str)
subroutine str_convert_to_uppercase(str)
pure elemental function lowercase_string(str)
function uppercase(str)
function lowercase(str)
! object-oriented
procedure, pass(self) :: camelcase
procedure, pass(self) :: capitalize
procedure, pass(self) :: lower
procedure, pass(self) :: snakecase
procedure, pass(self) :: startcase
procedure, pass(self) :: upper
function vstring_tolower(this[,first,last])
function vstring_toupper(this[,first,last])
function vstring_totitle(this[,first,last])
Some versions will return a new string, while some work in place. In at least one of the functions, it did not convert the case of characters enclosed between quotation marks.
These are the similar functions available in other programming languages:
capitalize, lower, swapcase, uppercapitalize, swapcase, upcase, downcasetoLower, toLowerInPlace, toUpper, toUpperInPlace, asCapitalized, asLowerCase, asUpperCaselower, upperuppercase, lowercase, titlecase, uppercasefirst, lowercasefirstto_upper, to_lower, to_uppercase, to_ascii_uppercase, to_lowercase, to_ascii_lowercase, make_ascii_uppercase, make_ascii_lowercaseMy top three name picks are:
uppercase/lowercase/titlecaseto_upper/to_lower/to_titleupper/lower/capitalizeEdit: for consistency with the character conversions functions to_lower/to_upper in the module stdlib_experimental_ascii it is maybe better to go for option 2.
I'd like to add to the list of facilities here the overloaded operator * between integers and strings, so that you can do, like in Python:
print *, 3 * 'hello' ! prints 'hellohellohello'
print *, 'world' * 2 ! prints 'worldworld'
It's easy to make and use. The only downside I can think of is a somewhat weird API when importing it:
use stdlib_experimental_strings, only: operator(*)
Yes, I have seen this kind of usage in one of the above mentioned libraries. I am not sure whether it is not perhaps better to promote the usage of the intrinsic repeat function. As the Zen of Python states: There should be one-- and preferably only one --obvious way to do it.
A benefit of repeat is precisely that you avoid the import statement.
Oops, I didn't know about repeat. Indeed it's the way to go so I withdraw my proposal. I need to brush up on my canonical Fortran. :)
It is really hard to have a day job and keep up with all these threads, so my apologies if I've missed something because I'm just skimming here. A few opinionated notes:
split: returns an array of characters where each entry is as long as the longest onejoin: joins an array of characterssub: from Ruby, replaces the first occurance of a substring with a substitution, optional argument to replace the last stringgsub: Replaces all occurrences with a substring//: Overloaded to allow concatenation of all real, logical, integer kinds with all character kindsto_i: Convert to integer (all kinds)to_r: Convert to real (all kinds)to_l: Convert to logical (all kinds)to_s: Convert anything to a string (all kinds)I need to look at the varying string and character array proposals in more detail.
FWIW, I personally prefer the Ruby Python OO approaches with methods because it will make import statements much simpler: Pull in the string class and you get all the methods along with the type/class declaration. Now some operators may need to be pulled in as well if you want to be able to concatenate a real (lhs) with a string (rhs, can't have a TBP operator to the left of the object IIRC).
I was thinking of starting a PR marrying my work on ZstdFortranLib with a UDT/Class approach rather than operating on raw character scalars and arrays which is awkward for things like split(). But now I need to catch up on the myriad of proposals and prior art, so don't hold your breath.
While there is an intrinsic implementation, repeat(), I still like the more concise syntax which has pretty clear meaning for anyone who has ever worked with languages like Python and Ruby. It would be nice if some of these syntactic sugar items were added to the standard rather than a standard library. But until then I would be happy with * overloaded for characters.
@zbeekman I am struggling with all the threads also, but that is good news. It means there is lots of momentum. If you can help us design a good low and high level API for strings (https://github.com/fortran-lang/stdlib/issues/69#issuecomment-570409245), that would be great.
@certik k
Now let's discuss integer to string conversion, the two implementations:
I like your first one the most. With integers you can use some math to count up how many digits there are, and if you need a sign on the front, which completely removes the need to declare the max string length AND to do the IO twice. Instead you use integer and floating point math which (hopefully) will be reasonably quick. IIRC, I implemented something to do this in JSON-Fortran but I'll have to look for it.
Also, I don't mean to whine about not being able to keep up, and I agree that it's good, but it's hard to keep track of all the balls in the air.
After a brief search there are at least 3 ways to do this without performing the conversion to a string then counting digits:
fortran
len = 1
if ( n < 0 ) len = 2
do
n = n / 10
if (n == 0) exit
len = len + 1
end do
log10fortran
len = floor( log10( real( abs( n ) ) ) + 1 )
if ( n < 0 ) len = len + 1
I would guess that 1 is the fastest way to do this, but it may depend on the compiler and hardware. 3 has conversion to a real, then log10 is probably computed iteratively, and it is converted back to an int, so 1. may be faster despite the loop.
@zbeekman great idea. I suspect even faster would be 1. together with putting the digits into the string right away. And thus using the "allocatable" approach. In your approach, you compute the length of the string in 1., but then you have to do a similar loop again when you do the actual string conversion. But your approach is definitely a huge improvement, so we could us my first approach above also with this.
Why not keep going a little here:
elemental function int_str_len(n) result(res)
integer, value :: n
integer :: res
res = merge(1, 2, i >= 0)
do
n = n / 10
if ( n == 0 ) return
res = res + 1
end do
end function int_str_len
I suspect even faster would be 1. together with putting the digits into the string right away. And thus using the "allocatable" approach.
I prefer not working with allocatable character function results due to compiler bugs, but whether or not the compiler is implicitly generating a loop for write(s, '(i0)') i may be interesting from a performance perspective...
Doh! Ah, the dangers of drinking from the fire-hose of fortran-lang/stdlib: It appears @ivan-pi has essentially already figured all this out and implemented a similar version above.
See also https://github.com/fortran-lang/stdlib/issues/69#issuecomment-570409245 how I propose to handle the compiler bugs.
Ruby is my favorite language for string processing, and IMO is the best at it. If no one objects (especially @ivan-pi) I'll put links in the first post on this issue
Feel free to edit the first post. I only listed the languages that came off the top of my head.
Doh! Ah, the dangers of drinking from the fire-hose of fortran-lang/stdlib: It appears @ivan-pi has essentially already figured all this out and implemented a similar version above.
I adapted that version from a 1988 book on Fortran tools but took the "one-shot" approach to declare a sufficiently sized buffer. The range intrinsic essentially returns the value of floor(log10(huge(x)) converting the integer to a real first.
The
rangeintrinsic essentially returns the value offloor(log10(huge(x))converting the integer to a real first.
Yes I saw that. And then you cleverly work backwards from the temporary array local variable that holds the largest possible integer you could create. Pretty neat!
I think at the end of the day the allocating the right length string issue is where we'll want some benchmarking with different popular compilers. I'm not convinced that internal IO would be slower than this, but I haven't bench marked it and would happily be wrong!
In the book where I found this approach they say:
The reason for not using internal files is mainly to avoid Fortran I/O if possible on microcomputers with limited memory, with the hope that the host compiler won't include Fortran I/O in the executable image; it is also kind of nice to be able to do it on your own. In our tests of the IBM-PC progams that do not use Fortran I/O in the primitives, replacing versions of
itoaandatoiusing internal files with the ones shown here reduced the size of executable program by 21k bytes.
I cannot say whether this is true also for modern Fortran compilers or, if it even matters given the large amount of memory available to us today.
To follow @gronki's original format of:
- name of the utility
- short description
- does it exist in other languages
- proposed example of usage
ruby
" hello ".strip #=> "hello"
"\tgoodbye\r\n".strip #=> "goodbye"
"\x00\t\n\v\f\r ".strip #=> ""
"hello".strip #=> "hello"
fortran
strip("\tgoodbye\r\n") ! "goodbye", where \t is a tab char, \n a new line etc.
strip(" hello ") ! "hello"
strip("Hello") ! "Hello"
!! Or maybe for a string class
s%strip() ! returns "hello" from " hello ", etc. as above
write(*,*) vs write(*,*, ADVANCE="NO").ruby
"hello".chomp #=> "hello"
"hello\n".chomp #=> "hello"
"hello\r\n".chomp #=> "hello"
"hello\n\r".chomp #=> "hello\n"
"hello\r".chomp #=> "hello"
"hello \n there".chomp #=> "hello \n there"
"hello".chomp("llo") #=> "he"
"hello\r\n\r\n".chomp('') #=> "hello"
"hello\r\n\r\r\n".chomp('') #=> "hello\r\n\r"
!! Or use a TBP on a string class with passed first argument
Fortran
chomp("hello") ! "hello"
chomp("hello\n") ! "hello"
chomp("hello\r\n") ! "hello"
chomp("hello\n\r") ! "hello\n"
chomp("hello\r") ! "hello"
chomp("hello \n there") ! "hello \n there"
chomp("hello", sep="llo") ! "he"
chomp("hello\r\n\r\n", sep="") ! "hello"
chomp("hello\r\n\r\r\n", sep="") ! "hello\r\n\r"
!! Or use a TBP on a string class with passed first argument
characters padded out to the max length of any given element, or a ragged edge array of strings in a string class. Called without an argument or a single space remove leading and trailing whitespace and runs of multiple white space characters.ruby
" now's the time ".split #=> ["now's", "the", "time"]
" now's the time ".split(' ') #=> ["now's", "the", "time"]
"mellow yellow".split("ello") #=> ["m", "w y", "w"]
"1,2,,3,4,,".split(',') #=> ["1", "2", "", "3", "4"]
Fortran
split(" now's the time ") ! ["now's", "the", "time"]
split(" now's the time ", sep=' ') ! ["now's", "the", "time"]
split("mellow yellow", sep="ello") ! ["m", "w y", "w"]
split("1,2,,3,4,,", sep=',') ! ["1", "2", "", "3", "4"]
characters or string class using optional glue, like python, but I think the arguments are backwards in python for the object oriented case. i.e., list.join(glue) seems to make more sense to me.Python
s = "-"
s.join(['1','2','3','4']) # '1-2-3-4'
fortran
join(['1','2','3','4'], glue='-') ! '1-2-3-4'
join(['1','2','3','4']) ! '1234'
!! Or for an OO class
s = split('this, that, the other', sep=',')
! this
! that
! the other
s%join(',') ! 'this, that, the other'
s%join() ! 'thisthatthe other'
string_replace as far as I can tell. Eventually it would be nice to support regular expressions like Ruby, but, for now, we should start smaller and just make it behave like:sed 's/<pattern>/<replacement>/g'sed. Behave like Python without the optional count argumentfortran
gsub("hello", "l", "L") ! "heLLo"
gsub("the quick brown fox jumped over the lazy dog", "the", "a")
! "a quick brown fox jumped over a lazy dog"
!! Or OO approach
s = "hello"
s%sub("l","L") ! "heLLo"
fortran
sub("hello", "l", "L") ! "heLlo"
sub("hello", "l", "L", back=.true.) ! "helLo"
sub("the quick brown fox jumped over the lazy dog", "the", "a")
! "a quick brown fox jumped over the lazy dog"
sub("the quick brown fox jumped over the lazy dog", "the", "a", back=.true.)
! "the quick brown fox jumped over a lazy dog"
!! Or OO approach
s = "hello"
s%sub("l","L") ! "heLlo"
s%sub("l","L", back=.true.) ! "helLo"
ruby
"hello".center(4) #=> "hello"
"hello".center(20) #=> " hello "
"hello".center(20, '123') #=> "1231231hello12312312"
fortran
center("hello",4) ! "hello"
center("hello",20) . ! " hello "
center("hello",20,"123") ! "1231231hello12312312"
!! Or OO approach with passed first variable as before
s = "hello"
s%center(20) ! " hello "
s%center(20,"123") ! "1231231hello12312312"
to_i, to_f, etc. https://ruby-doc.org/core-2.6/String.html#method-i-to_f* edit descriptorto_s/to_str method and Python's .str()to_int or whatever the names are to concatenate strings with integers, reals, complex and logical variables and handle the string conversion with the default rulesFortran
"file" // i // ".txt" ! "file1.txt" or "file999.txt" or "file-20.txt"
100.0_rk // " degrees celsius is when water boils"
! "100.00 degrees celsius is when water boils"
! This should have a editable edit descriptor to control conversion, probably want to use a "g" edit descriptor as the default with sensible values
"Test failed? " // .true. ! "Test failed? true"
Most Python and Ruby string methods with idiomatic, and straightforward-ish implementations in Fortran would be great to have. Cases where intrinsics are already present need not be re-written unless we create a string class vs a functional style. Some others have no obvious or straightforward implementation until more infrastructure (like regex handling) is in place. Regex handlers will almost certainly need to use an external library, unless someone knows how to write parsers and lexers using tools that emit Fortran.
I think and important issue for us is to decide:
I don't think it would be that difficult to provide both a functional interface and an OO interface, and the OO interface could leverage the functional implementation in most places.
Also worth considering: Do we want basename, dirname, name_we, extname type of functions to be part of a string class or a file/os class/module?
Do we want
basename,dirname,name_we,extnametype of functions to be part of a string class or a file/os class/module?
That should be the OS module I would think.
I think and important issue for us is to decide:
1. Functional only approach 2. OO only approach 3. Both
It looks like 3. it should be, as we can all agree on that one and move on to actually implement this (as opposed to keep discussing whether to do 1. or 2.).
Responding to some of @zbeekman proposals
Doesn't trim already do exactly this? Why have two functions to do exactly the same thing?
I think you need more examples about why this is needed. Is the idea not to remove trailing whitespace if it's not a record separator? I.e. chomp("hello \n") => "hello "
There is some subtleties about how split works that need to be pointed out. In Python (and other languages?, I'd have to double check) the separator argument is taken as a list of possible separators, not as a pattern that must be found. (i.e. split("Hello ,World", ", ") => ["Hello", "World"]).
Also, should empty strings be included in the resulting array? I think probably not, or you'd get something you probably didn't intend from a standard use case like split("A, list, of, words", ", ") => ["A", "", "list", "", "of", "", "words"] due to the multiple separators next to each other.
I've got an implementation of split (splitAt) here that behaves like Python's.
The replace function from the ISO_VARYING_STRING module combines these two uses into one interface (replace(string, target, substring, every, back)). I don't know if that's better or worse, just that there is already a precedent for it.
What should happen if the string can't be converted? Most other languages throw exceptions that can be caught, but Fortran can't do that. Providing an optional iostat argument would seem to fit in with the Fortran "style".
The question really is what format should real and complex numbers take. Use the f format specifier or g, or something else? To what precision? What about trailing zeros? or rounding for stuff like 1.00000000001? I've got an implementation here that picks the shorter string from regular vs scientific notation, removes trailing zeros, and give the full precision available for that kind if a number of significant digits isn't specified, but I'm not certain that's what everyone would prefer.
Like others, I have only skimmed the postings, so I may have missed it, but has the issue of meaningful trailing blanks been discussed? Normally trailing blanks are just an inconvenience and I guess with the allocatable-length strings we have now, the issue is less pressing, but some consideration does seem useful.
@zbeekman mentioned in https://github.com/fortran-lang/stdlib/issues/69#issuecomment-570725303 parsers and lexers implemented in Fortran. Some years ago I adjusted the SQLite "lemon" parser generator so that it will emit Fortran code - https://sourceforge.net/p/flibs/svncode/HEAD/tree/trunk/src/lemon/. I merely adjusted the code generation parts of the original code and I admit not having used it much, but it could be a starting point. In the same Flibs project: Paul Fossati created an interface for the PCRE library for regexps.
has the issue of meaningful trailing blanks been discussed?
Great question, I don't think we explicitly discussed this yet. I was hoping that we will assume and require that there are no trailing blanks (both in the low level as well as the OO API), unless the user wants them there (but in that case "x" and "x " will be treated as different strings). In particular, no need to call trim all over.
Quite possibly the absence of any magic character - \0 for C-like languages and trailing blanks for Fortran - may automatically resolve the issue, but we will have to take care that such features/quirks do not sneak in ;).
@everythingfunctional: I'm not in favor of unnecessary duplication either, and this is just a personal wishlist to shape discussion
Doesn't
trimalready do exactly this? Why have two functions to do exactly the same thing?
No, trim only handles __trailing blanks__ which are spaces unless I'm misunderstanding something. Strip handles leading and trailing whitespace of all forms.
Is the idea not to remove trailing whitespace if it's not a record separator?
Yes and to provide flexibility on what that separator might be, e.g., commas after all elements except the last one in a list of items.
There is some subtleties about how split works that need to be pointed out.
Agreed. I was quoting the way Ruby does it straight from Ruby's API docs, and I think the examples I posted spell out the semantics fairly well.
In my very opinionated opinion, Ruby is the best language for string handling, so I like APIs that match, where it makes sense, Rubys. But Python has good string handling too, certainly better than Fortran. Just proposing a way to do it, not saying we have to do it that way.
The
replacefunction from the ISO_VARYING_STRING module combines these two uses into one interface (replace(string, target, substring, every, back)). I don't know if that's better or worse, just that there is already a precedent for it.
I haven't had a chance to look at ISO_VARYING_STRING yet. If people like it and there's broad support (and, even better an implementation) then I would happily defer to replace and any other useful functions/utilities. But since we're making something new, I wanted to share what I like from Ruby.
What should happen if the string can't be converted? Most other languages throw exceptions that can be caught, but Fortran can't do that. Providing an optional iostat argument would seem to fit in with the Fortran "style".
Very good question. Where a function is doing the converting it should have an iostat argument. Typically you can convert from one character set into utf8/unicode, going in the other direction requires a check that the characters being converted from unicode exist in the other character set. For concatenation, you can convert asymmetric concatenation to ISO_10646 if it's present.
The question really is what format should real and complex numbers take. Use the
fformat specifier org, or something else? To what precision? What about trailing zeros? or rounding for stuff like1.00000000001?
Yes, I implemented similar logic in JSON-Fortran (with input and help from @jacobwilliams) I think it makes sense to provide some sane defaults with a function to let the user specify their own default output format for reals, complex, integers, etc.
Paul Fossati created an interface for the PCRE library for regexps.
I was thinking that using this might be wise. But it introduces a (perhaps critical) external dependency, that may be an unpopular opinion.
has the issue of meaningful trailing blanks been discussed?
One of the nice things about most of the ruby APIs is that they give you some flexibility to change semantics for, e.g., repeated blanks, or other patterns. But a lot of this extended capability relies on a regex syntax, which, right now, is probably a non-starter.
In order to move this discussion forward --- it seems the implementations are (relatively) straightforward, the hard part is to agree on an API: a set of functions, their name, arguments for the low level API, and similarly for the high level API. If that's the case, @zbeekman, @ivan-pi do you want to start a document where we'll start discussing the function names and their arguments and functionality?
I don't know if the best way is to open a few draft PRs with the document (I assume there will be a few competing versions at first), or if there is another way to discuss a full document.
Sure. I need to make myself familiar with N1375 and N1379 And see how it sits with what I had anticipated doing. There's no point reinventing the wheel, so if ISO_VARYING_STRING appeals to me and seems congruent with the opinions I've heard in this thread so far it might be a good starting point. @everythingfunctional seems to already have an MIT licensed implementation, so that could even be a good starting point.
ISO_VARYING_STRING operates on a derived type, so that is the high level API. The low level API would operate on character string directly.
I was hoping to also have some document that compares different languages, like @ivan-pi did above, so that we can stay compatible with the API, whenever it makes sense.
I was hoping to also have some document that compares different languages, like @ivan-pi did above, so that we can stay compatible with the API, whenever it makes sense.
I realized there is a page for this on Wikipedia:
https://en.wikipedia.org/wiki/Comparison_of_programming_languages_(string_functions)
As you might notice many of the string functions do not have a Fortran equivalent. Exactly those are the ones we should try and implement here.
I don't know if the best way is to open a few draft PRs with the document (I assume there will be a few competing versions at first), or if there is another way to discuss a full document.
By document are you implying simply a markdown file? (ideally these string function specifications would later become part of the documentation)
@ivan-pi the wikipedia page you posted is perfect, that's what I was hoping to create. Now when this is done, I think all we need is just some markdown document where we start listing the Fortran functions and their names and arguments, and we will consult the wikipedia page to see how other languages do that. Maybe even Wiki would be good enough to start. Do you want to start a wiki page at https://github.com/fortran-lang/stdlib/wiki ?
\
Missed this. It looks bright with promise. A lot to catch up on. A scanned the discussion and thought it was worth noting that conversion from a numeric value to a string is almost
always to build a message. Overloading the concatenation operator is a great way to do
that, but one of my favorite approaches is demonstrated in the MSG() function in the M_STRINGS module mentioned in the top post of the discussion. It allows constructs like
message=msg('The value is',A,'which squared is ',A**2,'and it is either ',.true.,'or not')
I find I use that type of function a lot.
More basic is that I saw mention of not using internal I/O to do the conversion to a string.
That is really important if you want your function to be safely called from an I/O function
such as a WRITE, and to be able to use it from within a PURE function. The algorithms to do a floating point value as well as an INTEGER value can be found back as far as the
1968 version of "Software Tools" in Ratfor!
But in order of frustration that they are not there in Fortran is case conversion, then string splitting, then conversion to and from strings, then Regular Expressions.
And I like the name "center" for centering a string, but I would vote for the name ADJUSTC() for a function that works on CHARACTER variables to complete the set (ADJUSTL(), ADJUSTR(), ...).
A lot to catch up on here but a great step in the right direction, I think. Is the main objective to create a specification for the Fortran standard or to create a working library in the Public Domain (or both?). It wasn't quite clear to me. Assuming a working library emerges what will the LICENSE be and how will it be hosted and documented? Maybe I missed that part of the discussion? I'm not sure I'm starting in the right place, actually.
As a footnote,
the next most common issue being solved when converting a numeric value to a string is to present the number in a form the language does not support, like a non-standard base or perhaps in Roman numerals or in English text, for example.
More basic is that I saw mention of not using internal I/O to do the conversion to a string. That is really important if you want your function to be safely called from an I/O function such as a WRITE, and to be able to use it from within a PURE function. The algorithms to do a floating point value as well as an INTEGER value can be found back as far as the 1968 version of "Software Tools" in Ratfor!
Did you avoid internal I/O in your library M_STRING? I never really thought about is internal I/O pure or not. But if that is really the case I would prefer to do our own string conversions routines.
And I like the name "center" for centering a string, but I would vote for the name ADJUSTC() for a function that works on CHARACTER variables to complete the set (ADJUSTL(), ADJUSTR(), ...).
Would you allow adjustc to have an optional width parameter or not? I like this name idea.
Is the main objective to create a specification for the Fortran standard or to create a working library in the Public Domain (or both?)
I think the initial purpose is to create a working library in the public domain (MIT or BSD license) which hopefully becomes standard in the feature. I guess a similar model is how some of the C++ Boost function libraries later become integrated into the language or the C++ standard library. You can read the related thread https://github.com/j3-fortran/fortran_proposals/issues/104 for more information.
After checking the 2018 standard it says specifically the unit cannot be an integer or asterisk, which implies internal I/O is OK; but my compiler gives an error if I include any WRITE statement, so I guess that internal I/O appears to be allowed.
I do have versions of a metamorphic class that takes any intrinsic type and does no I/O but it is not in the github versions of M_STRINGS. I was actually going to make a version this week, by coincidence. Now that I see it is my compiler and not the standard stopping me from declaring the procedure PURE and I am trying to read through everything here I think I will play catch-up here instead.
On January 8, 2020 at 3:03 AM Ivan notifications@github.com wrote:
> >More basic is that I saw mention of not using internal I/O to do the conversion to a string. That is really important if you want your function to be safely called from an I/O function such as a WRITE, and to be able to use it from within a PURE function. The algorithms to do a floating point value as well as an INTEGER value can be found back as far as the 1968 version of "Software Tools" in Ratfor! > Did you avoid internal I/O in your library M_STRING? I never really thought about is internal I/O pure or not. But if that is really the case I would prefer to do our own string conversions routines.> >And I like the name "center" for centering a string, but I would vote for the name ADJUSTC() for a function that works on CHARACTER variables to complete the set (ADJUSTL(), ADJUSTR(), ...). > Would you allow adjustc to have an optional width parameter or not? I like this name idea.,I find the length useful particularly with centering, but find overloading adjustl and adjustr to have a length parameter useful too. With centering in particular you are often doing something like TITLE=adjustc("my title",len(title)) and it takes quite a bit of fiddling using another routine around the fixed text otherwise.
> >Is the main objective to create a specification for the Fortran standard or to create a working library in the Public Domain (or both?) > I think the initial purpose is to create a working library in the public domain (MIT or BSD license) which hopefully becomes standard in the feature. I guess a similar model is how some of the C++ Boost function libraries later become integrated into the language or the C++ standard library. You can read the related thread [j3-fortran/fortran_proposals/#104](j3-fortran/fortran_proposals#104 https://github.com/j3-fortran/fortran_proposals/issues/104 for more information.— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/fortran-lang/stdlib/issues/69?email_source=notifications&email_token=AHDWN3J53MZOILMO7WSPY7DQ4WCDXA5CNFSM4KCFW352YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEILQSII#issuecomment-571935009 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AHDWN3KRISWEN277RKGU3VLQ4WCDXANCNFSM4KCFW35Q .
Re regexp: a couple of years ago I found a series of articles by Russ Cox about regular expression engines. The accompanying source code is not too difficult and I made a start rewriting it in Fortran. However, there is a flaw in my code and I never got around to correct it (the code is not too difficult but it is manipulating lists). I picked it up again. The result will not be a full-fledged RE engine à la PCRE, but it will be Fortran only and it should be useable for not entirely trivial tasks.
I never really thought about is internal I/O pure or not. But if that is really the case I would prefer to do our own string conversions routines.
Based on the fact that I was able to write pure toString functions here, internal I/O is pure.
@urbanjost thanks for the post. As @ivan-pi replied, the goal of this stdlib effort is to provide a Fortran Standard Library, i.e., both a library and a specification. See my answer to a similar question. The license is MIT (https://github.com/fortran-lang/stdlib/blob/006bedafc0d40ff381da2bd4455f61b5e11fc2ee/LICENSE), and we will only depend on 3rd party code that is MIT or BSD style licensed. The way we plan to achieve our goal it to have a large community designing the API and a rigorous (high bar) process to get new features in as documented in our WORKFLOW document. And we have been coordinating with the Fortran Standards Committee (this effort started at the J3 committee repository at https://github.com/j3-fortran/fortran_proposals/issues/104), and also we are planning to getting them involved in the step 5 in the workflow (at least informally). Our goal is to get a wide community agreement and acceptance to adopt stdlib as the Fortran Standard Library. We will continue working closely with the Fortran Standard Committee and coordinate with them. I can imagine many arrangements in the future, up to even the Fortran Standard itself specifying a Standard Library; but that is far in the future. Right now our job is to get the community to agree on the APIs and to provide specifications and implementations and to build a community around it.
(Update: we added the motivation into README: https://github.com/fortran-lang/stdlib#goals-and-motivation)
datetime-fortran used internal I/O in a pure function for a long time (since the beginning I think) and this built fine with gfortran and ifort.
After reading through this thread I found subtle issue with the proposed low-level API for character(len=*) variables.
My top three name picks are:
uppercase/lowercase/titlecaseto_upper/to_lower/to_titleupper/lower/capitalizeEdit: for consistency with the character conversions functions
to_lower/to_upperin the modulestdlib_experimental_asciiit is maybe better to go for option 2.
Taking just the basic functionality mentioned by @ivan-pi here, I implemented a stdlib_character module as demonstration in #310, the issue becomes quickly apparent once you try to use both stdlib_character and stdlib_ascii in the same scope.
I created an exploratory implementation of a functional string handling at awvwgk/stdlib_string as fpm project. A non-fancy string type is implemented there, which basically provides the same functionality as a deferred length character but can be used in an elemental rather than a pure way. The idea is to have a scaffold for the string type in stdlib which can be extended later but already provides everything we are used to have from the deferred length character without the rough edges.
The overall implementation comes close to iso_varying_string, but it is not an iso_varying_string implementation. The main difference to iso_varying_string are
get and put
Most helpful comment
I would vote that the low level API be based on functions, pure and elemental where possible and appropriate. I would stick with the Fortran convention of optional status parameters where there is the possibility of things going wrong, and if one is not provided and something goes wrong it crashes. I have tended to use that convention in any routines that go from a string to some intrinsic like:
Honestly, I thought the ISO_VARYING_STRING standard did a great job of covering all of the intrinsic functions available for
character(len=*)variables, and extending IO to work with that type (put,get,put_line). Aside from the strange interface forsplitI think it's a great starting point.