Julia: call normpath on include("foo/bar.jl") paths?

[mortenpi]

The forward slash in abstractdataframe/iteration.jl:33 below should be a backslash.

julia> DataFrames.eachcol |> methods
# 1 method for generic function "eachcol":
[1] eachcol(df::DataFrames.AbstractDataFrame) in DataFrames at C:\Users\morten\.julia\v0.7\DataFrames\src\abstractdataframe/iteration.jl:33

Only appears to be affecting package files (base and stdlib seemed fine) that are within a subdirectory of <package>/src/.

Present on the latest Windows nightly binary, but I am also seeing it on v0.6.2.

Julia Version 0.7.0-DEV.4543
Commit 00ac33a3ef* (2018-03-12 04:07 UTC)
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: Intel(R) Core(TM) i5-6300U CPU @ 2.40GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-3.9.1 (ORCJIT, skylake)

[fredrikekre]

What happens if you use include(joinpath("abstractdataframe", "abstractdataframe.jl")) instead of include("abstractdataframe/abstractdataframe.jl") here?

[mortenpi]

Yup, it appears that's where it's coming from. The path should probably be normalized?

[stevengj]

In general, you have to use joinpath for path separators everywhere in Julia.

There is an argument for just allowing forward slashes in all Julia paths, and automatically converting them to backslashes as needed on Windows, since / is not valid in Windows filenames. One tricky case is passing to Cmd objects (i.e. for spawning processes), since you want to distinguish between the case where the literal string /bar is being passed vs. a path that should be converted to \bar.

The argument for doing this only in include seems weaker to me, but it is not completely crazy. There is no ambiguity in the include argument regarding whether a path is intended, and there is some precedent (e.g. the C #include statement) for normalizing include paths. Furthermore, since literal paths are extremely common in include files, allowing / on all operating systems would clean up a lot of code that currently uses verbose joinpath calls.

[stevengj]

Moreover, as @mortenpi alludes to, the normpath function already converts forward slashes to backslashes on Windows, so if we just call normpath on the include argument that will solve the problem.

But should any other file functions use normpath? open etcetera? Note that normpath could be optimized to a no-op in many cases.

[stevengj]

Alternatively, since Windows path arguments are already converted to a UTF-16 copy by libuv etcetera for handling Unicode paths (grep for MultiByteToWideChar in libuv), we could implement this efficiently in a very low level way, converting slashes in all functions that explicitly use a path argument, by doing the slash conversion in-place on the UTF-16 copy.

[stevengj]

Actually, I take it back — I misunderstood what is going on here. Windows APIs already accept forward slashes in paths. Backslashes are apparently only required (for the most part) in command-line applications (to distinguish path separators from /options).

So the question here is just whether we should call normpath before displaying paths, or should we display them as they were entered?

(I now am starting to feel like 99.9% of the uses of joinpath in Julia code are useless verbosity.)

[yuyichao]

Last time I argued for not using joinpath it seems that people just don't like the display...... https://github.com/JuliaLang/julia/pull/21395

[vtjnash]

Backslashes are apparently only required

This is a weird part of the Windows API (well, that seems to be said a lot, but I digress). _Technically_, \\ is the only disallowed character, and / is a valid filename. Practically speaking, that's almost never encountered in the wild, since such a file would be invisible to most Windows tools (you should be able to prefix any path with \\?\ to indicate a desire to be able to access a file with a forbidden name, if necessary, although this particular name would only be valid for WSL or NFS). Hard to know if any of that matters though...

[stevengj]

@vtjnash, according to msdn, both forward and back slashes are "reserved characters" in filenames.

[vtjnash]

Yes, but note that the name of that section is "Conventions" not "Rules". Adding \\?\ to the beginning lets you disable the conventions (although NT or NTFS still rejects most of those characters, including /). But at the end of the day, it's up to the file system driver to decide what file names can be represented and which can't. As one post noted though, it's also pretty trivial to alter the FAT file tables with a hex editor and put any character in there that you want, including / and \\ (and also likely rendering that file inaccessible to most if not all file system drivers).