lsd with --color=always doesn't work with awk well

[BannedJ]

Expected behavior

The command: lsd -l | awk '{print $11}'
outputs:

4.txt
6.txt
7.txt
8.txt
aes_ecb.py
base64tohex.py
break_repeating_xor.py
detect_aes_ecb.py
detect_xor.py
fixedXOR.py
__pycache__
repeating_key_xor.py
single_byte_xor.py

I would expect for lsd to have the same functionality when called with --color=always

Actual behavior

Instead, the command lsd -l --color=always | awk '{print $11}'
outputs:

4.txt
2016
2016
8.txt
2019
16:44:26
16:52:07
2019
18:57:14
2019
18:10:34
16:34:05
14:59:14

So some dates, some hours, and a couple of file names.
I believe this might be helpful in troubleshooting the problem:

[tron@Archstation ~/crypto/set_1] $ ls -l --color=always
.rw-r--r-- tron wheel 19.4 KB Mon Mar 28 18:56:31 2016   4.txt
.rw-r--r-- tron wheel  3.8 KB Mon Mar 28 18:56:31 2016   6.txt
.rw-r--r-- tron wheel  3.8 KB Mon Mar 28 18:56:31 2016   7.txt
.rw-r--r-- tron wheel 63.9 KB Mon Mar 28 18:56:31 2016   8.txt
drwxr-xr-x tron wheel    4 KB Wed Jul 10 19:26:30 2019   __pycache__
.rwxr-xr-x tron wheel  220 B  Thu Jul 11 16:44:26 2019   aes_ecb.py
.rwxr-xr-x tron wheel  434 B  Tue Jul  9 16:52:07 2019   base64tohex.py
.rwxr-xr-x tron wheel  2.7 KB Thu Jul 11 16:32:10 2019   break_repeating_xor.py
.rwxr-xr-x tron wheel  329 B  Thu Jul 11 18:57:14 2019   detect_aes_ecb.py
.rwxr-xr-x tron wheel  1.1 KB Wed Jul 10 19:03:41 2019   detect_xor.py
.rwxr-xr-x tron wheel  487 B  Tue Jul  9 18:10:34 2019   fixedXOR.py
.rwxr-xr-x tron wheel  814 B  Thu Jul 11 16:34:05 2019   repeating_key_xor.py
.rwxr-xr-x tron wheel  987 B  Wed Jul 10 14:59:14 2019   single_byte_xor.py
[tron@Archstation ~/crypto/set_1] $ ls -l --color=always | awk '{print $3}'
wheel
wheel
wheel
wheel
wheel
wheel
wheel
wheel
wheel
wheel
wheel
wheel
wheel
[tron@Archstation ~/crypto/set_1] $ ls -l --color=always | awk '{print $4}'
19.4


63.9









[tron@Archstation ~/crypto/set_1] $ ls -l --color=always | awk '{print $5}'
KB
3.8
3.8
KB
4
220
434
2.7
329
1.1
487
814
987

[Peltoche]

Hello @BannedJ , thanks for the report.

It seems that awk doesn't handle well our tabulation. I don't know well awk but for me it should be able to handle dynamic indentation correctly so I maybe it don't like some of the colors code.

FYI you can do the same stuff with the --blocks option.

[BannedJ]

The problem stems from the fact that you color the spaces used for the dynamic indentation. I found that if you restrict coloring to non-whitespace characters awk is able to handle it. (I did this by hand swapping bytes in the output)
Since awk works well with the original ls output, I suggest adjusting lsd to that standard.
How does the --blocks option work? I couldn't find any documentation for it.

[meain]

@BannedJ Thanks for the further debugging that you did, that was really useful insight. I think we do have the color getting assigned to the padding as well. Also do you have any idea as to how normal ls handles things with spaces in between, as in things like 13 KB?

About the blocks option. The --blocks option, even though it does not solve your original issue, does let you choose the blocks in the output. As in you could do something like lsd -l --blocks name,size to just show name and size.

[bknowles]

IMO, the right solution here is to not have spaces between the digits and the unit designation.

So, instead of "19.4 KB", it should be "19.4KB", or maybe "19.4kB". Skipping the space would be consistent with /bin/ls -lah type of format.

A more pedantic person might ask what kind of "K" are we talking about here, is that binary K=1024, or base-10 K=1000, and whether that be represented in NIST or IEC standard format (see https://en.wikipedia.org/wiki/Binary_prefix).

[bknowles]

But if you're going to represent just bytes for size, please don't put a thousands separator in that number, unless explicitly requested.

I find that https://github.com/ogham/exa handles the binary vs. base-10 byte representation correctly, but they mishandle the thousands separator.

[bknowles]

Actually, it turns out that lsd --size short already does the right thing with regards to spaces. IMO, that behaviour should be made the default, and then this issue could probably be closed.

[bknowles]

@BannedJ Thanks for the further debugging that you did, that was really useful insight. I think we do have the color getting assigned to the padding as well. Also do you have any idea as to how normal ls handles things with spaces in between, as in things like 13 KB?

About the blocks option. The --blocks option, even though it does not solve your original issue, does let you choose the _blocks_ in the output. As in you could do something like lsd -l --blocks name,size to just show name and size.

I'm trying this version of the command, and I get an error:

$ lsd -l --blocks name,size
error: Found argument '--blocks' which wasn't expected, or isn't valid in this context

USAGE:
    lsd --long

For more information try --help

So far as I can tell, that option is not available:

$ lsd --help
lsd 0.15.1
An ls command with a lot of pretty colors and some other stuff.

USAGE:
    lsd [FLAGS] [OPTIONS] [--] [FILE]...

FLAGS:
    -a, --all               Do not ignore entries starting with .
    -A, --almost-all        Do not list implied . and ..
        --classic           Enable classic mode (no colors or icons)
    -d, --directory-only    Display directories themselves, and not their contents
        --help              Prints help information
    -h, --human-readable    For ls compatibility purposes ONLY, currently set by default
    -F, --classify          Append indicator (one of */=>@|) at the end of the file names
    -l, --long              Display extended file metadata as a table
    -1, --oneline           Display one entry per line
    -R, --recursive         Recurse into directories
    -r, --reverse           Reverse the order of the sort
    -S, --sizesort          Sort by size
    -t, --timesort          Sort by time modified
        --tree              Recurse into directories and present the result as a tree
    -V, --version           Prints version information

OPTIONS:
        --color <color>...              When to use terminal colours [default: auto]  [possible values: always, auto, never]
        --date <date>...                How to display date [default: date]  [possible values: date, relative]
        --depth <num>                   Stop recursing into directories after reaching specified depth
        --group-dirs <group-dirs>...    Sort the directories then the files [default: none]  [possible values: none, first, last]
        --icon <icon>...                When to print the icons [default: auto]  [possible values: always, auto, never]
        --icon-theme <icon-theme>...    Whether to use fancy or unicode icons [default: fancy]  [possible values: fancy, unicode]
        --size <size>...                How to display size [default: default]  [possible values: default, short]

ARGS:
    <FILE>...     [default: .]

[bknowles]

[tron@Archstation ~/crypto/set_1] $ ls -l --color=always | awk '{print $5}'
KB
3.8
3.8
KB
4
220
434
2.7
329
1.1
487
814
987

Sadly, adding the --size short option here doesn't seem to help:

$ lsd -l --color=always --size short | awk '{print $5}'
96B
Mon
352B
448B
1.8K
Fri
Thu
8B
2.1K
224B
128B
Mon
249B
3.5K
Fri
416B

Wed
Fri

Mon
4B
3K
4B

And removing the --color always option again gives us the correct output:

$ lsd -l --size short | awk '{print $5}'
Thu
Mon
Wed
Mon
Wed
Fri
Thu
Mon
Fri
Fri
Fri
Mon
Wed
Fri
Fri
Mon
Tue
Wed
Fri
Mon
Mon
Wed
Wed
Mon

[meain]

I'm trying this version of the command, and I get an error:

@bknowles We have not released that yet, my bad. But just so you know, this is how it would look like.

@Peltoche We should have a release pretty soon, we have had quite a few new features.

[Peltoche]

I will try to make a release today if possible but no promises (work and all...). Meain what do you needs to be able to make releases without me? I don't want to be a spof.

[meain]

@Peltoche I am not sure. I have created a draft release with the changelog. Not sure how you are handling adding the Assets.

[Peltoche]

You needs to create a new tag. After some thoughts it seems very complicated for you to create a release at the moment. There is a lot of documentation missing and there is some stuff you can't do like publish a new release on crates.io or snapcraft.

I have created https://github.com/Peltoche/lsd/issues/258 in order to follow this issue.

[Peltoche]

This issue should be fixed, don't hesitate to reopen it if needed