Powershell: -tail only line based?!?

[sslyle]

Why is -tail only line based?
What if I want the last 100 characters from a 2G file that isn't line delimited? As is that case with, say json, xml, & other data files?
Also know that in this case -Tail hangs forever if the file is not line delimited.

Honestly! Sometimes I truly wonder if Microsoft even wants people to use there software.

[kvprasoon]

Why is -tail only line based?

Its designed to tail lines, From the docs

-Tail
Specifies the number of lines from the end of a file or other item.

For json, we have ConvertTo-Json and for XML we have Select-Xml or you can read and cast to [xml]

[sslyle]

Can you provide an example?
Last 30 characters from a 2 gig file that is one long string, no carrage
returns or linefeeds

On Tue, Nov 27, 2018, 11:01 PRASOON KARUNAN V <[email protected]
wrote:

Why is -tail only line based?

Its designed to tail lines, From the docs

-Tail
Specifies the number of lines from the end of a file or other item.

For json, we have ConvertTo-Json and for XML we have Select-Xml or you
can read and cast to [xml]

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/PowerShell/PowerShell/issues/8343#issuecomment-442112533,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFZh2irsVOzL4i_-j37_VXLBgj6k2WAtks5uzWG2gaJpZM4Y1ueK
.

[mklement0]

You can combine -Tail with -AsByteStream, in which case you get a stream of the requested number of bytes as [byte] instances (which become an [object[]] array of [byte] instances when captured in a variable).

Note, however, that that is not the same as simply passing the raw bytes through.
PowerShell doesn't support outputting raw binary data via the pipeline - see #1908

[sslyle]

You understand people want easy right? Easy is easier for you to provode
and support than accurate is.

There is no reason Get-Content can have a -l N option to importantly show
just the last N characters. Well. No reason except flawed design choices

On Tue, Nov 27, 2018, 11:50 Michael Klement <[email protected] wrote:

You can combine -Tail with -AsByteStream, in which case you get a stream
of the requested number of bytes as [byte] instances (which become an
[object[]] array of [byte] instances when captured in a variable).

Note, however, that that is not the same as simply passing the raw bytes
through.
PowerShell doesn't support outputting raw binary data via the pipeline -
see #1908 https://github.com/PowerShell/PowerShell/issues/1908

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/PowerShell/PowerShell/issues/8343#issuecomment-442131631,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFZh2m64B-rhvqBNRvzrR6CMOkALkTdmks5uzW08gaJpZM4Y1ueK
.

[mklement0]

@sslyle:

You could channel all the extra energy you're expending on insults and antagonizing into writing up a proper feature request.

^{Disclaimer: I'm just a fellow PowerShell user. The opinion expressed is solely my own.}

[SteveL-MSFT]

@sslyle please be aware of our Code of Conduct. You're more likely to get your issue addressed if you contribute more positively.

[HumanEquivalentUnit]

Can you explain what situations you might want the last N characters from a JSON or XML file? Those are both structured formats, and you would likely start reading in the middle of a closing tag or a JSON data field. Perhaps whatever high level check you are doing ("that the structure ends properly"?) would be better added in the XML or JSON processing areas rather than Get-Content.

What if I want the last 100 characters from a 2G file that isn't line delimited?

PowerShell allows you to use .Net framework directly to do things it hasn't wrapped in cmdlets yet, so you can do this by opening the file and Seek()ing to a point somewhere before the end, e.g.

$numBytes = 20
$file = Get-Item -Path .\bigfile.txt

# Open file as a stream, 
# and seek back from the end.
$stream = [System.IO.File]::OpenRead($file.FullName)
[void]$stream.Seek(-$numBytes, [System.IO.SeekOrigin]::End)

# Read some bytes from it into a buffer,
# and decode them into text
$buffer = [byte[]]::new($numBytes)
[void]$stream.Read($buffer, 0, $numBytes)
$stream.Close()
[System.Text.Encoding]::UTF8.GetString($buffer)

NB. you need to choose the [System.Text.Encoding] which matches the file's encoding, and account for the fact that reading 20 bytes won't always read 20 characters. UTF16 is 2-bytes-per-character so for that you'd need to read twice as many bytes as the character count. UTF8 is a variable width encoding so there's no way to know how many bytes to read, but it maxes out at 4-bytes-per-character, so you'd need to read four times the character count as bytes, and then trim down the resulting string if it turned into too many characters.

[sslyle]

I can write code. Write my own command/script to do this. In fact that is
what I restored to.
The point is PowerShell is a technology.
The point of technology is it is supposed to make our lives easier.

On Wed, Nov 28, 2018 at 3:26 AM HumanEquivalentUnit <
[email protected]> wrote:

Can you explain what situations you might want the last N characters from
a JSON or XML file? Those are both structured formats, and you would likely
start reading in the middle of a closing tag or a JSON data field. Perhaps
whatever high level check you are doing ("that the structure ends
properly"?) would be better added in the XML or JSON processing areas
rather than Get-Content.

What if I want the last 100 characters from a 2G file that isn't line
delimited?

PowerShell allows you to use .Net framework directly to do things it
hasn't wrapped in cmdlets yet, so you can do this by opening the file and
Seek()ing to a point somewhere before the end, e.g.

$numBytes = 20
$file = Get-Item -Path .\bigfile.txt

Open file as a stream,

and seek back from the end.

$stream = [System.IO.File]::OpenRead($file.FullName)
[void]$stream.Seek($file.Length - $numBytes, [System.IO.SeekOrigin]::Begin)

Read some bytes from it into a buffer,

and decode them into text

$buffer = [byte[]]::new($numBytes)
$stream.Read($buffer, 0, $numBytes)
$stream.Close()
[System.Text.Encoding]::UTF8.GetString($buffer)

NB. you need to choose the [System.Text.Encoding] which matches the file's
encoding, and account for the fact that reading 20 bytes won't always read
20 characters. UTF16 is 2-bytes-per-character so for that you'd need to
read twice as many bytes as the character count. UTF8 is a variable width
encoding so there's no way to know how many bytes to read, but it maxes out
at 4-bytes-per-character, so you'd need to read four times the byte count
and then trim down the resulting string if it turned into too many
characters.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/PowerShell/PowerShell/issues/8343#issuecomment-442360731,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFZh2j1r69UwiN1Xv1efWXqjAEttQvJqks5uzkjMgaJpZM4Y1ueK
.

--

"In preparing for battle I have always found that plans are useless, but
planning is indispensable."

• General Dwight D. Eisenhower

“For the things we have to learn before we can do them, we learn by doing
them.”

• Aristotle, The Nicomachean Ethics

"A leader is someone who steps back from the entire system and tries to
build a more collaborative, more innovative system that will work over the
long term."

• Robert Reich

[vexx32]

Fantastic. As has been mentioned, this is not a simple solution due to file encoding potentially making it tricky to determine what "X characters from the end" actually _is_.

I'm sure you or someone else could probably implement this as a feature if its behaviour could be concretely defined. 🙂

Without a concrete definition of exactly what you're after, we'll just be talking in circles and not really get anywhere, however.

[sslyle]

Are you saying a '-L ' option (since '-Last' is already taken)
added to Get-Content that is also incorporated into the current ' -Encoded'
option is just too difficult?
I'm confused.
...
But doesn't matter.
Good luck and keep doing the things you do well.
Delete the original post.
Ban my account.

On Wed, Nov 28, 2018, 08:40 vexx32 <[email protected] wrote:

Fantastic. As has been mentioned, this is not a simple solution due to
file encoding potentially making it tricky to determine what "X characters
from the end" actually is.

I'm sure you or someone else could probably implement this as a feature if
its behaviour could be concretely defined. 🙂

Without a concrete definition of exactly what you're after, we'll just be
talking in circles and not really get anywhere, however.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/PowerShell/PowerShell/issues/8343#issuecomment-442449798,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFZh2m5A_H3lzJ8TYQjFAw3vZHG_eMeLks5uzpIxgaJpZM4Y1ueK
.

[vexx32]

Nope. I said your request needed to be clarified. As Steve mentioned, please see the Code of Conduct he linked; if you want us to assist, we need to be clear on what you're actually asking for.

PowerShell's style is typically verbose and I would imagine a more sensible parameter name would likely be -TailBytes or some such. -Last is not _taken_ on Get-Content but implies probably too similar a thing to -Tail without a distinction of what it's doing.

If you just wanted the last N bytes, though, as @mklement0 mentioned, you already can:

$Bytes = 30
Get-Content .\file.txt -AsByteStream -Tail $Bytes

Your request isn't clear. Do you want the last N bytes, or the last N _properly encoded/decoded characters_? Those are two different things entirely. The latter is perfectly doable, though a little messy to do, and yes you could conceivably design such a parameter.

[sslyle]

But -Tail ONLY works with files that have /
As I've already said

On Wed, Nov 28, 2018, 08:57 vexx32 <[email protected] wrote:

Nope. I said your request needed to be clarified. As Steve mentioned,
please see the Code of Conduct he linked; if you want us to assist, we need
to be clear on what you're actually asking for.

PowerShell's style is typically verbose and I would imagine a more
sensible parameter name would likely be -TailBytes or some such. -Last is
not taken on Get-Content but implies probably too similar a thing to
-Tail without a distinction of what it's doing.

If you just wanted the last N bytes, though, as @mklement0
https://github.com/mklement0 mentioned, you already can:

$Bytes = 30
Get-Content .\file.txt -AsByteStream -Tail $Bytes

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/PowerShell/PowerShell/issues/8343#issuecomment-442455376,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFZh2uIk59UHENyBuoHoaAMhbLkXwEJJks5uzpZkgaJpZM4Y1ueK
.

[sslyle]

.

[vexx32]

Get-Content -AsByteStream -Tail 30 returns only the last 30 bytes of a file for me, regardless of whether the file is line-delimited or not.

Are you seeing something different?