Privatebin: Equal Sign in the end of link

[team-unix]

Am new to github and this is not exactly a "fault" of privateBin but I didn't know where else to post this. Hope this is okay.

Steps to reproduce

1. Create (one time) pastebin as per normal without password
2. Always get link with equal sign in the end (like so https://mydomain.com/?f17e950f270ade74#mELpc75muMOLAIDrMa/aQzEVF5SQw788m2JbadIgu4c=)
3. Paste created link into text-processer or e-mail software (for me MS Outlook)

What happens

Automatic clickable link creation from outlook doesn't take equal sign into account and "inexperienced" users click the link and get password query because ofcourse the link without the equal sign isn't correct.

What should happen

Equal sign shouldn't be last sign of link to have better automatic link creation among different processers.

Is there a way to get rid the equal sign in the end of the url without having to use an url shortener?

Additional information

PrivateBin version:

I can reproduce this issue on https://privatebin.net: Yes

[rugk]

Yeah, this is due to the paste key being base64-encoded and this encoding ends with a =.

However, as far as I see it, we can certainly just drop that = at the end and automatically append it when loading a paste. I see the problems with that not always being recognized as a link.

However, there is one problem: Base64 may end with zero, one, or with two = as a padding so the input length is dividable by three.
But that may be omitted if we know the length of the input; and actually we know – don't we @elrido?

[elrido]

While yes, we do always encode a 256 bit piece of random data, AFAIK it depends on the used base64 implementation if it is able to handle incorrect padding or not. We could work around that and simply try to add one, then two = if the decoding of the key fails.

I would suggest to mark this as help wanted as it affects only some infrequent use cases (may be frequent to the OP, while most folks will avoid Outlook like the plague).

[rugk]

Well… I indeed would like to know what other implementations do not regard = as a link.

[team-unix]

Hello guys. Thanks for trying to help out. It seems, that only Microsoft Office products have this dumb issue with disregarding trailing equal signs. Luckily I have found this article on stackoverflow

https://stackoverflow.com/questions/26674217/trailing-equals-sign-discarded-in-hyperlink-on-microsoft-outlook

As a workaround it is possible to put the link into sharp brackets like so

Maybe someone else will have this problem too so I guess this could help out others.

I don't know how important a full solution to this is. But atleast where I come from (working for an IT-Security company) Outlook/Exchange is basically used everywhere, so I would assume there's others falling into this pit.

[rugk]

So yeah, I guess we would be fine with a workaround anyway, would not we, @elrido? Personally, I don't care, the change should be small enough. :smile:

[HLeithner]

Why not using base64url Encoding without padding like JSON Web Signature used by Lets Encrypt for example.

[rugk]

Why not using base64url Encoding without padding like JSON Web Signature used by Lets Encrypt for example.

The problem is, we need to stay backwards-compatible. So while we could change it for now and all later paste (links) created by PrivateBin, we also need to be able to open old links that still have the equal sign.

That said, the general idea was already discussed, see https://github.com/PrivateBin/PrivateBin/issues/377#issuecomment-440393697. But, @HLeithner, do you have more details/link about how JSON Web Signatures do base64 encoding?

[HLeithner]

sadly base64url is incompatible with the default one, so compatibility has to be done by a flag or anything that marks new data ad base64url encoded.

There is a npm package for base64url that can be used for encoding and decoding, that it seams autodetection of base64 strings is not possible.

But wikipedia and the rfc and a blog post tells us that we can replace + with - and / with _ and as already said in comment https://github.com/PrivateBin/PrivateBin/issues/377#issuecomment-440946044 we know the length and can add the missing =. I'm not sure if the npm package can do this automatically.

At least this method would be safer for all mail clients.

[elrido]

If we consider switching the scheme as part of the ongoing paste format change: How about human-oriented base-32 encoding (z-base-32)?

Its goals seem to overlap with the outlined issues from above:

The rationale for our encoding is different -- it is to represent arbitrary
sequences of octets in a form that is as convenient as possible for human
users to manipulate. [...] Anticipated uses of these URIs include cut-
and-paste, text editing (e.g. in HTML files), manual transcription via a
keyboard, manual transcription via pen-and-paper, vocal transcription over
phone or radio, etc.

Here are some examples from that doc:

#bits base-2                           base32     base64     z-base-32
----- ------                           ------     ------     ---------
1     0                                AA======   AA==       y
1     1                                QA======   gA==       o
2     01                               IA======   QA==       e
2     11                               QA======   gA==       a
10    0000000000                       AAAA====   AAA=       yy
10    1000000010                       QCAA====   gIA=       on
20    10001011100010001000             BC4IQ===   CLiI       tqre
24    111100001011111111000111         6C74O===   8L/H       6n9hq
24    110101000111101000000100         2R5AI===   1HoE       4t7ye
30    111101010101011110111101000011   HVK66QY=   PVXvQw==   6im5sd

Here for comparison a 256 bit sequence to give a feel for the length of the result:

ASCII:      0123456789012345678901
base64:     MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMQo=
z-base-32:  gyaurc3wgw5dqqb3gyaurc3wgw5dqqb3gyao

So it would be slightly longer, but more robust to both human and machine errors. There is a JS implementation that is only about 4 KiB (non-minified).

Thoughts?

[HLeithner]

Can we put this discussion a bit forward? Its really a pain sending links per email to outlook users and get feedback that they can't open it...

Is there a encoding standard with a-zA-Z0-9 ?

Maybe we could at least guess a "=" or "==" if decoding fails?

[elrido]

hexadecimal, z-base32? Both will make the URL longer, hexadecimal much more then the latter.

[HLeithner]

z-base-32 seams lowercase only.

Base58 seams much better for the use case, it only ALNUM without similar looking characters, so it could be easily printed.

https://en.wikipedia.org/wiki/Base58

[elrido]

ASCII:      0123456789012345678901
base58:     E5djjChETuxcW5K1t3tmTiLSb3vQRW
base64:     MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMQo=
z-base-32:  gyaurc3wgw5dqqb3gyaurc3wgw5dqqb3gyao
hex:        30313233343536373839303132333435363738393031

So it is shorter then base64 and hex. z-base-32 deliberately supports upper and lower case characters to allow for mistakes in spelling.

In base58 the same characters in lower and upper have different meanings, same as base64, which may be a source of ambiguity. base58 avoids the ambiguity of 0 (zero), O (capital o), I (capital i) and l (lower case L).

It is shorter then base64 and z-base-32, as well as hex (which we use for the paste ID and could replace as well). No weird characters, similar upper/lower limits as the current base64, so over all an improvement over the status quo. I like it!

There is a 117 lines of code JS implementation of this at https://github.com/cryptocoinjs/base-x

Aside: The "base 58 check" flavour used for bitcoin addresses adds a checksum and formatting and is therefore longer, so I would suggest to use this.

[HLeithner]

I implemented the base58 and created a pull request. This PR is backward compatible with old keys because the it only effects new keys. We don't decode old symmetric keys.

You can test this at https://privatebin.at

[elrido]

For future reference: Base58 will strip leading NULL bytes (0x00) from the string. This leads to shorter encoded strings if there happen to be random NULL bytes generated at the beginning of the string.

Property based unit tests failed due to this when they happened to generate such input strings, so the tests were adjusted for this. I have to retest if this will cause decryption to fail if the original string is used for encryption and the shortened key is used in the decryption. I probably have to prepend NULL characters if the string is too short (our keys are always 256 bits long) to avoid this.

[elrido]

Difficult to reproduce in practice, but I managed to capture a case where the 256 bit key happened to start with a NULL byte and sure enough the decryption of that paste failed.

Since I don't want to reject such keys and regenerate new ones (that would reduce the amount of possible keys and be a weakening of the crypto) I instead added a left padding of the key with leading NULL bytes, if it happens to be less then 32 bytes (=256 bits) long after the base58 decode.

[HLeithner]

That explains why I got a really short token on my first test.