PrivateBin doesn't automatically delete the files of expired pastes from the file system

[whalehub]

Steps to reproduce

1. Create a paste with a short expiration time (e.g. five minutes).
2. Wait for the paste to expire.
3. Check the file system to see if the paste's file still exists.

What happens

The expired paste's file still exists on the file system after it has expired. It is only deleted when someone tries to access it after it has expired.

What should happen

The expired paste's file should be deleted once it has expired, independent of people trying to access it after it has expired.

Additional information

I'm not 100% sure about this, but I could have sworn that this is how PrivateBin worked prior to v1.3.4.

The reason why I think this is because my PrivateBin instance has a maximum paste duration of 24 hours and I have a cron job scheduled to delete empty folders from PrivateBin's /srv/data folder every hour.

Until I upgraded to v1.3.4, the /srv/data folder was always empty after waiting 24 hours for all pastes to expire, but now the files of expired pastes continue to exist on the file system until someone tries to access them from their browser.

Basic information

Server address: N/A

Server OS: Debian 10

Webserver: Caddy v1.0.5

Browser: Chrome v80.0.3987.158

PrivateBin version: 1.3.4

I can reproduce this issue on https://privatebin.net: No (due to lack of access to their file system)

[rugk]

There is an integrated "cron"-like mechanism that periodically (at each paste access after some time) checks some pastes to find deletes ones.

See https://github.com/PrivateBin/PrivateBin/wiki/Configuration#purge

[whalehub]

@rugk Thanks for the pointer!

[ZerooCool]

I looked on my side, very interested to see the purge mechanism.
I don't know when it actually happened, but today, following my tests yesterday, I can see that the messages created are no longer present.

On the other hand, there are always the two subfolders !!!
So, we always have a trace, with two useless files, that one day we will have to save! I find it a bit annoying, right?

Example:

/ data / ef / a2 / message

purge ok

/ data / ef / a2

Another question I ask myself!
As administrator of the server, is it possible or not to consult the contents of the messages?
The mechanism would like me not to be able to, especially if these are private messages.
Unless I can see the messages, unless there is a password?

How to manage messages that would be checked as never to be deleted?
What happens if a user copies the Larousse encyclopedia several times in a message and asks to keep the message? So, the solicity storage space will increase, without knowing what is actually stored by the users, and, why !? Can you provide more information on the method of administration?
Should you ban storage for life, to ensure that the message can one day be purged?

[elrido]

Hello @ZerooCool, and welcome to the project.

As administrator of the server, is it possible or not to consult the contents of the messages?

You can see the JSON contents of the pastes and comments, including some meta data that isn't exposed in the API, but the paste and comments themselves are encrypted and you (should not) know the key, as that is randomly generated by the paste creator. The various properties in the JSON string are explained in the wiki page on the encryption format.

How to manage messages that would be checked as never to be deleted?

You have to ask yourself if you want to allow such messages or not. If you allow it, you just have to bring sufficient disk space to handle them. To give you a ballpark figure - the privatebin.net demo instance did allow unlimited pastes for a few years, but the default was set to a week and the size limited to 2 MiB. The total disk space used never grew above 1 GiB. This means that we had only ever up to 500 pastes in total on that instance. With 2 MiB it is unlikely that a whole encyclopedia gets uploaded, but of course a malicious user can use scripting and one of the CLI clients to upload a large number of pastes. Hence we also have a per IP rate-limit set up and only one paste every 10s can be created. All of these values can be adjusted in the configuration file to suit your needs. The defaults are a good start and if unsure I would recommend to just monitor disk usage and bandwidth of your server, if you don't do so already.

Finally, something that can happen, when you run a public instance that becomes popular, is that someone uploads something that isn't big, but illegal in some jurisdiction. You may then get emails or letters from lawyers, government organizations or law enforcement agencies (they will easily get your contact information via your domains whois entry or your providers abuse service) requesting you to take down a particular paste. As you already found out, given a URL containing the pasteID, even without a key or password, it is easy to find the corresponding file in the data folder and it's sub directories and delete it. I usually just use this command on the Linux VM in question (executed within the data directory, as a user with write access):
find [insert path to privatebin data here] -name [insert paste ID here]* -delete

Keep hacking that planet! ;-)

[ZerooCool]

Ok thank you for your response.

In a spam issue, I suppose it would be interesting to have a tool, allowing you to clean up a batch of messages, written by the same IP.

But, that, I suppose that it does not exist yet, and, that it would be necessary to test with use, to see if it could be used.

[elrido]

We don't store the IP or rather it's salted hash of the paste creator in the paste. This increases their privacy and also decreases your risk of being held accountable, but makes this task difficult. You would have to find the IPs that send lots of POST requests in the webserver logs and then correlate their request times with the paste creation dates. Actually in that scenario it might be easiest to just delete any pastes created in a certain time frame in bulk.

In practice I do occasionally have had issues on my instances with spammers, but mostly due to their bandwidth usage, slowing all of my services down, not disk space usage. I usually set up a temporary rule to block packets from the subnet that IP is in, as deduced via IP whois query, on the router in front of the server cluster, so they can't just bounce their IP by rebooting their router or switch to a different cloud instance. This causes them to still send packets, but normal TCP behaviour in the routers between them and mine will throttle these requests, as there will no longer be any replies from my end. This binds their resources (their TCP connections will hang) and at least reduces the upstream bandwidth (they usually send you lots of small pastes to reduce their own bandwidth need, so the replies are larger then the POST requests themselves).

This of course wont help in case of a more sophisticated DDoS attack where they'd use custom TCP handlers that don't wait for replies and circumvent the IP or subnet based blocks via wide distribution, but that would be something to escalate to my provider as they can mitigate to some extent on the BGP level.

[ZerooCool]

You would have to find the IPs that send lots of POST requests in the webserver logs and then correlate their request times with the paste creation dates.

Yes, it"s my mean.

Actually in that scenario it might be easiest to just delete any pastes created in a certain time frame in bulk.

Hum, Yes but, No. This solution can delete a good message.

I create a new issue, for that.