Freshrss: Maximum number of articles to keep by feed

[Wanabo]

I'm missing an option to restrict the max number of articles to keep by feeds!
There is however a "Minimum number of articles to keep by feed"

Please add an option for "Maximum number of articles to keep by feed" or change "Minimum number of articles to keep by feed" to "Maximum number of articles to keep by feed"

Pro's:

• limit database growth
• reduce number of articles like frequently updated traffic info, which has no historic value

[Frenzie]

Looks like this was implemented by 2a5aa34ad2796df00fa09de41361c20764ccdfa8?

[Alkarex]

@Frenzie No, that is something else, with only an impact of the RSS outputs, not the number of articles kept in database :-)

[Frenzie]

Gotcha.

[Frenzie]

Cross-referencing the related #824 about automatically marking items older than x days as read.

[aledeg]

@Wanabo have you tried the option to purge older messages? Will this work for you if we allow that setting on the feed level? We could also add other periods like days and years.
Let me know because I've started to look at that issue and I would like to finish it ASAP.
Thanks

[aledeg]

I was talking about that configuration option.

[Wanabo]

@aledeg Not sure what allow that setting on the feed level means.
Is the button Purge Now also periodically executed by cron? If not it should be!
[quote]We could also add other periods like days and years.[/quote] That is a good idea, don't forget weeks. ;)

But I like to be a little more flexible than:
purging after some period
Because not all feeds are equal! I mean feed A get's 100 messages a day and feed B get 1 message a month.
So, Minimum number of articles to keep by feed for type B feeds and
Maximum number of articles to keep by feed for type A feeds.
But perhaps these options should be moved to the individual feed settings page.

[aledeg]

@Wanabo What I meant was that the "Remove articles after" settings can be added on feed level, this way you can set your feed A with 100 messages a day to purge automatically after let say 7 days and your feed B with 1 message a day to purge automatically after 2 months.
This way the setting is not depending on the global configuration. But on the feed configuration.

I'd better go this way because it will be easier to implement with what we have. But if you think it's not suitable for your use case, let me know.

[Wanabo]

Yes that would be workable (ok). Besides the purge setting per feed I also like keep minimum and keep maximum articles per feed regardles of the purge setting. But I'm ok what you decide on this matter.

[aledeg]

I think we can proceed step by step and see if we need the extra steps along the way!

[aledeg]

I've made the first step, you can now choose the type of period. You are not limited to months.
See #2279 if you're interested in trying before it is ready for a merge.

[squromiv]

I also prefer the case, where a user can set up minimum and maximum number of articles per feed. Specially, I am interested in maximum.

[aledeg]

@squromiv Thank you for your input. If there is more people willing to have that kind of behavior, I'll go the extra mile to get it this way. I still want to include the purge parameter per feed. And also add the maximum configuration per feed.
I will still proceed step by step to validate that everything is working properly. Please follow #2279 to see the progress. I'll try to work on it regularly. It won't happen overnight though because I will be pretty busy in the upcoming weeks.

[squromiv]

@aledeg
Thanks for support. Actually, I do not understand, what is the purpose of "Minimum number of articles to keep by feed". _From a logical point of view, it has no sense_ in contradistinction to "Maximum number of articles to keep by feed".

In my opinion, good realization of such features already exists

(RSSOwl application)

[aledeg]

@squromiv @Wanabo I have a first draft in #2335. What do you think about it? Should I continue this way?
@Alkarex @Frenzie Same question for you 2?

[Frenzie]

I don't have much of an opinion on it. I suppose a use case might be to keep only 10 to 20 entries of some news website, to keep it from overflowing the feeds. However, my personal policy is that a feed that's too active is first in line for removal.

The code looks fine to me at a glance, no obvious code smells.

[aledeg]

@Frenzie Thank you for taking the time to read it!

[squromiv]

@aledeg
1) It will be great, if maximum number can take value under 100 and take a multiple of one.
2) Is it very difficult to make setting of maximum number and maximum age at the same time?

[aledeg]

@squromiv Thank you for testing it.

1. is easy to achieve. I choose to step by 100 but I can remove that.
2. I have to check if I can do both. My problem is to force to choose at least one. I have to make some adjustments and some tests.

Once we have something working properly on that level, I will duplicate the logic to have something on the feed level.

[Alkarex]

@aledeg A quick feedback. (I have not had time to read the PR yet - focusing on 1.14.1 for now). It is a good idea to keep a relatively large number of articles in each feed (50+ or even 100+ for some feeds). No doing so will make that at future refresh, some old articles will be added as unread again to the database.

I think we should always use all three parameters: minimum number of articles (as explained above), and whichever comes first of max date and max number.
Edit: and minimum number wins over max date

[squromiv]

@Alkarex What is the logical purpose of "minimum number"? As for "maximum number", this parameter helps to reduce database size and to get rid of old articles. (?)

[Alkarex]

Minimum number serves two purposes:

• a technical one, as explained just above
• and ensures that one can always see the latest articles of feeds, also of the ones with rare updates

[aledeg]

I have something working now. Testing welcomed!

[aledeg]

@Alkarex Instead of having a minimum number of article, maybe we can store something in the feed so we do not reimport entries that have been previously deleted.
I haven't checked how the id is generated but if the id is consistent through out the imports, we can store the limit id in the feed and skip the one that are older.

I am not sure if what I am saying is intelligible!

[Frenzie]

@aledeg If you have that then you're close to having the groundwork in place for a delete functionality akin to more traditional feedreaders. You'd have these entries in the database that are marked deleted (and therefore not displayed, except perhaps in trash) but they're only actually removed from the db at some later time. Perhaps also so they can be restored, but mainly to prevent stuff from continually showing up as new.

I suppose a trash would necessitate even more statuses though, where deleted from trash still isn't deleted from db. ;-)

Because my read feeds in ye olde Opera or QuiteRSS are a bit like bookmarks or Wallabag.

Delete is never having to see it again. Keeping it around as read is an archive of potentially useful stuff, but stuff that you wouldn't necessarily mind losing. Favorite is the cream of the crop that you want to keep.

In FreshRSS my archive of potentially useful stuff and the things I never want to see again are unfortunately mixed.

Basically I'm missing the negative opposite of favorite, regardless what you want to call it. And technically speaking, those neg-favs can be removed from the db in certain conditions.

[aledeg]

@Frenzie So if I understand correctly what you're saying is:

• The purge action should be a soft delete instead of a hard delete this way entries are still there but they do not appear anywhere in the application.
• There should be another script running on cron to hard delete those soft deleted entries.
• There should be an action available on entries to soft delete entries from the application.
• There should be some kind of trash where soft deleted entries could be undeleted.

Let me know if we are on the same page.

[Alkarex]

There was a mechanism several versions ago to avoid displaying old articles that are still in database, but I removed it for performance and simplicity.
The entries table is the most expensive to filter / index.

We already have favourites > not read > read, as well as the custom labels.

If we nevertheless want to add a new state (I am not sure of the need / benefit):

1. We could have more values than { 0, 1 } in is_read, e.g. { 0, 1, 2 } or { -1, 0, 1 }
2. We could make a special label for "archived" or "actually read".

[aledeg]

If we're going that way, I am not sure we need another state. I think it would be better to have a new column holding the soft delete date with a proper index on it. I am using that at work with tables with millions of rows without noticeable slow downs.
I don't want to start anything if we do not all agree on what is missing if it's missing something.

[Frenzie]

@aledeg That's an apt summary.

@Alkarex It's mainly about not wanting to keep everything around, but also not wanting automatic deletions. If favoriting were the answer, I'd have to favorite a third to half of my stuff. ;-)

[Alkarex]

Related discussion https://github.com/FreshRSS/FreshRSS/issues/2018 (maybe we should continue there, since the issue of "maximum number of articles to keep by feed" is comparatively easy to implement, without any big change, and is only one of the dimensions of the overall discussion about article states. Indeed, only some minor changes are needed to implement the screenshot of https://github.com/FreshRSS/FreshRSS/issues/1167#issuecomment-475634521 ).

Anyway, I still need to understand a bit better, both the needs, the implementation options (back-end and UI), and the interaction with APIs / mobile apps.

If we have articles that are in DB, but not shown, that is another state, right? A new column with a delete date might not be sufficient (or efficient), as we would still need to filter out those articles (fast). This is something is_read with a third state could do.

Here is a bit of brainstorm:

If we introduce a new column (especially if indexed), it should really have some added value. We could combine a number of needs. For instance (I think we have an issue on that somewhere), this column could be the read date, which would allow showing recently read articles. I would personally like this feature, but that would however not work well with the mobile apps, limiting the interest.

The minimum number of articles to keep must include all the most recent ones, read or not, marked for deletion or not.

We might also have to distinguish the duration after which to delete read articles vs. unread articles. This implies even more options.

On the UI, I am a bit concerned about how to expose those things in a simple way. For @Frenzie 's most direct need, it should be a new action to mark articles for deletion. A keyboard shortcut would be easy. But is that sufficient? I would be worried about introducing yet another icon. And here again, that is not compatible with the APIs / mobile clients.

A bit like @marienfressinaud wrote on https://github.com/FreshRSS/FreshRSS/issues/2018 I am a bit perplex and unsure of the need to introduce yet more states, given what we already have (and with what we could easily add such as max articles, max date for read articles, max date for unread articles, better shortcuts for labels). Maybe it is those existing features that should be promoted / renamed, for instance new system labels such as "read later".

With @Frenzie 's example:

• See an article to read later -> star or label
• See an article to keep -> star or label
• Unsure -> mark as unread
• Read / delete -> mark as read

Stars, labels, read/unread, are supported by APIs and mobile clients, so the more we can do with them, the better.

Thoughts?

[aledeg]

I have no answers for the use cases. I think @Frenzie have more input on that. But you don't need to be worried about performance regarding soft-delete (if done properly). As stated earlier, I use that technic very efficiently at work because we are not allowed to hard delete. And we do have table counting millions of lines.

Regarding Api, we have to keep in mind that when it is soft-deleted, it's like if it's deleted, you don't see the records. So the purging system will mark them as deleted and they will be filtered automatically.
I don't think it will change a thing on the use of clients.

[aledeg]

Regarding the screenshot, it's done in the PR I've submitted!

[Alkarex]

@aledeg I will read it :-)

[Alkarex]

@aledeg For the record, I am quite familiar with SQL optimisation and indexes, and therefore knows that everything has a cost ;-) (query time, but also insertion time and memory) The question is more on how much we believe is acceptable for a feature.
I have found again the discussion and my patch removing the filter for hiding old articles:
https://github.com/FreshRSS/FreshRSS/issues/493#issuecomment-56265675
https://github.com/FreshRSS/FreshRSS/commit/a5a0bd9a350836cb7e504fdb501c82cda37d2562
The SQL filter was more complicated and less efficient than what we could do with additional fields, but still.
Before implementing a change to our central SQL request, I think it is good to perform manual tests of the SQL performance.

[Frenzie]

Note that you can read what I wrote through the lens of "_if_ you implement some kind of hiding (and hence take whatever the performance hit may be), then…"

[Alkarex]

Implemented in https://github.com/FreshRSS/FreshRSS/pull/2335