Browser-laptop: History is missing entries (only shows last 500)

[luixxiul]

Describe the issue you encountered: Entries on about:history are automatically disappearing from session-store-1, though the option to clear history is not enabled. The history of bookmarked pages are retained.

Use the browser for days. Open more than one hundred pages like daily use. For me the browser does not retain entries earlier than 4 days.

If you use the browser daily, open about:history and check if entries dated earilier than 9/11/2016 (as of today) were retained. Taking a screenshot would help to verify the bug.

Expected behavior: The entries should be retained forever unless they are set to be removed or removed manually.

• Platform (Win7, 8, 10? macOS? Linux distro?): both Windows 10 and macOS

• Brave Version: 0.12.8 on Windows 10, master branch on macOS

• Any related issues:

From a perspective of a user this seems to be a serious bug and should be fixed as soon as possible. Definitely by 1.0.

[alexwykoff]

Appears to be the same in 0.12.8 on OS X, worth some degree of investigation.
cc: @bsclifton

[srirambv]

User Randy Waffle has also reported this issue.

When I look in the search history for something I viewed the day before
there's nothing there. The history is only listed for 7 days ago and
older. I don't see any search history from the last 7 days.

[luixxiul]

@srirambv I think it is similar but different because

The history is only listed for 7 days ago and older

I believe that issue is rather related with #4224 or #5072

@bsclifton Is this issue #5590 fixed with #5600?

[bsclifton]

This is a problem that we're tracking a fix indirectly with https://github.com/brave/browser-laptop/issues/1864

Here's how bookmarks and history works in a nutshell (@bbondy, @darkdh, or others- please correct me if I'm wrong). It's important we all understand this. We can then create issues (or close existing issues) as needed; please help me with this :smile:

cc: @alexwykoff @srirambv @luixxiul @bradleyrichter

• Bookmarks, bookmark folders, history, and pinned tabs are all the same object, a SiteDetail
• SiteDetail is made up of:
  
  
  
  • title _the page title, as set by the web page (not editable)_
  
  
  • customTitle _folder name or name given by user_
  
  
  • location _the URL; used for all except folders_
  
  
  • folderId _only used for folders_
  
  
  • favicon _the URL for the favicon_
  
  
  • parentFolderId _ID of what folder it's in; if in the bookmarks toolbar root it's 0. If in "other bookmarks", it's -1; otherwise ID of folder_
  
  
  • lastAccessedTime _last time site was visited_
  
  
  • count _number of times this site has been visited_
  
  
  • tags _this is an array, which is initially empty. The tag is like "bookmark" for bookmark, "bookmark_folder" for BM folder, "pinned" for a pinned tab, and empty for history._
  
  

Notes about usage:

• When page is visited, the lastAccessedTime is updated. This is the "history". There's only one entry, so unfortunately visiting a URL will remove the last visit since it'll update the timestamp with the current visit.
• When a bookmark is added, a bookmark tag is added.
• When page is pinned, the pinned tag is added

So the problems with this:

• a given URL can only ever have 1 history item. The appearance of more is likely because of visiting different URL fragments. For example:
• https://brave.com/
• https://brave.com/#faster
• https://brave.com/#safer
• https://brave.com/#devs

These technically make the same HTTPS request but differ because of the hash (and the fragment that follows it). These will jump you to an anchor tag (and in certain frameworks are sometimes used for routing). The above example would count as 4 entries. However, if you have brave.com bookmarked and you visit it once today and once tomorrow, you'll only see the one for today.

[luixxiul]

Thanks @bsclifton for explaining :-)

When page is visited, the lastAccessedTime is updated. This is the "history". There's only one entry, so unfortunately visiting a URL will remove the last visit

given URL can only ever have 1 history item.

I notice an entry which has been visited only once disappeared too.

[bsclifton]

@luixxiul also I forgot to state- we only show the 500 most recent entries. However, the logic for this may have been wrong and was possibly fixed with https://github.com/brave/browser-laptop/issues/5595

Can you please recheck this? Thanks 😄

[luixxiul]

OK, I see.

we only show the 500 most recent entries

Do you mean that the entries earlier than that are erased?

[bsclifton]

@luixxiul entries earlier than that do exist, but aren't shown. We can definitely show all of them, but it may impact performance

[luixxiul]

how about pagination?

[bsclifton]

@luixxiul that sounds really good! Would you be able to describe your expectations in more detail by creating a new issue (and mention Brad in there?) :smile: I think this a great idea (especially needed once we have a true history). I know he would like to refine the screen further, by only having one table (instead of one table per day)

[NejcZdovc]

@bsclifton @luixxiul should we increase this, now that we will have split sites, otherwise I would suggest that we add pagination, but not as numbers but as infinite scroll. This means that when you scroll and you are on let's say entry 400, we get history sites from 500-1000 and append them to the list, so that user don't see and of list with a normal scroll. But if you scroll fast we would just display a loader and a text "We are loading history for you".

[bsclifton]

@NejcZdovc definitely we should increase / not have a limit (now that they're being split) 😄

[luixxiul]

I almost forgotten about the initial issue, but I reported with this:

Entries on about:history are automatically disappearing from session-store-1

So I assume the history has not been just hidden but cleared.

@NejcZdovc does the PR you're working on, #10136, fix the issue as well?

[NejcZdovc]

@luixxiul I don't have any clear functions in there, but we have hard limit there were we only display 500 history entries. But we still keep all record past that number in session.

[bbondy]

I'm not sure we should increase, I think just moving to chromium history instead. I worry it'll introduce perf problems.

[bsclifton]

cc: @kevinlawler who is implementing the history API

Once that work is implemented and we're integrated with it, we can likely close this issue

[kevinlawler]

We are currently storing Chromium history in parallel without exposing it. Chromium stores the last 90 days of history. This is different in some ways than the last 500 entries, which is what we currently are offering through the browser-laptop side. As @bbondy points out, removing the 500 cap raises concerns about performance problems.

Possibly the right thing to do is to switch over to Chromium while instituting a "new" 500 entry governor (via LIMIT), then look into removing the cap afterwards. There is a strong likelihood this handover will be complicated, in particular on the UI side: the rationale behind this is that 500 entries can effectively be stored in memory without penalty, and rendered also more or less without penalty (in particular, while I haven't confirmed this is actually the case, it would be possible to avoid entirely building any paging or infinite scroll functionality). My 90-day history, for instance, has 10K or 44K rows, depending on how you want to count it, which is 20x or 80x, and there are sure to be users with more intense access patterns than mine, meaning there's a very real possibility we could run into performance issues using the existing arch.

The worry then is that a table scan on 90-day SQLite history will be cost prohibitive. If, for instance, we want to display history results in an infinite scroll, and we want to select LIMIT,OFFSET pages from SQLite, then any of that that doesn't exist will have to be built, and these are sizeable sub-projects.

Another possibility to consider might be re-skinning whatever Chromium exposes as a history page. Possibly they either already have what we need or have built some of these components that we can reuse. This is easier to describe, but it isn't necessarily the case that the integration is substantially easier than refitting the existing history page to work through a different API.

[bbondy]

I think we should expose all of the chromium extension APIs for history and then re-build the about:history page and leave the sites history alone for now since it's used in various other places. As a separate task start to move over other things as we can.

[bsclifton]

We'll get this functionality by default when we switch to brave-core, so I don't think it's worth the time investment on browser-laptop. Closing as wontfix