Bootstrap-select: Large lists and virtualization
I'm running into performance issues with lists larger than ~4,000 items. Are there any plans for supporting very large lists? Either through virtualization, like http://bl.ocks.org/jasondavies/3689677. Or forcing a search to narrow down the result so there aren't too many items. I suppose this would also require bootstrap-select to be based off of JSON rather than a
ataft
All 71 comments
This is also an issue for me. Would love to see it resolved.
Nedlinin
on 4 Mar 2016
I do have plans to implement this in an upcoming update.
caseyjhol
on 4 Mar 2016
+1
stewkuhlo
on 7 Jun 2016
I just added this feature in the virtual-scrolling branch. It's still a WIP, and there are a few features missing, some I'm aware of (the startsWith feature for example), and some I'm sure I'm not. Try it out - I'm looking for feedback.
caseyjhol
on 9 Mar 2017
Wow, this is so great, thanks! I've checked it out on some large lists, and the performance is excellent!
Here are some items from my initial look at it:
1) As I scroll, the listbox width shrinks and expands to fit the virtual content
2) Items are not showing in my list, but they show up when I search. This is probably because I'm loading values when the user opens the list ('show.bs.dropdown') and then calling selectpicker('refresh') when they're loaded.
3) Performance is excellent up to 100-200K items, which is more than enough. The select list with all the options is still in the DOM, which I suppose is a requirement for bootstrap-select to work, but if the list could be based off a JSON object rather than a select then there'd be no limits.
Excited for it to be released. Thanks again!
ataft
on 9 Mar 2017
Can you link me to an example (JSFiddle or Plunker) of your setup so I can troubleshoot and see exactly what you're referring to? I can't recreate issue 1 - what web browser are you using?
caseyjhol
on 9 Mar 2017
I was struggling with ajax-bootstrap-select due to performance issues. With the given modifications it works as charm. But minor issues.
- Undefined option is shown while searching
- Already cached data is not shown
Checking the code to resolve the same.
Thanks a million
longlivetmj
on 9 Mar 2017
I'm using Chrome 56.0.2924.87, but the same issue occurs in Internet Explorer 11 too. In IE, the handle on the scrollbar stays throughout scrolling at least, but in Chrome it jumps around like crazy.
Here's a JSFiddle: http://jsfiddle.net/andytaft/54m1ybzn/1/
I had to copy the raw source into the fiddle as I wasn't sure if/how to link to the github source (JSFiddle wants a CDN)...
ataft
on 9 Mar 2017
Thanks - able to recreate it now. Working on a fix. Just updated it with some more performance improvements (opening the menu with a lot of options was previously very slow).
caseyjhol
on 9 Mar 2017
The issue with the width has been fixed. Working on the disappearing option issue.
caseyjhol
on 13 Mar 2017
I tried this out, but it doesn't seem to work with optgroups yet?
andrews05
on 14 Mar 2017
Give it another shot now - there was a bug in my code preventing it from working. Looks like there are still some issues using the arrow keys to navigate the menu with optgroups.
caseyjhol
on 14 Mar 2017
Yup, that fixed it!
New issue is when I search I get duplicate results. Every match it finds is added again at the end of the list.
andrews05
on 14 Mar 2017
Noticed that too...working on a fix.
caseyjhol
on 14 Mar 2017
Search should work properly again - still working on getting the arrow keys to navigate optgroup menus properly.
caseyjhol
on 15 Mar 2017
Thanks! Something I think could be really helpful in managing such large menus would be an ability to show just the checked items, so you can see at a glance which ones are checked.
andrews05
on 15 Mar 2017
Next issue: the checkmark doesn't get toggled when you select an item while searching.
andrews05
on 15 Mar 2017
Made some more changes, and arrow key navigation is better, but still not close to being ready. Let me know if any of your issues persist or if you notice anything else.
caseyjhol
on 16 Mar 2017
Checkmark works correctly now, thanks.
Initialisation speed is definitely improved over previous version but can still be slow. Do you think there are any further optimisations that could be made here?
andrews05
on 20 Mar 2017
Made another update - keyboard navigation should work correctly now. I still need to clean up some of the code/variables, and I do have plans for more optimizations. How many options are in the menu you're testing? What kind of performance are you seeing currently?
caseyjhol
on 23 Mar 2017
I've got ~6000 options and initialisation time is measuring ~1500ms. This is only slightly faster than the previous version but the time for the intial render (harder to measure) is noticeably quicker.
Dividers are causing problems with the scroll position. Possibly because they have margin rather than height?
andrews05
on 23 Mar 2017
Is that using the latest version I just released tonight? The divider margin issue should be fixed. What browser are you using? In the latest version of Chrome, 5000 options takes between 200-300ms for me. 50000 options is around 2.5 seconds. I've still got ideas to improve performance too.
caseyjhol
on 23 Mar 2017
I'm using the Safari Technology Preview. Upon testing the standard release of Safari, it works fine! Must be an issue with the current tech preview, sorry about that.
I still have an issue with dividers in the latest release though, tested in Chrome also.
Searching with optgroups seems to be causing errors now, but not always - not sure of exact circumstance.
andrews05
on 23 Mar 2017
Can you link me to an example of your use case? Having trouble recreating that in my test environment.
caseyjhol
on 23 Mar 2017
http://jsfiddle.net/54m1ybzn/5/
I've randomised the strings here but its otherwise identical to the one I'm actually using.
The fiddle seems to have a problem with scrolling for some reason, so you can't see the divider issue here, but you can see the errors with optgroups.
andrews05
on 23 Mar 2017
Thanks - the optgroup search issue should now be fixed.
caseyjhol
on 23 Mar 2017
Great. Still trying to create a test case for the dividers but in the meantime I identified the biggest cause of my slow initialisation, affecting both Safari and Chrome: On line 795 if you change $this.index() to $this.prev().length it makes a huge difference, since one of my optgroups has 4k options in it.
andrews05
on 23 Mar 2017
So there's another issue where the scrolling doesn't work within an iframe, which is why jsfiddle was causing me problems. Anyway, I've attached a file here demonstrating the divider issue.
biglist.zip
Honest question: Do you think this whole virtualisation thing is worth the effort and added complexity? Especially given the greater potential for problems in unusual situations. I'm starting to think that huge menus like this are simply outside of the scope of this plugin and would be better served by a different approach.
andrews05
on 24 Mar 2017
I think it's probably worth it, if not just for the performance improvements alone when loading a page with multiple large selectpickers.
caseyjhol
on 24 Mar 2017
It's hugely important for my projects. Generally, select list DOM limitations have been causing me grief forever, especially in cases like @caseyjhol mentions where there are multiple on one page. This feature will set bootstrap-select apart from the others.
Thanks for your hard work on this!
ataft
on 24 Mar 2017
@ataft, are your issues more with initialisation or with displaying/scrolling the menu?
In my case I've decided it will be more usable if I require the use of the search to find items, displaying only the checked items when not searching so you can easily see what's selected. It would be highly unlikely have huge quantities of items selected at a time, so displaying/scrolling isn't really an issue.
This may not be suitable for everyone, but I think some sort of limiting on the number of items displayed at a time may be more appropriate than being able to scroll through thousands of items at high speed.
Certainly the initialisation speed can be improved without requiring virtual scrolling. I'm just worried that the virtual scrolling may end up causing more problems than it solves.
andrews05
on 24 Mar 2017
Keep in mind this is in the very early stages. Once I get the virtual scrolling set up properly, I don't think there should be any additional issues. Also, this will be key for ultimately allowing dynamic options.
caseyjhol
on 25 Mar 2017
Okay, I'll trust you :)
In the meantime, I've submitted a PR for some init improvements for the current version. Are you planning to do another release before you finish virtual scrolling?
Btw, what's your target browser support?
andrews05
on 26 Mar 2017
Virtual scrolling is still a ways out from being available, so there might be another release before it's ready. I'd like to target IE8+, but I'll see how feasible that is. Right now, the virtual scrolling only works good in Chrome, so I've got to make some optimizations to get it working in Firefox and IE. Also, the plan will probably be to only do virtual scrolling if there are 100+ options, and I'd like to make it optional. That way it will be easy to fall back if for whatever reason there's a browser where virtual scrolling doesn't work great it.
caseyjhol
on 27 Mar 2017
Sounds good. I can tell you the current version requires at least IE 9 though, due to use of things such as previousElementSibling.
andrews05
on 27 Mar 2017
Hi,
I tried to use the new version of your script in my angular2 project and webpack complains can't read position of undefined. That's happening in the scroll() function. I've found after small investigation in lines 597-598
var size = that.viewObj._currentLis.length;
var rows = 2 + Math.ceil(that.sizeInfo.menuInnerHeight / liHeight);
that.viewObj.position0 = Math.max(0, Math.min(size - rows, position));
that those vars have undefined or NaN values and later in the code top and bottom margins can't be set.
Beside this exception in the console, the script is running and everything works properly. My guess is the list of elements is empty yet.
Fellach
on 12 May 2017
Hi @caseyjhol,
when are you planning on merging this feature into release? I've tried to use in production, but scroll method works in Firefox only for first two items, then the list jump to the second position always. But probably you know that already, as you've written it works only in Chrome.
The issue doesn't occur in the version from this fiddle: http://jsfiddle.net/andytaft/54m1ybzn/1/ (pasted here: #issuecomment-285423147)
Thanks in advance for you reply.
Fellach
on 25 May 2017
Not sure yet when this will be released. I've been working on a new method of virtualization, since the current method works so poorly in Firefox and IE (or, rather, doesn't work at all).
caseyjhol
on 25 May 2017
Virtualization is way over my head, but ag-grid (https://www.ag-grid.com/) implements it in one of the best ways I've seen. It works incredibly fast in Firefox and Chrome, and very well in IE.
Excited for the featue! Thanks again for bootstrap-select!
ataft
on 26 May 2017
@andrews05 @ataft @Fellach I've made some changes. It now works in Firefox, as well as IE11+. Will add fall back code for older versions of IE later. Let me know what issues you're encountering now. Still have more optimizations planned to improve page load.
caseyjhol
on 29 Jun 2017
I'm able to get it to work in this (http://jsfiddle.net/54m1ybzn/7/) fiddle, but I'm getting an error in my application. Perhaps it has something to do with my app using Backbone, but I'll need a little time to dig into it further. Or, perhaps it's something that you are able to diagnose easily. Here is the error:
Uncaught TypeError: Failed to execute 'appendChild' on 'Node': parameter 1 is not of type 'Node'.
at Selectpicker.liHeight (bootstrap-select.js:1115)
at Selectpicker.setSize (bootstrap-select.js:1188)
at HTMLButtonElement.<anonymous> (bootstrap-select.js:1562)
at HTMLButtonElement.dispatch (jquery.js:4435)
at HTMLButtonElement.elemData.handle (jquery.js:4121)
at Object.trigger (jquery.js:4350)
at HTMLButtonElement.<anonymous> (jquery.js:4901)
at Function.each (jquery.js:374)
at jQuery.fn.init.each (jquery.js:139)
at jQuery.fn.init.trigger (jquery.js:4900)
liHeight @ bootstrap-select.js:1115
setSize @ bootstrap-select.js:1188
(anonymous) @ bootstrap-select.js:1562
dispatch @ jquery.js:4435
elemData.handle @ jquery.js:4121
trigger @ jquery.js:4350
(anonymous) @ jquery.js:4901
each @ jquery.js:374
each @ jquery.js:139
trigger @ jquery.js:4900
jQuery.fn.(anonymous function) @ jquery.js:7487
(anonymous) @ filter.js:300
I just pushed out an update and that bug should now be fixed.
caseyjhol
on 29 Jun 2017
Thanks! Now my app is not showing the values when I open the selectpicker, but when I enter a search value they show up. I created a fiddle with the latest commit, but the values load fine in this example. If you are unable to diagnose the problem easily, then I'll set up an example more similar to my app to try and troubleshoot.
Also, I found a bug when you open the selectpicker, close it, then try to re-open it. Check out this fiddle and you'll see the following error:
Uncaught TypeError: Cannot read property 'cloneNode' of null
at scroll (bootstrap-select.js:668)
at Selectpicker.createView (bootstrap-select.js:586)
at HTMLInputElement.<anonymous> (bootstrap-select.js:1829)
at HTMLInputElement.dispatch (jquery.js:4435)
at HTMLInputElement.elemData.handle (jquery.js:4121)
scroll @ bootstrap-select.js:668
createView @ bootstrap-select.js:586
(anonymous) @ bootstrap-select.js:1829
dispatch @ jquery.js:4435
elemData.handle @ jquery.js:4121
It seems that the data-actions-box is causing some problems, as shown in this fiddle:
http://jsfiddle.net/54m1ybzn/10/
This issue doesn't appear to be related to my issue from the previous post, as I removed the data-actions-box from my app and it still didn't work...
ataft
on 30 Jun 2017
The issue with opening/re-opening the select has been fixed. You're running into a conflict in your JSFiddle because you're also loading bootstrap-select 1.5.4 JS/CSS as external resources. Removing those causes the menu to appear normally.
caseyjhol
on 30 Jun 2017
Doh, sorry about that. And thanks for the fix!
I figured out what my issue is. I'm loading the values on 'show.bs.select' and then calling selectpicker('refresh') when they are loaded. This doesn't appear to work with virtualization while the selectpicker is open. See this fiddle: http://jsfiddle.net/54m1ybzn/11/. You should see "Three" appear in the list when you open it, but you have to close re-open it to see the new value. Also, you'll notice that the checkmark doesn't show up when selecting values by clicking or using "Select All".
ataft
on 30 Jun 2017
@ataft The issue with refresh should be fixed now.
caseyjhol
on 3 Jul 2017
This is great! Initial testing looks good in my app, but I'll continue to bang away on it. I've put together a fiddle on the most recent version with 40K values in case others want to test: http://jsfiddle.net/54m1ybzn/12/
One very minor thing, you'll see that the "selected-text-format" only goes up to 100 Items Selected when you select all the items...
ataft
on 4 Jul 2017
Hi @caseyjhol,
I've found the & is encoded to entity & in the option's name. Is it intentional?
Fellach
on 18 Jul 2017
@Fellach That wasn't intentional (a remnant from the previous method of using $.html() to get option content). It should work now.
caseyjhol
on 18 Jul 2017
Confirm, works like a charm. Thank you
Fellach
on 19 Jul 2017
Available in v1.13.0-alpha.
caseyjhol
on 29 Jul 2017
Works pretty good so far, great job! Only thing I noticed was the search box isn't focused automatically when you open the menu.
andrews05
on 31 Jul 2017
@andrews05 Hrmmm, it's focused in all of my testing. What browser/OS are you using?
caseyjhol
on 31 Jul 2017
Tried it again and it works. Sorry, must have had something wrong the first time.
andrews05
on 31 Jul 2017
So far the alpha release is looking good. I've run into an issue when doing a refresh on an open list. Check out this fiddle: http://jsfiddle.net/54m1ybzn/14/. Open the list, do not scroll, and select an item. You'll notice that the checkmark does not appear until you start scrolling. This occurs in both IE 11 and Chrome 59.0.
I'm having a couple issues specific to IE 11 as well, but I'm not able to reproduce them in a fiddle, only in my app. Here are the problems:
- I get an "Unable to get property 'liHeight' of undefined or null reference" error (Line 613: li.height = this.sizeInfo.liHeight;). It only occurs when I have two selectpickers on one page, one with values selected, the other with no values selected. Then close the one with nothing selected, and then re-open, then the error occurs. It was difficult to reproduce, but I can do it every time now that I know the exact sequence of events.
- In IE and only in my app (not the fiddle), I have to click an item twice for it to be selected (display above and show the checkmark). Scrolling and then selecting an item causes it to jump to the top of the list, not selecting the item because I'd have to click it twice in order for this to happen.
If you're not able to identify the IE-only items above, then I'll work towards getting a working fiddle that can show them.
ataft
on 4 Aug 2017
@ataft I was able to reproduce your first issue, and I know what the problem is (setSize is only called when calling refresh while the menu is open, otherwise just liHeight is called, as setSize will be called anyway next time the menu is open). However, if calling refresh on show.bs.select, the open class hasn't been added yet (not until shown), which is causing that particular issue. I'll see if there's a better way to check, or if I can get away with just always calling setSize.
I wasn't able to recreate either of your IE11 issues.
caseyjhol
on 4 Aug 2017
@caseyjhol Thanks for looking into it. I've tested and believe that I can use shown.bs.select instead of show.bs.select, so no need for you to do anything. It worked OK in prior versions, so you may want to address the issue just in case, but I don't need you to. Thanks!
Also, I figured out what my IE issues were. On show.bs.select, I was setting the placeholder attribute on the search box to "Search...", as shown below:
In IE, the first click on an item was taking the focus from the search box, then the second click was selecting the item. Since the search is always focused (was it always this way?), I don't need the placeholder there anyways. I had it there originally because my app users are, let's just say not very web-saavy, and wouldn't necessarily know that that neat little box at the top was for searching.
Thus, I have no open issues with the virtualization. It's really a great feature, thanks so much!!!
ataft
on 5 Aug 2017
I'll check into it - I'd like to keep behavior as close as possible. You should be able to set the placeholder using liveSearchPlaceholder, instead of setting it manually on show.
$.fn.selectpicker.Constructor.DEFAULTS.liveSearchPlaceholder= 'Search...';
Aha, it's the placeholder itself that's causing the issues in IE. Check out the following fiddle using the liveSearchPlaceholder code (using IE of course):
http://jsfiddle.net/54m1ybzn/18/
Also, the "Select All" function on a medium-sized list (~2000 values) is instant in Chrome, but takes a few seconds in IE. Is there anything to be done here? It's crashed my IE browser a couple times too. I despise IE like everyone else, but unfortunately it's the preferred browser at many enterprise-level organizations (probably for legacy apps)...
ataft
on 5 Aug 2017
Hrmmm...that might be related to #851. Although, I noticed that Internet Explorer implements placeholder differently than other browsers - once the input is focused, the placeholder is hidden (rather than remaining visible until there is text in the input). The input is always focused to allow users to start typing right away, so the placeholder is never visible.
I'll look into the performance issues with Select All in IE.
caseyjhol
on 6 Aug 2017
I found one more IE issue, but it's impossible for me to reproduce in a jsfiddle for some reason. When I scroll all the way to the bottom, the items all disappear, as shown in the attached "before" and "after" images. In IE, I noticed that the .dropdown-menu.inner "margin-top" is being set to 0 when I reach the bottom, but that in Chrome that doesn't happen. It's been very difficult to troubleshoot and it's somewhat sporadic, but I'm hoping maybe it's an easy fix for you. Seems like it's in the code around here:
https://github.com/silviomoreto/bootstrap-select/blob/virtual-scrolling/dist/js/bootstrap-select.js#L724
ataft
on 2 Feb 2018
@ataft Please check out v1.13.0-beta and let me know if the problem persists. https://github.com/snapappointments/bootstrap-select/releases/tag/v1.13.0-beta
caseyjhol
on 14 Feb 2018
Looks great, thanks again for your quick response and awesome library!
ataft
on 14 Feb 2018
@caseyjhol wanted to say thanks here, ran into performance issues on IE11 , this fixed it
tomasfakta
on 21 Feb 2018
Released in v1.13.0!
caseyjhol
on 19 Apr 2018
thank you for your amazing work! i'm a bit confused about how to get the selected options(multiple mode) once the submit button is clicked.is there a specific way or a function to do that ?
HoudaMessaoudi
on 18 Feb 2019
There seems to be an issue as of 1.13.9. If the dropdown goes into virtualization mode (from having many options), it adds a margin-bottom. If you update the dropdown to contain fewer options, thus smaller than the virtualization threshold, the margin-bottom stays, so you just get this huge gap at the bottom.
Easy fix at line 1265:
} else {
menuInner.firstChild.style.marginTop = '';
menuInner.firstChild.style.marginBottom = '';
}
dliebner
on 9 Oct 2019
@dliebner That issue was fixed in v1.13.10.
caseyjhol
on 14 Nov 2019
Not Worked for me,
any other possibilities?
cesarhfborges
on 30 Jul 2020
Not Worked for me,
any other possibilities?
Late reply, but if you still need it:
in bootstrap-select.js, find the line with the if statement if (isVirtual === true) and add the else to that statement. The line number may be different if you've modified the file.
McKinleyWC
on 15 Mar 2021
Related issues
spyhunter88
路
4Comments
dacavcosta
路
3Comments
dw00001
路
3Comments
ddinchev
路
3Comments
neuropass
路
3Comments
Most helpful comment
I do have plans to implement this in an upcoming update.