Packagist: From Solr to Algolia

[julienbourdeau]

Context

@maxiloc met @naderman at Laracon US and they discussed the possibility of using Algolia to power the search on packagist.org. Nils said we should make a PR.

Ideally, I think it would be great to completely get rid of Solr but I'd like to do it step by step.

I'm opening this issue so we can discuss the general idea. I will open pull request as we move forward and we can keep track of what's going on here.

Algolia's Advantages

Here is a list of advantages of switching to Algolia, some are inherent to Algolia, some are just related to the PRs we'll open and could probably be achieved with Solr.

• ⌛️ Faster that the current search
• 👍 More relevant (see below)
• ✏️ Typo-tolerant (_laarvel_ query will find Laravel packages, see below)
• ⛱ No Solr setup to maintain and updates
• 💰 No server cost

For reference, Algolia powers yarnpkg.com.

Tasks list

• [x] Index packages in Algolia
• [x] Replace main search on the frontend
• [ ] Power RSS (browse page)
• [ ] Next step TBD

Help required

Packagist is not an easy project and some aspect can be hard to grasp. As a first time contributor, working on something pretty deep, I would require some help to full understand what's going on.

More importantly, I would need an extract (even just a chunk) of the databases (MySQL and Redis). I totally understand that this might feel a little weird, but in order to test the indexation properly, that would really help. I'm willing to sign anything you want if needed.

Examples

Relevancy examples

Before

After

Typo-tolerance

Before

After

[alcohol]

No server cost

This is a bit of a non-issue, since solr just runs on the same machine as packagist.org itself. @Seldaek can give you more specifics about the resources it consumes, but I am guessing they are negligible.

The search functionality could definitely benefit from improvements in regards to relevancy and typo-tolerance though.

[naderman]

@alcohol yeah the concern was really just not having to worry about solr if at all. But mainly the purpose of this would be to improve the results.

[alcohol]

I do not really have an opinion on the subject, but I would like to ask what the impact of this change would be in regards to other users being able to setup their own packagist instance. Does it makes things easier? More complex?

Also, from the perspective of the Composer organization, do we really want to hand off such an integral part of the website/application to a third party? What kind of stability/uptime guarantees can we expect? What if Algolia is experiencing issues/downtime, how can we communicate those easily? Will there be a fallback alternative?

[julienbourdeau]

Setting up a local environment will be easier because you won't have to install and setup a Solr instance. You can query the production so you have search working straight away.
Then if you work on Indexing, you can simply create an account and set your credentials.

About the uptime, I'll give you more details but I know it's excellent.

I understand the concern of handing off the search to a third party but there is no downside really. If one day you decide that you're not happy with the experience we provide, you're free to go at any time.

I'd like to point that we power search for similar projects like yarnpkg.com.

[Seldaek]

So the new search is live, indexing almost complete, I added some facetting too, looks pretty cool so far :)

I tweaked the settings a little. Kinda curious if we can still tweak to improve results or not.

Another thing I am wondering is what you meant in the initial post here by "Power RSS (browse page)"?

[julienbourdeau]

I thought the browse page (and link like rss/view all) were powered with Solr, but I might have got that wrong. I'll check again. ^^

[Seldaek]

I see, no those are not handled with solr.

Anyway looking at twitter feedback it seems we need to tweak the ranking.. Right now the popularity metric is sorted last. I don't think it should necessarily be first though, but the way we had it with solr was it was used as a rank boost but if some fulltext match happened to rank very high it would still overtake a high popularity package. We don't only want to push popular stuff to the top IMO otherwise some niche packages never get found. Is this possible?

[julienbourdeau]

Yeah I'm looking into it. We used a custom script to index the data at first, there might be something missing.

Otherwise, I think we can add the active filters widget, it might not always be obvious that a facet is active. I opened a PR #812.

[GerardSmit]

It's kinda sad that the order by is removed because of this. When I search a package I want find a package that is well used so I know it's well tested. However, when I search for example for middleware dispatcher, the sorting doesn't make no sense for me. The packages that are downloaded more then 1k are on page 2.

Maybe this is still work in progress, but why is it live at this moment?

[vworldat]

Then if you work on Indexing, you can simply create an account and set your credentials.

@julienbourdeau So if someone decides to host a private packagist instance for their business or just for fun, it will be free of charge as well? Otherwise I'm curious how this benefits the community in the long term in comparison to having a working search index without any runtime cost out of the box.

[julienbourdeau]

@GerardSmit Right now the idea is to improve relevancy so you find what you're looking for without thinking about how to order your results. You can find the PR for relevancy here #813 and test it on this live site. If you're not convinced, could you detail how you use the sorting and what you expect? Thanks!

@vworldat If you mean hosting a copy of packagist, with your own set of repositories, you can use the Algolia free tier if it fits.
It benefits the community because we believe we can provide a better search experience with better relevancy.

[alcohol]

Wait, so we are really not going to offer the functionality anymore that allows a user to sort results based on downloads for example?

That makes me quite sad :-(

[javiereguiluz]

Thanks for improving the search experience with Algolia. It's great. However, I agree with @alcohol: I'd love to see the "sort results by downloads or stars" feature back. Thanks!

[stevie-mayhew]

Unsure if anyone else has mentioned it but I couldn't find elsewhere.

Currently I am seeing an error when attempting to search from the packagist.org site which has the following message:

AlgoliaSearchError: Operations quota exceeded, change plan to get more Operations.

Looks like it may need to be flagged as an open source project if that hasn't already been done?

EDIT: further reading may indicate its actually on the OS plan but reaching the limits:
The standard volume of the AOS Plan is 100k records and 200k operations monthly

[Seldaek]

Search is back.

[Seldaek]

Closing this as I think it's kinda done, if needed we can open new issues for other specific things.