Colly: Check if URL in queue before adding duplicate

[dnlo]

Thanks for Colly!

I am using redis storage and I noticed duplicate URLs are added to the queue.
I suspect this is by design?

Can you suggest a good way to check the queue for a URL before adding a duplicate?

[vosmith]

The colly Storage interface used for queues does not have a way to check if a request is defined in a queue, but if you are using Redis you may be able to build and marshal a colly.Request, and search the list using LINDEX. The key for your list should be :queue unless you have set a Prefix on your redisstorage.Storage struct, in that case it would be your_prefix:queue.

Its not ideal but, let me know if that works for you.

[dnlo]

Thanks, I've decided to use redis as a storage and let colly.Visit() handle the queue duplicate links issue, it work for now, eats up a lot of RAM though.

[tegk]

Just use a Map with xxHash to deduplicate. If RAM is a issue use Bloom Filter

[rjpr]

Best way I've found to do this is using a set in redis, using SADD returns a 0 if the value already exists. That combined with a local map makes for some quick de-duplication.