Android: Domains visited get leaked to DDG servers

[Tritonio]

https://github.com/duckduckgo/Android/blob/ed91c9e551d2a9e1559199f110bd94c076784c71/app/src/main/java/com/duckduckgo/app/global/UriExtension.kt#L60

This seems to be leaking all(?) the domains that users visit to your servers.

[tagawa]

Hi @Tritonio and thanks for your feedback. The purpose of the request you observed is to retrieve a website's favicon so that it can be displayed in certain places within the app or on the results page. We use an internal favicon service because it can be complicated to locate a favicon for a website. They can be stored in a variety of locations and in a variety of formats. The service understands these edge cases and simplifies retrieval within our apps and our search engine.

At DuckDuckGo, we do not collect or share personal information. That's our privacy policy in a nutshell. For more detailed information on that, you can checkout our privacy policy at https://DuckDuckGo.com/privacy. The favicon service, as with all our services, adheres to this privacy policy in that the requests are anonymous and do not collect or share any personal information.

If you have further questions, please let me know.

[stefan01]

Hi @tagawa,
i do trust DDG not using the data in a bad manner, but still i don't get your decision.
The best privacy products are made from companies you don't need to trust using your data in a good way, because they collect no data.
So for a privacy product it is really hard for me to understand why you chose a minimal performance increase (?) over a critical privacy issue (in my opinion).
I think with this decision you are risking a major privacy distrust.

[donfuxx]

I understand that there are different favicon standards and that in some cases it can be difficult to locate it.
I believe that those are edge cases though and for a vast majority of websites a simple host/favicon.ico should work.
My suggestion for an increased privacy here is:

1. Attempt to load favicon directly from website by simply appending /favicon.ico to hostname (should be the most easy and most common case anyway)
2. In case of no success fallback to use omniscient ddg favicon endpoint

[stefan01]

Just to understand: Why can't the same algorithm, which is being used in the DDG endpoint, be utilized in the app?
If I understand correctly the only reasons are slightly increase performance, reduced network traffic and easier implementation.
Is that correct?

[masstransithonchkrow]

Hi there! If DDG reviews this, please use the header data on my web pages to determine what the favicon is. I do not have a default favicon for my site. My DeskThemePacks use PNGs for favicons based on the currently viewed theme's normal cursor.

By looking for favicon.ico where it doesn't exist, it affects both my site's performance and SEO.

[timmc]

If it was just search results, I wouldn't care.

But this appears to be in use for rendering tabs that you're visiting, regardless of how you got there:

https://github.com/duckduckgo/Android/blob/db728523240e377277deb88af71d751eb8bdb5dd/app/src/main/java/com/duckduckgo/app/tabs/ui/TabRendererExtension.kt#L44

(ETA: Or am I misunderstanding how this app is used? I'm not a smartphone user, and only use DDG in a regular browser.)

[tonyxu-io]

It’s shocking to see how DDG staff responded to this issue. Doesn’t seem to care much about user privacy concerns.

[jeevank]

@CDRussell @tagawa This needs to be reopened and looked into. This is a serious privacy issue regardless of your "privacy policy"

[sheerun]

This made me really doubt privacy of DuckDuckGo. How it can be private and secure if you hire security experts who allow it to happen since 2019? "it can be displayed in certain places within the app or on the results page"

[davewood]

Take the code from Firefox iOS or Android-components. We spent a lot of time on these and it is all on device.

https://github.com/mozilla-mobile/android-components

https://github.com/mozilla-mobile/Firefox-iOS 

[https://news.ycombinator.com/item?id=23708166]

[javabeanz]

privacy should be the whole raison d'etre of DDG. time for a fork DuckDucGone ?

[scabros]

I don't know why all the fuss... You all know that if you are using his app, you already "trust" that his search service is respecting your privacy, right? C'mon, they are being transparent, as usual.

[tonyxu-io]

I don't know why all the fuss... You all know that if you are using his app, you already "trust" that his search service is respecting your privacy, right? C'mon, they are being transparent, as usual.

@scabros - Please checkout the reply above: https://github.com/duckduckgo/Android/issues/527#issuecomment-652721495 This is not about using their search service, we all know search provider know exactly what you are searching for. But this particular issue is about using the browser, it's when user directly visit the website from browser address bar or reference link.

[svenssonaxel]

So essentially, the trade-off is between performance and auditability of privacy. I predict that the user segment DDG appeals to will overwhelmingly prefer the latter.

Yes, we already trust DDG, but only because we have to trust someone and others have proved to be untrustworthy. The issue isn't about whether the user trusts DDG, it's about minimizing the need for trust and maximizing the ability to verify privacy. Please consider reopening this issue.

[eighthave]

I think this would make this app qualify in F-Droid for the _Tracking_ anti-feature, there is no good reason why every website URL should be sent to DDG servers. Regardless of intent, it does key bits of data, and there are better technical solutions to finding the favicon.

[solomoncaygnuyou]

I believe reopening and rectifying this design choice would go a long way to preserving the trust that DDG has worked hard to earn in the privacy-oriented community. Even if users trust that DDG is not using this data in any undesirable way, it would provide more peace of mind to users that the data not be sent in the first place.

I am strongly in favor of changing this functionality to no longer leak visited domains.

[holderbaum]

A long time DDG user and advocate here.

First and foremost, thank you for building the most awesome search engine there is right now! :slightly_smiling_face:

I must say that I am pretty appalled by this implementation. Yes, I trust DDG with my search data since I have to do searches and DDG always seemed like they actually care about privacy. But at the same time, a privacy aware browser should never send close to all the user activity to it's server, even if it is "only" the domains that are browsed. I think most DDG users could live with a slightly less then optimal display of favicons in order to gain the assurance, that their domain history is indeed private to their device. That latter is literally the main reason, why most people install your browser in the first place.

I don't think this should be a discussion about "privacy policies". You are integrating a piece of code and infrastructure in your product, that has the capabilities to spy on people. It just shouldn't be there in the first place.

You should not keep this issue just closed like this, this is a serious matter and it is obvious from the community reactions that this needs to be discussed. (https://news.ycombinator.com/item?id=23708166)

[black-snow]

How is this still closed?

[stefanct]

This "feature" has been added in March 2018: https://github.com/duckduckgo/Android/commit/03f99c4a381d880ab850f0ddde9f692057a840e0#diff-63ac5c0d645555fe179e72977d9c1728

And as of now it is still unchanged in the develop branch: https://github.com/duckduckgo/Android/blob/e2f2d54a6b4452277467db403a3546512401b493/app/src/main/java/com/duckduckgo/app/global/UriExtension.kt#L83

(I did not look into the control flow to determine when this gets executed at all though.)

[thors]

I don't know why all the fuss... You all know that if you are using his app, you already "trust" that his search service is respecting your privacy, right? C'mon, they are being transparent, as usual.

1. This is not only about the search service.
2. Trust is only a last resort, after they did everything reasonably possible to avoid acquiring unnecessary data in the first place (which they apparently don't).

There are plenty of good browsers on the market. This browser had once privacy as a main selling point, but as it looks, not any more.

[RokerHRO]

@tagawa : Trust is a very brittle thing. It grows slowly and takes years to built it. And only one line of code can shatter it. But instead to react professionally and contritely you made it worse to stamp on the shards to make sure no useful piece of trust will survive. You had one job! :-(

[bluesign]

Sorry but this is not enough reason. There is a simple question you should ask to yourself.

• Would you be ok to use a third party for this with same privacy policy?

[zink-chimaera]

At DuckDuckGo, we do not collect (...) personal information

Meanwhile DDG literally collecting personal information from my device. 😕

DDG changed security issue: Ignored

[pythoneer]

For those that don't really understand what is happening:

It is really about that code snippet from here:

private const val faviconBaseUrlFormat = "https://icons.duckduckgo.com/ip3/%s.ico"

fun Uri?.faviconLocation(): Uri? {
    val host = this?.host
    if (host.isNullOrBlank()) return null
    return Uri.parse(String.format(faviconBaseUrlFormat, host))
}

The function for retrieving the Favicon is asking for the current host the user is on (e.g. apple.com) and sends it to a service hosted by duckduckgo.com that is saved in the const faviconBaseUrlFormat. %s is replaced with the actual host the user is on and is send to the duckduckgo.com servers. As an example (change it yourself to whatever host you want):

https://icons.duckduckgo.com/ip3/apple.com.ico

I guess they have a cache for already visited hosts etc.

This sends every host the user visits (or the Favicon is requested for) to the duckduckgo.com server. I think it does not really matter what you do or do not do with the data. Its not about trust – its about the minimum you need to trust and this is clearly breaking a lot of peoples perception of what the browser should achieve. As it currently stands, i would consider this even worse of what i imagine googles chrome would do to its users. To make it clear, i don't know exactly what google chrome is doing – its the perception of what i think certain vendors are doing, and i would not expect this from a "privacy first" browser from duckduckgo.com

Tired of being tracked online? We can help.

Please help!

[GarbageHamburger]

99% of the websites I have seen have either

1. a <link rel="shortcut icon">, which directly points to the favicon
2. if not, a /favicon.ico

I really don't see the point of this URL anyhow. It's useful for search results (because you'd have to download every page to parse its favicon otherwise, and your search results are already served directly from DDG so there'd be no point), but using it in tabs is not the right way. I'm willing to say this is just laziness rather than malice, though: rather than writing code to figure out the favicon from the page, they just used whatever they normally used on the search results page.

NOTE: DDG results page uses external-content.duckduckgo.com instead of proxy.duckduckgo.com, and

~$ host proxy.duckduckgo.com
proxy.duckduckgo.com is an alias for external-content.duckduckgo.com.
~$ host icons.duckduckgo.com
icons.duckduckgo.com is an alias for external-content.duckduckgo.com.

They also seem to be caching the most common favicons like https://github.com/duckduckgo/Android/issues/527#issuecomment-652882558 mentioned. i.e. visit "apple" on DDG and the top link has its favicon set as https://duckduckgo.com/assets/icons/favicons/apple.png.

Please reopen and fix.

[lethargosapatheia]

What I find most disturbing is that fact that they haven't said a word on this topic for a year now.

[GarbageHamburger]

What I find most disturbing is that fact that they haven't said a word on this topic for a year now.

I would expect them to respond now that the link blew up on HN.

[alex-tee]

At DuckDuckGo, we do not collect [...] personal information.

first post says otherwise.

[jonathanronen]

We love DDG for privacy, not for favicons. Privacy starts from not collecting data.

[NeverUsedID]

If trust is a balloon, this is a needle found in a haystack...

[tgy]

Adding a comment from Hacker News here that I find relevant when reading this thread.

There's an interesting disease showing up here in the responses.

I accept DDG's statement that this is about a favicon and that they "do not
collect or share any personal information", and despite that, I also agree with
others that DDG should be on the safe side and just stop doing this small
thing. It's just the safer and more moral thing to do (So DDG, as many are
suggesting, plz stop doing it. Today is good).

But... the reaction here is "they made a mistake, let's pile on like kids in a
playground" ignoring the genuinely huger issue of the amount of info and mining
that google et al. do. There's no measure of proportion in the responses,
someone is making a mistake then there's a wolfish, pack-like desire to get 
stuck in and hurt someone.

Which is why politicians rarely admit mistakes, because it's taken as a sign of
weakness, not strength, to admit you were wrong. DDG isn't the big evil on the
web but from reading some of these you'd think it was the 2nd google.

This isn't about DDG, just the proportionality of responses in public errors
and what society you'd like to have.

(no affiliation to DDG)

[b0z1]

I remember when gitlab made a mistake and many commits were lost.
It was a huge deal and they made a livestream and fixed it live.
Thats transparency!

Thats what i would expect from duckduckgo here.
See they made a mistake and fix it.

But closing the issue just like that is a statement. A huge statement.
"We dont care about your privacy"

[tagawa]

Sorry for the frustration this has caused. We're re-opening this to update the app to do this locally ASAP. Please see the follow-up comment by our Founder/CEO here: https://news.ycombinator.com/item?id=23711597

[Bonjur]

Thank you, that is the response people here want to hear.

[ganzgustav22]

I've just de-installed the Duckduckgo app and also won't be using their search engine anymore. Trust ist lost. Their CEO can put his statement where the sun doesn't shine.

[GarbageHamburger]

I've just de-installed the Duckduckgo app and also won't be using their search engine anymore. Trust ist lost. Their CEO can put his statement where the sun doesn't shine.

I don't think this kind of cut-throat response is the thing to do. @tagawa already said the issue is being worked on. Of course, it sucks that this was ignored for a year, but fixing it is the right response.

[chris-pauley]

https://news.ycombinator.com/item?id=23711597

The CEO of DDG just committed to resolving this by doing favicon lookups locally. This is actually a really solid response from the company. If they follow through, that makes DDG a pretty awesome company for owning up to a mistake and fixing it. Not something you'd see from any of the other major search providers.

[solomoncaygnuyou]

Adding a comment from Hacker News here that I find relevant when reading this thread.

@tgy
That comment is a fine example of 'whataboutism'. DuckDuckGo has no control over Google, so it there is no point in rhetorically asking, "Why care about this issue with DuckDuckGo when Google does so much worse?" DuckDuckGo can however improve its own services when there is clearly improvement needed.

Nobody is "piling on" DuckDuckGo for this. You are witnessing a community of privacy-oriented users express dissatisfaction with a subpar response to a clearly contentious implementation in their software. How else do you expect changes to be made to the software? People have to use their voice or else nothing will be done.

As we can see by this thread, our voices were finally heard (a year after the issue was originally opened) when enough people piped up in a small window of time.

[azihassan]

This will exacerbate my anatidaephobia

[RokerHRO]

@tgy :

[…]
But... the reaction here is "they made a mistake, let's pile on like kids in a
playground"  […]
Which is why politicians rarely admit mistakes, […]

As a software developer I know for sure the difference between a "mistake", that happens without intention, and a feature that is always built in _with_ intention.

And, sorry, but you cannot argue that a company that cares about privacy seriously adds an online service that tracks (and can collect) user's behavior only "by accident".

And the answer of the CEO is also not a good excuse for that:

"[…] our services are encrypted and throw away PII like IP addresses by design"

That is just an empty promise that we cannot verify (and you can hear such phrases from nearly every cloud service provider), so it is useless. Or even worse: If the CEO only publish such PR statements, instead of plausible explanations for this issue, it still looks like a feature that was added intentionally and now – oops – someone found it.

[lethargosapatheia]

I would really give them the benefit of the doubt, I wouldn't compare them to google or whatever. Moreover, they would compromise themselves badly, if something really fishy were to be revelead. This is part of the core of their business.
I'll put it down to negligence in this case - not that that's an excuse, as I've already said, it's already been a year, yet it's quite different.

[niksmac]

I still don't see a meaningful reason behind this decision, especially from a company which is claiming to be privacy focused. This could've been handled in a better way for sure. Hope they will fix it soon. Nevertheless it was a simple thing which can be avoided altogether.

[seungjulee]

Tritonio opened this issue on 9 Jul 2019

@CDRussell CDRussell closed this on 12 Jul 2019

It took a year, and HN front page to get it resolved.

[u8983478934]

I've always put DDG in the same category of fishy """pro-privacy""" software companies like Brave that their whole raison d'etre is to rely on aggressive marketing attacking big companies like Google.

[thors]

Adding a comment from Hacker News here that I find relevant when reading this thread.

Read it, laughed, and ignored it there (because I didn't want to create an account there just to debunk it)

There's an interesting disease showing up here in the responses.

(So DDG, as many are suggesting, plz stop doing it. Today is good).

Here I agree

But... the reaction here is "they made a mistake, let's pile on like kids in a
playground" ignoring the genuinely huger issue of the amount of info and mining
that google et al. do.

The main problem is not the mistake they made in implementation. The issue, why people start piling up is that they don't acknowledge it as a mistake. That, imo, justifies to continue screaming until they do change the implementation.

Which is why politicians rarely admit mistakes, because it's taken as a sign of
weakness, not strength, to admit you were wrong.

That is bullshit. DDG doesn't receive flack for admitting a mistake, they are exactly receiving flack for not admitting it,

DDG isn't the big evil on the
web but from reading some of these you'd think it was the 2nd google.

DDG, like Google, is a company and obliged to maximise its investors profits (well, they are not publicly traded, so maybe not to the same extent). They are not good or evil (neither is Google), they have a selling point (privacy), which they, right now, appear to contradict with their visible actions. If they don't pay attention on their visible actions, in spite of the marketing they do, they might lose their selling point.

This isn't about DDG, just the proportionality of responses in public errors
and what society you'd like to have.

No. This is about DDG. They can reopen the issue, change the implementation, and I will immediately stop commenting about this issue (or maybe make positive comments, how they took customer concerns serious)

[sentialx]

doesn't chromium literally have an event for favicon update? WebView even has the favicons database known in Chrome, so it's even possible to get a favicon before a page loads (you can check it by typing chrome://favicon/https://duckduckgo.com) https://developer.android.com/reference/android/webkit/WebViewClient#onPageStarted(android.webkit.WebView,%20java.lang.String,%20android.graphics.Bitmap)

WebChromeClient.onReceivedIcon:
https://developer.android.com/reference/android/webkit/WebChromeClient#onReceivedIcon(android.webkit.WebView,%20android.graphics.Bitmap)

[thors]

I've just de-installed the Duckduckgo app and also won't be using their search engine anymore. Trust ist lost. Their CEO can put his statement where the sun doesn't shine.

Good luck finding a better option, privacy-wise...

[RUSshy]

i knew it, duckduckgo is operated by the CIA/NSA

no wonder why they refused audits from independent orgs

[Tritonio]

I've just de-installed the Duckduckgo app and also won't be using their search engine anymore. Trust ist lost. Their CEO can put his statement where the sun doesn't shine.

Good luck finding a better option, privacy-wise...

Firefox Focus is IMO better privacy-wise.

[holderbaum]

Thank you, DDG Team, for taking this matter seriously. Very good response from Gabriel on HN! :heart: :partying_face:

[us31t]

[DDG Marketing] We respect your privacy, we are a privacy company! We fight BigCorp with transparency!
[DDG Dev] Ok, lets build a feature! Hmm... what could we do next? Ahhh I got a great idea! Lets send the entire users browser history to our servers, so we can ... hmmm lets see ... how could we argue this ... a yes! Show a favicon!
[OS Community] Files Ticket: Hey guys you are sending the entire browser history to your servers, is that by mistake? Could you please anyway stop that? You said you respect our privacy!
[DDG Dev] Who are you? What are you talking about? There is nothing wrong with us sending your browser history to our server! We have a privacy policy, can`t you read? This is not a bug this is a feature! - Ticket Close
[OSS Community] WFT just happened? Are you nuts?
[One year Later] Topic shows up on HN
[DDG Management] Oh hello I just woke up (after one year?), I am new to this issue and I do not know what is going on, because I am the CEO! We will remove the feature again! We really did not intend to develop a feature! We are new in this whole privacy thing! Please forgive us. Instead you really should trust us again! See we have this [Enter_A_Super_Duper_Secret_Privacy_Buzzword_That_Is_Not_Available_For_Verification] process, that removes ALL personal information! Really! I am the CEO! CEOs are always right and never lie in public! Could you please stop flaming now?
[Some Trolls] Thank you DDG, that you are talking this seriously!
[OSS Community] WFT just happened? Are you nuts?

[TBC] - This August on Netflix!

[resynth1943]

I love how the Reddit squad have hijacked this issue with their egregious accusations and demands. It really does reflect badly on everyone here if this is the standard for communications.

The amount of off-topic content here is shameful, and very stressful for anyone that then has to clean up your mess. I really don't see how you expect a collaborative response when this is the attitude here.

That being said, I would be interested in a technical explanation for this functionality, if the DuckDuckGo developers would be so kind to provide one.

EDIT: every down vote saves a kitten's life! Save the kittens! :cat:

Many thanks,
Synth

[Tritonio]

I love how the Reddit squad have hijacked this issue with their egregious accusations and demands. It really does reflect badly on everyone here if this is the standard for communications.

The amount of off-topic content here is shameful, and very stressful for anyone that then has to clean up your mess. I really don't see how you expect a collaborative response when _this_ is the attitude here.

That being said, I would be interested in a technical explanation for this functionality, if the DuckDuckGo developers would be so kind to provide one.

Many thanks,
Synt

In 2019 I mentioned it on reddit, then made the bug report here as someone suggested and got a link to their privacy policy on both places. The same arguments that we are making now were made a year ago as well. The only difference (that I can see) is the number of people that got interested in this. So don't dismiss this chaotic response as unnecessary. I, for one, am quite grateful for it.

[tgy]

This isn't about DDG, just the proportionality of responses in public errors
and what society you'd like to have.

No. This is about DDG. They can reopen the issue, change the implementation, and I will immediately stop commenting about this issue (or maybe make positive comments, how they took customer concerns serious)

@thors I think you misunderstood what the HN commenter meant by "This isn't about DDG". In this particular sentence "this" refers to her own comment. Her own comment is not about DDG itself but about the reactions here.

I agree with you that the continuous screaming is necessary for DDG to react. A sad reality.

I also believe that the whole point of open sourcing the code of the app is exactly what we are witnessing here. This allowed the community to notice this issue and make DDG fix it.

[GarbageHamburger]

I love how the Reddit squad have hijacked this issue with their egregious accusations and demands. It really does reflect badly on everyone here if this is the standard for communications.

The amount of off-topic content here is shameful, and very stressful for anyone that then has to clean up your mess. I really don't see how you expect a collaborative response when _this_ is the attitude here.

That being said, I would be interested in a technical explanation for this functionality, if the DuckDuckGo developers would be so kind to provide one.

Many thanks,
Synth

You're being equally off-topic about it by complaining here though? You don't need to virtue signal.

[resynth1943]

In 2019 I mentioned it on reddit, then made the bug report here as someone suggested and got a link to their privacy policy on both places. The same arguments that we are making now were made a year ago as well. The only difference (that I can see) is the number of people that got interested in this. So don't dismiss this chaotic response as unnecessary. I, for one, am quite grateful for it.

I'm not, and I heavily doubt anyone that has to clean it up is happy, either.

Sadly, it seems that comments that divulge on the technical side have been ignored by the Reddit squad, while egregious accusations have been upvoted. This, in my opinion, should raise questions about what the priorities of this squad really are.

You're being equally off-topic about it by complaining here though? You don't need to virtue signal.

I may be in your opinion, but I do not see any accusations or conspiracy theories in that one.

---

I think it's quite sad that people can dismiss a genuinely good product for one simple, and human, error. As a wise man once said: "don't throw the baby out with the bathwater!"

EDIT: every down vote saves a kitten's life! Save the kittens! :cat:

Many thanks,
Synth

[ddevault]

Thank you for re-opening and prioritizing this, @tagawa.

However, this problem demonstrates gross incompotence for a browser team supposedly concerned with privacy. Will you please do a post-mortem on how this code made it through your code review process in the first place, as well as how it managed to stay in place for a full year after it was pointed out that it represented a privacy problem?

"Sends every URL you visit to the vendor's servers" is the single worst thing DuckDuckGo could have done for privacy in this web browser, and that needs to be accounted for. There was a major failure in the code review process, ticket review process, and in how you treat your community. A standard marketroid "by design" response with washy promises that "we'll take very good care of this highly sensitive personal data, just trust us" is not something I want to see in the future from this team.

[ddevault]

By the way, step one should be removing this feature and shipping a fixed browser ASAP, even if it breaks favicons, and then working on a more robust solution afterwards. I cannot overstate the gravity of this problem.

[Noir-]

Behavior like this "feature" is one of the central points of criticism against "the tech giants". Everyone who's more into this topic knows what these companies doing and that they also always promise to take care. And we see how this is working out in reality. That's why privacy aware users want to minimize the potential for abuse by living the need-to-know principle. So I can't deny that this "feature" changes my view on the privacy awareness of the developers. This was a conscious design decision so treating it like a bug or "human error" seems inappropriate to me.

But please, don't lose your mind over this. Many privacy-friendly projects have such flaws. Mozilla, Signal, LineageOS, you name it. Making a software which is usable, privacy friendly and accepted by a large user base is a hard task. And I can't stress enough how important a big user base for a technology is. When people keep shitstorming the user base will shrink and there will be less room for improvements. In the end, only the tech giants will benefit from this.

[Tritonio]

This, in my opinion, should raise questions about what the priorities of this squad really are.

Squad? What squad? This is a bunch of random users that are pissed, and rightfully so. There is no squad. I called this "chaotic" on purpose. And I still believe it was needed.

I think it's quite sad that people can dismiss a genuinely good product for one simple, and human, error. As a wise man once said: "don't throw the baby out with the bathwater!"

These people obviously don't have the same definition for "simple error" as you (I don't think it's simple btw), or have less tolerance for errors, or care more about their privacy, or have a higher tendency to look for patterns and conspiracies. Nothing "sad" about this, it's just different personalities. They are not threatening violence or anything so what's the problem? Also, most don't complain about the product, bug happen, they complain about ignoring the bug report and for missing the point of why data like that shouldn't be sent to their servers in the first place, if I am not mistaken.

[nathgit]

https://news.ycombinator.com/item?id=23711597 comment from DDG founder 4hrs ago.

[catehstn]

Closing as duplicate of #877, fixed in #878.

[resynth1943]

This, in my opinion, should raise questions about what the priorities of this squad really are.

Squad? What squad? This is a bunch of random users that are pissed, and rightfully so. There is no squad. I called this "chaotic" on purpose. And I still believe it was needed.

What was not needed was totally off-topic spam, and paragraphs of complaints and egregious accusations against DuckDuckGo, claiming they are best buddies with the USA, etc. etc.

I suggest you read the entirety of this thread, instead of asking me where it went wrong.

I think it's quite sad that people can dismiss a genuinely good product for one simple, and human, error. As a wise man once said: "don't throw the baby out with the bathwater!"

These people obviously don't have the same definition for "simple error" as you (I don't think it's simple btw), or have less tolerance for errors, or care more about their privacy, or have a higher tendency to look for patterns and conspiracies. Nothing "sad" about this, it's just different personalities. They are not threatening violence or anything so what's the problem? Also, most don't complain about the product, bug happen, they complain about ignoring the bug report and for missing the point of why data like that shouldn't be sent to their servers in the first place, if I am not mistaken.

Personally, I think it was an oversight on behalf of the DuckDuckGo developers. Nevertheless, it has been fixed, so the squad doesn't need to rampage any more.

---

I'd like to say a massive thank you to the DuckDuckGo developers for being so receptive to this issue! I have just seen that it has been fixed.

When can we expect a release in the app stores, please @catehstn?

[Tritonio]

What was _not_ needed was totally off-topic spam, and paragraphs of complaints and egregious accusations against DuckDuckGo, claiming they are best buddies with the USA, etc. etc.

I suggest you read the entirety of this thread, instead of asking me where it went wrong.

I read it. I see one comment that mentions NSA which qualifies as a conspiracy theory. I see a couple more saying it's fishy which is not a conspiracy theory. How many do you see that mention conspiracy theories? The complaints are absolutely warranted, and it's the reason why DDG took notice in the first place.

[RokerHRO]

Closing as duplicate of #877, fixed in #878.

Interesting workflow: Closing the old ticket as "duplcate" to a freshly created issue. Why that? To improve project's metric in "time between creation and fixing an issue" (where issues marked as "duplacates" won't count).

[catehstn]

Phased rollout on Android is already underway, iOS to follow.

[joshmanders]

@resynth1943

EDIT: every down vote saves a kitten's life! Save the kittens! 🐱

I'm just here to save kittens.

[resynth1943]

@joshmanders

@resynth1943

EDIT: every down vote saves a kitten's life! Save the kittens! cat

I'm just here to save kittens.

LOL, same. I downvoted them, too :D

[pluma]

This seems like a simple misalignment of expectations.

In DDG's minds, people using the DDG browser likely already use the DDG search engine because that's how they built their brand recognition. And if you already use the DDG search engine you're obviously fine with how the DDG search engine gets its favicons (because the alternative would by our browser making requests to each server which would be the true privacy nightmare). So rather than implement all the nuances of favicon determination (because "just look at favicon.ico" or even "check for a link element with a shortcut icon rel" only solves the 80% case, not the 20% of weird edge cases) in the browser they figured why not just reuse this purpose-built proxy they already had lying around. Makes perfect sense.

But apparently a lot of people using DDG (or at least a lot of people who have opinions about DDG) do NOT trust the DDG search engine or at least think that data minimization (even when that data allegedly doesn't really get stored anywhere) is more important than reliably showing favicons.

At best this just shows how high a bar DDG has managed to set for itself that its users are holding it to. At worst some people are just jumping on the bandwagon to concern troll about behavior that is already leaps and bounds ahead of the competition in terms of privacy and data protection.

[LinAGKar]

This seems like a simple misalignment of expectations.

It's not just that. Sure, this doesn't matter much if you get to the page via the search engine, but you can go directly to a web page without using a search engine, in which case the search engine should not know about it.

[Tritonio]

why not just reuse this purpose-built proxy they already had lying around. Makes perfect sense.

At best this just shows how high a bar DDG has managed to set for itself that its users are holding it to.

Not leaking every single domain a user visits is not a high bar. It's the bar you need to pass not to be considered spyware. FFS why are you people trying to defend an idiotic code decision? They are fixing it, great, I'm really happy for that. That doesn't mean we should trivialize the matter.

is already leaps and bounds ahead of the competition in

Leaking the domains to a server puts you leaps behind your competition and even your non competition since even regular browsers that don't focus on privacy end up being more private than you.

DYOR but this is why I think these bugs are very serious:

40.114.178.124 is the IP that icons.duckduckgo.com resolves to and WHOIS says that it's managed by Microsoft. If this is Azure, which I would take a bet it is, then that means that for the last year or more, Microsoft could, technically tap into that server's memory to see who requested which domain and when. So not only do you need to trust DDG, you also need to trust Microsoft that they didn't do anything bad with the info. Does anyone still remember PRISM? This kind of passive access to user data is exactly what PRISM was about. Although I could be wrong and maybe the actual server is running in DDG's premises and Microsoft is simply acting as a reverse proxy.

[lemiesz]

Seems like even if you could not fix the favicon serving issue, just deleting the favicon and not having any would be a better user experience. Given that putting the favicon up is breaking a core value proposition of your service.

[nilnilnil]

Hi all, CTO of DuckDuckGo here. I thought it might be helpful if I briefly shared some of our internal thinking around this issue so folks can see how we got here and how we plan to move forward.

The logic behind how we’ve been displaying favicons in our apps is a function of how we operate our private search engine. Since we already have an anonymous favicon service through our search engine, using it has a number of benefits: it avoids more requests to known non-anonymous websites that are visited, it's way faster since it runs server side, saves user bandwidth, and the only externally visible attribute is that the app is connecting to DuckDuckGo.com (as the favicon location is actually encrypted in the path in transit). To our team, utilizing this anonymous service we had made for the search engine seemed like an optimal principled choice across a set of criteria.

We want to be clear that at no point was the actual visited domain otherwise exposed. This favicon service is fully anonymous on our end, and URL parameters (like the favicon domain) are encrypted in transit, just like the search engine (with search queries). This is also why when this issue was raised in the past, we decided to keep the solution as is. At no point was this ignored.

However, we understand that there is an alternative method of getting the favicons locally that a lot of folks prefer while still maintaining our privacy standards. We also believe that method is in line with our product vision of "Privacy, simplified.", considering its a somewhat simpler method than the one we had been using.

So, we went ahead today and implemented the change for both Android (https://github.com/duckduckgo/Android/pull/878) and iOS (https://github.com/duckduckgo/iOS/pull/667) that will move this logic onto the client, and we will no longer be using the favicon service in our apps. These changes are currently in the release phase and are rolling out live now.

We really appreciate the feedback and exchange of ideas on this topic, and on ways to further improve the privacy of our products in general.

[calimeroteknik]

while still maintaining our privacy standards

That's quite a curious wording to talk about a solution that objectively raises privacy to a much higher standard.
Well, it's good to hear that more privacy "still maintains" your privacy standards.

Please remember, my dear DDG: don't be Google.

[resynth1943]

I'm personally surprised nobody has talked about the fact that DuckDuckGo's search engine has a strange redirection function when you click on a link. I would love to talk about that at some point, because I really don't understand why it's there.

Many thanks,
Synth

[alectrocute]

There's about seven red herrings in this huge thread. A lot of people who suddenly use DDG when it's convenient to complain about it. Right. @Tritonio was the only one whom I saw explain the actual vulnerability here (provided you have a positive bias of trust for DDG, which disclaimer, I do):

40.114.178.124 is the IP that icons.duckduckgo.com resolves to and WHOIS says that it's managed by Microsoft. If this is Azure, which I would take a bet it is, then that means that for the last year or more, Microsoft could, technically tap into that server's memory to see who requested which domain and when. So not only do you need to trust DDG, you also need to trust Microsoft that they didn't do anything bad with the info. Does anyone still remember PRISM? This kind of passive access to user data is exactly what PRISM was about. Although I could be wrong and maybe the actual server is running in DDG's premises and Microsoft is simply acting as a reverse proxy.

Ah, SIGINT. Spooky SIGINT.

This quote points at Microsoft Azure's IP for example's sake, but that doesn't account for the countless ISPs your data travels through, even on a basic DNS level without vendor participation. A nefarious agency could MITM and use this as spyware for a correlation++ attack specifically targeting DDG users. That should terrify all of us because we love privacy, (right?) and when the right adversary arrives – "could" is usually a synonym for "will inevitably".

I fully 100% believe it's innocent, I even feel it makes perfect sense in the mind of a rushed developer. I am happy to see it fixed. Now, with that said, I'd like to propose a solution so the product marketing people can be happy too.

If you want to have this feature locally, make it secure and do the same thing – just bundle a massive SVG sprite sheet with the client (realistically, how big would it be?) and do some sort of table association to pull them up on demand if the hostname has a match. Otherwise, don't make an outbound network request, just return and wait! If you wanted to change the way you do it (and follow more-or-less standard practice), you can rely on site-provided webContent <head> data as others have mentioned.

Anyway, I'll pass the mic off now. I think @nilnilnil and DDG exec team did a great job of fixing the issue. Up and onwards!

[goranmoomin]

@resynth1943 It's to not expose the referrer so that the site can't know what the user searched.

[tonyxu-io]

@resynth1943 It's to not expose the referrer so that the site can't know what the user searched.

This can be prevented by:

<a href="example.com" rel="noreferrer">link</a>

[resynth1943]

Interesting. Thank you, @pcr910303. Personally, I'd love to see Referer die a death, it's about damn time.

[resynth1943]

@resynth1943 It's to not expose the referrer so that the site can't know what the user searched.

This can be prevented by:

<a href="example.com" rel="noreferrer">link</a>

That is curious. When I click on a link in the search engine (here's one about ghosts), I get something like this:

https://duckduckgo.com/l/?kh=-1&uddg=https%3A%2F%2Fen.wikipedia.org%2Fwiki%2FGhost_(Swedish_band)

<html><head><meta name='referrer' content='origin'></head><body><script language='JavaScript'>window.parent.location.replace("https://en.wikipedia.org/wiki/Ghost_(Swedish_band)");</script><noscript><META http-equiv='refresh' content="0;URL=https://en.wikipedia.org/wiki/Ghost_(Swedish_band)"></noscript></body></html>

Couldn't this be used to track browsing history, too? Technically, query parameters are sent to the server. Wouldn't mind knowing more about this, if they would explain more about it. An interesting thought would be, if the user has JavaScript enabled, why couldn't DuckDuckGo cook up a URL that looks a little more like this:

https://duckduckgo.com/l/#/kh=-1&uddg=https%3A%2F%2Fen.wikipedia.org%2Fwiki%2FGhost_(Swedish_band)

See that tiny change? That little hash prevents the URL from being sent to the server! I wonder how that would work with referers, too. Of course, parsing that hash would need a little bit of JavaScript, but if the user has it enabled, then...

Maybe there's a technical reason why they use this redirection service, instead of noreferrer? Perhaps backwards-compatibility?

EDIT: Looks like I was right.

---

EDIT: Of course, this wouldn't be a problem if they used POST instead of GET, but I'm told Chrome doesn't play nicely with that.

[tagawa]

@resynth1943 You're right, the redirect is for backwards-compatibility reasons (there's more about it on this help page) but it can be disabled in the search settings, in which case just the referrer meta tag is used. That's set to origin by the way.

Re. POST and GET, that can also be toggled in the same settings page with GET as the default.

[PranavBhattarai]

DuckDuckGo just went viral but in a very WRONG way.
This is bad + sad.

[waltercool]

Come on guys, stop trolling on this post. DuckDuckGo already made an statement on this.

Don't like it, fork it. Stop crying and demanding work from others.

[scabros]

I am absolutely AMAZED on how people OVERREACTED about this. First disclaimer: I have NO connection to DDG, I am just a user. I work as sysadmin and web developer full time since 2004.

I do agree this is a "feature" that could be implemented locally to let the users feel more comfortable with his privacy concerns.

But this, for me, doesn't represent an anti-privacy practice, as a lot of people are complaining. DDG is basically a search engine. In my own phone I have DDG AND Firefox focus, and use the last as my default browser. Either way, again, I think is a good choice to make the requested change to the local implementation.

What really irritates me is the tone some people come here, complaining in a way that is practically accusing DDG being the same as google & Co. Folks, the distance is HUGE, and i think behind all this there is a key point: it's not about if DDG is receiving your data or not (because, hey, they are a search engine, you know, if you don't input anything, the can't search anything for you). The key point here is what they DO with your data. And they ALWAYS were very clear about that. They anonymize it and process it, to have some valuable info that they could use to improve their products. The "favicon incident" from the users perspective, is just an example of another anonymized service for DDG.

And then you have people coming here saying things like this:

• best privacy products are made from companies you don't need to trust using your data in a good way, because they collect no data
• Doesn’t seem to care much about user privacy concerns
• How it can be private and secure if you hire security experts who allow it to happen since 2019?
• There are plenty of good browsers on the market. This browser had once privacy as a main selling point, but as it looks, not any more.
• Meanwhile DDG literally collecting personal information from my device
• As it currently stands, i would consider this even worse of what i imagine googles chrome would do to its users.
• But closing the issue just like that is a statement. A huge statement. "We dont care about your privacy"
• I've just de-installed the Duckduckgo app and also won't be using their search engine anymore. Trust ist lost. Their CEO can put his statement where the sun doesn't shine.
• And, sorry, but you cannot argue that a company that cares about privacy seriously adds an online service that tracks (and can collect) user's behavior only "by accident".

All this people, use android? or they have hacked their phones and installed librem? could they all do a better job competing against google?
the error was there and it needed to be fixed, it required doing some pressure as his user base, i get it... but this?
"the internet" is full of people very very very bored, and always ready to point out the mistakes of others, as if they did everything right... but they cant even just point an issue in an open, free and transparent service (there are not many out there...), manage his own anger, fears and frustration and being polite to keep asking for it.

To ALL DDG staff, thank you for being there, thank you for giving us an alternative to G, thank you for listening, thank you for being open and transparent and thank you for being condescendent even with those that dont deserve it (because you know there are many that do).
(PS: english is not my first language, I hope you understand )

[resynth1943]

Come on guys, stop trolling on this post. DuckDuckGo already made an statement on this.

Don't like it, fork it. Stop crying and demanding work from others.

Who's trolling?

[resynth1943]

@scabros that's fair enough, dude, and I agree with that. People were rude, malicious and forming egregious remarks about the company that just made a mistake.

I can see people have already down voted your post. I've got 31 down votes on a rational opinion, so good luck communicating with the children here.

DuckDuckGo really isn't the second Google.

Far from it.

And that's why I use it.

[JasonMFry]

Unfortunately this isn’t the only “mistake” DDG has made: DuckDuckGo's privacy abuses-- current, historic, and by proxy. People may be overreacting to this post if this post represented a single mistake, but given the context of the post I linked, I suspect most haven’t reacted nearly strongly enough.

Edit: link to hacker news instead of directly to the article

[us31t]

First of all - I am a big fan of DDG

• I promote it whenever I can
• I use it anywhere possible
• This comment is meant to help the DDG team to understand this PR issue. Even tough - _we_ _(OS_Community)_ are very concerned that this explanation is necessary in the first place! _We_ normally do not rise up from our cellars - because _we_ code. But now where the realms of management and C0D3 collide again, and the management does not show any signs of understanding - _we_ have to take action.

This is why I will now explain, (again) what the problem with your answer is and what it indicates.

Hi all, CTO of DuckDuckGo here. I thought it might be helpful if I briefly shared some of our internal thinking around this issue so folks can see how we got here and how we plan to move forward.

Your 'internal thinking' is what has raised this issue in public! It is basically the bare reason for our concerns! Because it shows, that you do not even understand our privacy argument! It is not about the bug, it is your action and the, like @calimeroteknik puts it, "curios wording", that makes it appear, like you have not understood our critic at all.

Time of Events

1. @Tritonio found a curios line of code that sends unnecessary requests to one of your backends
2. @tagawa said: "The purpose of the request you observed is to retrieve a website's favicon so that it can be displayed in certain places within the app or on the results page"
   https://github.com/duckduckgo/Android/pull/878/files#diff-63ac5c0d645555fe179e72977d9c1728
3. Then @stefan01 and later _we_ said, please stop this! There is no reason to send data in the first place! A big part of privacy focused software is, to avoid to creating unnecessary data in the first place. It is not necessary! Please do not send our browser history or any unnecessary request, to any server! No matter how "secure" this server is. Please load the favicon locally, we love you and we do not like heartbreaks!
   In the worst case - security is only a illusion - only the hacker with root access knows, what is really going on and maybe like @Tritonio indicates Microsoft.
4. DGG show no real understanding and argues in a "curios way" - and now you @nilnilnil continue

Bugs can happen! Bad decisions can happen! But arguing with "trust us, it anonymous", it "saves user badwidth", and its "the only visible attribute", on top of that "we already have a anonymous [...] service"! It is pure poison for trust, because it indicates, you entirely missed the point! Either on purpose or not. Both would be alarming!

The logic behind how we’ve been displaying favicons in our apps is a function of how we operate our private search engine. Since we already have an anonymous favicon service through our search engine, using it has a number of benefits: it avoids more requests to known non-anonymous websites that are visited, it's way faster since it runs server side, saves user bandwidth, and the only externally visible attribute is that the app is connecting to DuckDuckGo.com (as the favicon location is actually encrypted in the path in transit). To our team, utilizing this anonymous service we had made for the search engine seemed like an optimal principled choice across a set of criteria.

Its not that we do not understand your reasoning or the reasoning of @tagawa. The problem is your reasoning, seems to entirely miss our point!

Now you continue - as CTO! This indicates no real insight at all!

We want to be clear that at no point was the actual visited domain otherwise exposed. This favicon service is fully anonymous on our end, and URL parameters (like the favicon domain) are encrypted in transit, just like the search engine (with search queries). This is also why when this issue was raised in the past, we decided to keep the solution as is. At no point was this ignored.

We said, please do not mix the search engine code and your browser code! They should not relate on each other if not necessary (for example if the user actually wants to do a search on DDG).
Arguing "we already have a anonymous [...] service" is very hard to understand from a technical point of view. It might make sense for a CFO, but not for nerds or hackers!

However, we understand that there is an alternative method of getting the favicons locally that a lot of folks prefer while still maintaining our privacy standards. We also believe that method is in line with our product vision of "Privacy, simplified.", considering its a somewhat simpler method than the one we had been using.

Like @calimeroteknik already said: Luckily higher security standards meet your standard! -_-

So, we went ahead today and implemented the change for both Android (#878) and iOS (duckduckgo/iOS#667) that will move this logic onto the client, and we will no longer be using the favicon service in our apps. These changes are currently in the release phase and are rolling out live now.

You only took this action after enormous public pressure, there was no sign of insight or internal intention to resolve this! This is why you have been compared with BigCorp, because they act like this. (Or like pointed out by some people, are even doing a better job than you, handling these kinds of issues)

[LARP MODE ON]
[Loading Game Data...]
You currently have (3) quests!
[(1) Closed Quest] Even under enormous public pressure, do not remove the criticized code, before at least one year has past!
[(1) Open Quest] Understand the whole point, and communicate it in a acceptable way.
[(1) Bonus Quest] Build trust back by transparent actions in the future, which reflect that you understood what we said
[LARP MODE OFF]

We really appreciate the feedback and exchange of ideas on this topic, and on ways to further improve the privacy of our products in general.

You are welcome! I hope this helps!

[calimeroteknik]

We really appreciate the feedback and exchange of ideas on this topic, and on ways to further improve the privacy of our products in general.

I overlooked the implications of this. It's much more interesting than it seems.

Consequence 1:

It means that DDG has started a game of

the privacy defects we introduce and you don't notice, we can keep

…with the general public.
The dynamics of this game can be summed up to "we keep hiding things around, and you keep looking for them".

That's immoral, but I believe it's legal and therefore legally allowed to do this. (after all, DDG users are not paying customers)
Additionally, it's a private company: a subpoena can compel it to lie to the public and to its customers.

Consequence 2:

The current evidence unambiguously proves that DDG is commiting at least one of:

• incompetence concerning basic privacy notions ;
• basic malevolence, motivated by financial profit and potentially by government pressure.

A real plight, if you want to get away with only PR!

I think that, in order of preference for the public, your options are:
1st. Admit DDG was not respecting privacy in places where it could have, and that the team attempted to not admit it;
2nd. Keep silent, and not do that sort of thing again, so you don't get busted for such practices in the future;
3rd. What you've been doing: say "hey, it's not THAT bad"⁽¹⁾ and "we appreciate the feedback"⁽²⁾.

(1) This is a pretty effective psychological strategy, because it induces self-disgust in people who are not trained to fight it internally. Studying WW2 propaganda helps with understanding that one.
(2) Following it with such benevolent-looking words, like this, is pretty efficient at maximizing your chances at people getting tired of arguing. Unfortunately, that can backfire, which is happening right here.

My sharp words should not hide my encouragements towards duckduckgo fighting its innate corporate nature.
I never took the time to write such a reply to Google, for instance, because I think it's a lost cause whereas DDG isn't, yet.

[The-Compiler]

This is why you have been compared with BigCorp, because they act like this. (Or like pointed out by some people, are even doing a better job than you, handling these kinds of issues)

FWIW Google has a service used in their Chromium browser (and also used by Firefox and others) for clients to find out whether a given host is malicious (e.g. a known phishing page). Their API can be used in a way in which hostnames are not sent to Google, and even Google Chrome does things that way. (Then again, there might be other features in Chromium sending hostnames to Google, such as the omnibox autocompletion)

[waltercool]

Who's trolling?

Everyone who keeps posting here, DDG already received a PR and merged at the codebase https://github.com/duckduckgo/Android/pull/878

Is non-sense to keep posting "furious" and "frustrated" comments here. Again, if people dislike DDG Android app in any sense, don't use it, or just fork it.

The whole idea of OpenSource isn't treating developers as slaves who fulfill all demands of everyone, or to make threats at reddit or any other websites (posted bit above) full of conspiracy stuff and making claims of things they aren't true. This kind of behaviors of developers being kind of slaves from users must STOP.

[calimeroteknik]

@waltercool the discussion here is not about the software.
It is about how DuckDuckGo, the company, responds when privacy problems are pointed out.

Admitting it was a mistake would be a start as far as this discussion goes, and that hasn't happened as far as I can tell.

[waltercool]

@waltercool the discussion here is not about the software.
It is about how DuckDuckGo, the company, responds when privacy problems are pointed out.

Admitting it was a mistake would be a start as far as this discussion goes, and that hasn't happened as far as I can tell.

I do understand your concern, but please look the "issue" and "official response". Everything else happened after the ticket was closed, basically someone said: "Hey look this code you are leaking all url from users!", and they said "ehhm, no".

A good way to report "a software issue" (because the original claim IS a "code" claim), would be creating a PR proposing a replacement of the DDG favicon server. Companies do opensource not only to expose their code to others, but for community to contribute bugs with change proposals. Reporting a bug doesn't solve anything, proposing solutions does.

And following this, a claim related to "privacy concerns" and not "a code issue" happened at issue https://github.com/duckduckgo/Android/issues/877, which the reported posted a change request at Asana platform, and they accepted it.

[javabeanz]

favicons can even be weaponized : https://github.com/devanshbatham/FavFreak

[ichelash]

DDG is no difference to google...no privacy at all #my opinion though

[resynth1943]

I love how GitHub seems to be removing any and all functions that users can use to report abuse. Especially when it's a whole page of spam about video games (?) that's irrelevant to the project.

Nevertheless, I hope DuckDuckGo can learn from the more fruitful responses, one of which has been authored by @calimeroteknik. I think DuckDuckGo's official response was quite... flattering to themselves, and their project. I'll seek to be more wary of products such as this in the future.