Three.js: Docs: Facilitate indexing of the docs by certain search engines

Created on 3 Apr 2020  路  14Comments  路  Source: mrdoob/three.js

Description of the problem

When I want to look at the docs for a particular Three.js class, I type 'three.js [classname]` into my search engine of choice (DuckDuckGo). The only relevant result is the docs, but not the specific docs page.
image
I then have to click the docs and search for the class again.

Feature request

If possible, facilitate indexing of all three.js docs pages. This may mean switching from '/docs/#api/etc' to '/docs/api/etc' urls, which could replace or forward to the existing pages.

Search engine

  • [ ] All of them
  • [ ] Google (only one I tried that worked)
  • [x] DuckDuckGo
  • [x] Bing
  • [x] Yahoo

All 14 comments

Well, yes. When using my search engine of choice, it looks as expected:

image

When clicking on the search result, you directly land on the Matrix4 page.

Sorry, I clearly don't vote to make the proposed change and break all existing URLs just because search engines can't do their job right.

Does DuckDuckGo support hashbangs, or can we not use the URL fragment at all?

This isn't necessarily a large change, it might just mean adding a simple redirect.

This isn't necessarily a large change, it might just mean adding a simple redirect.

How would this look like? Can you configure redirects with GitHub pages?

It used to be the case that #! was recommended with Google Search, and # basically wasn't supported. Redirecting from one to the other is just a JavaScript change.

Now it doesn't matter with Google Search. I don't know about DuckDuckGo. But you're right, if the hashbang thing doesn't work and we need to use real /docs/foo paths, we can't do this on GitHub Pages.

Something I probably should have mentioned is that you already have URLs without the '#' that can be found with a slightly longer search term.
image

However, their title of "[name]" makes them difficult to see at first glance.
image

If you just fixed these pages to have the proper title, maybe duck duck go and the other search engines I mentioned would index them more readily.

This is a related issue that may explain the mentioned pages: https://github.com/mrdoob/three.js/issues/10937

So their crawler doesn't support javascript?

I can't speak to what DuckDuckGo or any other engine does internally, and I'm not suggesting you break any links or do the impossible. It seems like it would be possible to fix the missing [name].
https://github.com/mrdoob/three.js/blob/3c13d929f8d9a02c89f010a487e73ff0e57437c4/docs/api/en/math/Matrix4.html#L11

@mrdoob If you would be open to a PR, I would be willing to go through and add a to each html file or replace the [name] (which is currently templated in by https://github.com/mrdoob/three.js/blob/dev/docs/page.js#L59)</p> </div> <div class="card-footer"> <div class="row"> <div class="col"> <img src="https://avatars0.githubusercontent.com/u/20015102?v=4&s=40" style="width:20px; height:20px;" class="mr-2 rounded float-left" alt="finnbear picture"> <strong>finnbear</strong> <span class="text-muted ml-1">on 4 Apr 2020</span> </div> <div class="col text-right"> </div> </div> </div> </div> <div class="card card-custom mb-4"> <div class="card-body pt-3 pb-3 markdown"> <p>Have you considered to report this issue to DuckDuckGo first? Fixing the issue on their side seems more appropriate. It then works for all websites using the current docs approach.</p> </div> <div class="card-footer"> <div class="row"> <div class="col"> <img src="https://avatars2.githubusercontent.com/u/12612165?v=4&s=40" style="width:20px; height:20px;" class="mr-2 rounded float-left" alt="Mugen87 picture"> <strong>Mugen87</strong> <span class="text-muted ml-1">on 4 Apr 2020</span> </div> <div class="col text-right"> </div> </div> </div> </div> <div class="card card-custom mb-4"> <div class="card-body pt-3 pb-3 markdown"> <blockquote> <p><strong>@mrdoob</strong> If you would be open to a PR, I would be willing to go through and add a <title> to each html file</p> </blockquote> <p>Sure! As long as you are willing to continue updating all the relevant files for the rest of your life 馃榿</p> <p>More seriously, this is an issue with DuckDuckGo's crawler.</p> </div> <div class="card-footer"> <div class="row"> <div class="col"> <img src="https://avatars0.githubusercontent.com/u/97088?v=4&s=40" style="width:20px; height:20px;" class="mr-2 rounded float-left" alt="mrdoob picture"> <strong>mrdoob</strong> <span class="text-muted ml-1">on 4 Apr 2020</span> </div> <div class="col text-right"> 👍<span class="ml-2 mr-3">1</span> </div> </div> </div> </div> <div class="card card-custom mb-4"> <div class="card-body pt-3 pb-3 markdown"> <p>What was I thinking? There is a much easier, standards-compliant way to fix this issue: adding a sitemap. Please review #19037 :smiley: </p> <p>(And yes, I contacted DuckDuckGo. Their automated system said they had a high volume of messages and would take time to respond. And this is a problem with the other search engines I mentioned, not just DuckDuckGo)</p> </div> <div class="card-footer"> <div class="row"> <div class="col"> <img src="https://avatars0.githubusercontent.com/u/20015102?v=4&s=40" style="width:20px; height:20px;" class="mr-2 rounded float-left" alt="finnbear picture"> <strong>finnbear</strong> <span class="text-muted ml-1">on 4 Apr 2020</span> </div> <div class="col text-right"> 👀<span class="ml-2 mr-3">1</span> 👍<span class="ml-2 mr-3">1</span> </div> </div> </div> </div> <div class="card card-custom mb-4"> <div class="card-body pt-3 pb-3 markdown"> <p>It's the same on Bing: </p> <p><img loading="lazy" src="https://user-images.githubusercontent.com/5307958/78518756-8fb14200-77eb-11ea-9368-4fea7225a045.png" alt="bing" /></p> <p>Maybe DDG uses Bing?</p> </div> <div class="card-footer"> <div class="row"> <div class="col"> <img src="https://avatars2.githubusercontent.com/u/5307958?v=4&s=40" style="width:20px; height:20px;" class="mr-2 rounded float-left" alt="looeee picture"> <strong>looeee</strong> <span class="text-muted ml-1">on 6 Apr 2020</span> </div> <div class="col text-right"> 👍<span class="ml-2 mr-3">1</span> </div> </div> </div> </div> <div class="card card-custom mb-4"> <div class="card-body pt-3 pb-3 markdown"> <blockquote> <p>Maybe DDG uses Bing?</p> </blockquote> <p>Well, technically yes.</p> <blockquote> <p>DuckDuckGo's results are a compilation of "over 400" sources, including Yahoo! Search BOSS, Wolfram Alpha, Bing, Yandex, its own web crawler (the DuckDuckBot) and others.</p> </blockquote> <p>From: <a rel="nofollow noopener" target="_blank" href="https://en.wikipedia.org/wiki/DuckDuckGo">https://en.wikipedia.org/wiki/DuckDuckGo</a></p> <p>I don't think either Bing or DuckDuckGo is to blame though. By using google analytics, you may be giving Google an advantage when it comes to indexing the page as the user sees it :wink: </p> <p>Having a proper sitemap is a good first step for any search engine, hence the PR. By having the sitemap point to the urls with the '#', and thus the proper title, instead of the raw html pages, the [name] issue may be fixed.</p> <p>In the unlikely event that it doesn't fix the [name] issue, some sort of <title> would have to be baked into the HTML. I'm still willing to do that, unless the resulting PR would be rejected if it lacked a mechanism to keep the titles 100% up to date (by contrast, this PR includes code to autogenerate the sitemap)</p> </div> <div class="card-footer"> <div class="row"> <div class="col"> <img src="https://avatars0.githubusercontent.com/u/20015102?v=4&s=40" style="width:20px; height:20px;" class="mr-2 rounded float-left" alt="finnbear picture"> <strong>finnbear</strong> <span class="text-muted ml-1">on 6 Apr 2020</span> </div> <div class="col text-right"> </div> </div> </div> </div> <div class="card card-custom mb-4"> <div class="card-body pt-3 pb-3 markdown"> <blockquote> <p>More seriously, this is an issue with DuckDuckGo's crawler.</p> </blockquote> <p>Yes, I don't think a PR makes sense in this context. Hence, a sitemap is also not necessary.</p> <p>Again, this needs to be fixed on search-engine website.</p> </div> <div class="card-footer"> <div class="row"> <div class="col"> <img src="https://avatars2.githubusercontent.com/u/12612165?v=4&s=40" style="width:20px; height:20px;" class="mr-2 rounded float-left" alt="Mugen87 picture"> <strong>Mugen87</strong> <span class="text-muted ml-1">on 6 Apr 2020</span> </div> <div class="col text-right"> </div> </div> </div> </div> <div class="card card-custom mb-4"> <div class="card-body pt-3 pb-3 markdown"> <p>Closing, see <a rel="nofollow noopener" target="_blank" href="https://github.com/mrdoob/three.js/issues/19031#issuecomment">https://github.com/mrdoob/three.js/issues/19031#issuecomment</a>-609000135.</p> </div> <div class="card-footer"> <div class="row"> <div class="col"> <img src="https://avatars2.githubusercontent.com/u/12612165?v=4&s=40" style="width:20px; height:20px;" class="mr-2 rounded float-left" alt="Mugen87 picture"> <strong>Mugen87</strong> <span class="text-muted ml-1">on 1 May 2020</span> </div> <div class="col text-right"> </div> </div> </div> </div> </div> <div class="col-12"> <div class="card card-custom mb-4"> <div class="card-body pt-3 pb-3 markdown text-center helpful"> <div class="title">Was this page helpful?</div> <div class="mt-1" onMouseLeave="rating(593578365, 0);"> <i class="fas fa-star inactive" id="star-1" onMouseOver="rating(593578365, 1);" onclick="rate(593578365, 1);"></i> <i class="fas fa-star inactive" id="star-2" onMouseOver="rating(593578365, 2);" onclick="rate(593578365, 2);"></i> <i class="fas fa-star inactive" id="star-3" onMouseOver="rating(593578365, 3);" onclick="rate(593578365, 3);"></i> <i class="fas fa-star inactive" id="star-4" onMouseOver="rating(593578365, 4);" onclick="rate(593578365, 4);"></i> <i class="fas fa-star inactive" id="star-5" onMouseOver="rating(593578365, 5);" onclick="rate(593578365, 5);"></i> </div> <div class="description text-small"><span id="rating-val">0</span> / 5 - <span id="rating-count">0</span> ratings</div> </div> </div> <div class="mb-4"> <ins class="adsbygoogle" style="display:block" data-ad-client="ca-pub-3835332123789605" data-ad-slot="3452512275" data-ad-format="auto" data-full-width-responsive="true"></ins> <script> (adsbygoogle = window.adsbygoogle || []).push({}); </script> </div> </div> </div> </div> <div class="col-12 col-lg-4"> <div id="ph-above-related"></div> <div class="card card-custom issue-box"> <div class="card-body pt-3 pb-5"> <h2 class="mb-4">Related issues</h2> <div> <strong> <a href="/three-js/42137533/blender-exported-model-deforms-when-animation-is-applied">Blender exported model deforms when animation is applied</a> </strong> </div> <div class="text-muted text-small mt-2"> <img src="https://avatars3.githubusercontent.com/u/4322567?v=4&s=40" style="width:20px; height:20px;" class="mr-2 rounded float-left" alt="jlaquinte picture"> <strong class="pr-1" dir="ltr">jlaquinte</strong>  路  <span class="px-1" dir="ltr">3</span><span>Comments</span> </div> <hr /> <div> <strong> <a href="/three-js/189120025/setfaceculling-not-working-for-the-renderer">setFaceCulling not working for the renderer</a> </strong> </div> <div class="text-muted text-small mt-2"> <img src="https://avatars3.githubusercontent.com/u/9973182?v=4&s=40" style="width:20px; height:20px;" class="mr-2 rounded float-left" alt="boyravikumar picture"> <strong class="pr-1" dir="ltr">boyravikumar</strong>  路  <span class="px-1" dir="ltr">3</span><span>Comments</span> </div> <hr /> <div> <strong> <a href="/three-js/205172883/another-fbx-loading-issue">Another FBX Loading Issue</a> </strong> </div> <div class="text-muted text-small mt-2"> <img src="https://avatars2.githubusercontent.com/u/232101?v=4&s=40" style="width:20px; height:20px;" class="mr-2 rounded float-left" alt="filharvey picture"> <strong class="pr-1" dir="ltr">filharvey</strong>  路  <span class="px-1" dir="ltr">3</span><span>Comments</span> </div> <hr /> <div> <strong> <a href="/three-js/3295461/how-to-clone-models-in-program">How to clone models in program</a> </strong> </div> <div class="text-muted text-small mt-2"> <img src="https://avatars3.githubusercontent.com/u/1419566?v=4&s=40" style="width:20px; height:20px;" class="mr-2 rounded float-left" alt="fuzihaofzh picture"> <strong class="pr-1" dir="ltr">fuzihaofzh</strong>  路  <span class="px-1" dir="ltr">3</span><span>Comments</span> </div> <hr /> <div> <strong> <a href="/three-js/223315374/outlinepass-doesn-t-work-with-pixel-ratio-2">OutlinePass doesn't work with pixel ratio 2</a> </strong> </div> <div class="text-muted text-small mt-2"> <img src="https://avatars2.githubusercontent.com/u/5774080?v=4&s=40" style="width:20px; height:20px;" class="mr-2 rounded float-left" alt="jack-jun picture"> <strong class="pr-1" dir="ltr">jack-jun</strong>  路  <span class="px-1" dir="ltr">3</span><span>Comments</span> </div> </div> </div> <div class="sticky-top pt-4"> <ins class="adsbygoogle" style="display:block" data-ad-client="ca-pub-3835332123789605" data-ad-slot="3919948963" data-ad-format="auto" data-full-width-responsive="true"></ins> <script> (adsbygoogle = window.adsbygoogle || []).push({}); </script> <div id="ph-below-related-2" class="mt-4"> <ins class="adsbygoogle" style="display:block" data-ad-client="ca-pub-3835332123789605" data-ad-slot="3919948963" data-ad-format="auto" data-full-width-responsive="true"></ins> <script> (adsbygoogle = window.adsbygoogle || []).push({}); </script> </div> </div> </div> <div class="col-12 col-lg-4"> </div> </div> <div class="skyscraper-container"> <ins class="adsbygoogle" style="display:block" data-ad-client="ca-pub-3835332123789605" data-ad-slot="7879185320" data-ad-format="vertical" data-full-width-responsive="true"></ins> <script> (adsbygoogle = window.adsbygoogle || []).push({}); </script> </div> </div> <div class="mt-5 spacer"></div> <footer class="mt-5 pb-2 py-4 text-center mt-auto"> <div class="container"> <a class="navbar-brand logo mr-5" href="/"> <img src="/assets/img/logo.svg" width="40" height="40" alt="bleepingcoder logo"> bleeping<strong>coder</strong> </a> <div class="mt-4"> bleepingcoder.com uses publicly licensed GitHub information to provide developers around the world with solutions to their problems. We are not affiliated with GitHub, Inc. or with any developers who use GitHub for their projects. We do not host any of the videos or images on our servers. All rights belong to their respective owners. </div> <div> Source for this page: <a href="https://www.github.com/mrdoob/three.js/issues/19031" rel="nofollow noreferrer" target="_blank">Source</a> </div> </div> <hr class="mb-5 mt-5"> <div class="container"> <div class="row"> <div class="col-sm-4 col-lg mb-sm-0 mb-5"> <strong>Popular programming languages</strong> <ul class="list-unstyled mb-0 mt-2"> <li class="mb-2"> <a href="/python" dir="ltr">Python</a> </li> <li class="mb-2"> <a href="/javascript" dir="ltr">JavaScript</a> </li> <li class="mb-2"> <a href="/typescript" dir="ltr">TypeScript</a> </li> <li class="mb-2"> <a href="/cpp" dir="ltr">C++</a> </li> <li class="mb-2"> <a href="/csharp" dir="ltr">C#</a> </li> </ul> </div> <div class="col-sm-4 col-lg mb-sm-0 mb-5"> <strong>Popular GitHub projects</strong> <ul class="list-unstyled mb-0 mt-2"> <li class="mb-2"> <a href="/microsoft/vscode" dir="ltr">vscode</a> </li> <li class="mb-2"> <a href="/numpy/numpy" dir="ltr">numpy</a> </li> <li class="mb-2"> <a href="/ant-design/ant-design" dir="ltr">ant-design</a> </li> <li class="mb-2"> <a href="/mui-org/material-ui" dir="ltr">material-ui</a> </li> <li class="mb-2"> <a href="/vercel/next-js" dir="ltr">next.js</a> </li> </ul> </div> <div class="col-sm-4 col-lg mb-0"> <strong>More GitHub projects</strong> <ul class="list-unstyled mb-0 mt-2"> <li class="mb-2"> <a href="/rust-lang/rust" dir="ltr">rust</a> </li> <li class="mb-2"> <a href="/moment/moment" dir="ltr">moment</a> </li> <li class="mb-2"> <a href="/yarnpkg/yarn" dir="ltr">yarn</a> </li> <li class="mb-2"> <a href="/mozilla/pdf-js" dir="ltr">pdf.js</a> </li> <li class="mb-2"> <a href="/JuliaLang/julia" dir="ltr">julia</a> </li> </ul> </div> </div> </div> <hr class="mb-5 mt-5"> <div class="container text-muted"> 漏 2025 bleepingcoder.com - <a href="/bleeps" rel="nofollow">Contact</a><br /> By using our site, you acknowledge that you have read and understand our <a href="/cookies" rel="nofollow">Cookie Policy</a> and <a href="/privacy" rel="nofollow">Privacy Policy</a>. </div> </footer> <script src="https://code.jquery.com/jquery-3.5.1.slim.min.js" integrity="sha256-4+XzXVhsDmqanXGHaHvgh1gMQKX40OUvDEBTu8JcmNs=" crossorigin="anonymous"></script> <script async src="https://cdn.jsdelivr.net/npm/popper.js@1.16.0/dist/umd/popper.min.js" integrity="sha384-Q6E9RHvbIyZFJoft+2mJbHaEWldlvI9IOYy5n3zV9zzTtmI3UksdQRVvoxMfooAo" crossorigin="anonymous"></script> <script src="https://stackpath.bootstrapcdn.com/bootstrap/4.4.1/js/bootstrap.min.js" integrity="sha384-wfSDF2E50Y2D1uUdj0O3uMBJnjuUD4Ih7YwaYd1iqfktj0Uod8GCExl3Og8ifwB6" crossorigin="anonymous"></script> <!--<script defer type="text/javascript" src="//s7.addthis.com/js/300/addthis_widget.js#pubid=ra-5fb2db66acbd74b2"></script>--> <script type="text/javascript" src="/assets/js/main.js"></script> <script src="https://cdn.jsdelivr.net/gh/google/code-prettify@master/loader/run_prettify.js"></script></body> </html>