Angular.js: Track Firebase hosting migration

[petebacondarwin]

• [x] @Narretz setup errors.anguarjs.org firebase redirects
• [x] @IgorMinar double check all certs
• [x] @petebacondarwin double check the old ngix config to ensure we haven't missed anything
• [x] @Narretz resolve code.angularjs.org timeout issues for large files
• [x] @IgorMinar update dns to point to the new sites
• [x] @IgorMinar update blog.angularjs.org to point to blogger or blog.angular.io
• [x] @petebacondarwin shut down old ci.angularjs.org jobs that were keeping the site up to date
• [ ] @Narretz remove old deployment code from angularjs.org project
• [ ] @petebacondarwin update hosting details doc
• [x] migrate ci.angularjs.org
  
  
  
  • [x] @petebacondarwin obtain and install a cert on the machine
  
  
  • [x] @IgorMinar update dns details ci.angularjs.org
  
  
• [x] @petebacondarwin shut down old cloud VMs
• [ ] @Narretz follow up on firebase quote limit issues in order to ensure that we don't get shut down
• [x] @Narretz code.angularjs.org robots.txt issues (see https://github.com/angular/angular.js/issues/16417#issuecomment-362948998)
• [x] @Narretz docs.angularjs.org sitemap and robots.txt issues (see https://github.com/angular/angular.js/issues/16432#issuecomment-362786501)
• [x] @Narretz restore escaped fragment indexing https://github.com/angular/angular.js/issues/16432

[IgorMinar]

@petebacondarwin thank you!

[IgorMinar]

@Narretz is it correct that the timeout issue is resolved?

[Narretz]

@IgorMinar yes the issue is resolved. It was an encoding issue not a file size issue. The zip files were additional gzipped for faster upload but this confused gcs / Firebase. I replaced all the available zips with their default form and now it works. The deployment has been updated too

[IgorMinar]

So I guess that means that we are good to go then. Right?

I book Wed 9-10am PT on my calendar to make this go live + sent out an invite to you two.

[IgorMinar]

@petebacondarwin what about blog.angularjs.org? have you thought about it? if we don't change anything, I believe that it will break because of the certs and the current proxy setup.

I think we'll need to do something like:

• https://support.google.com/domains/answer/6069231?hl=en
• and https://www.howtoshout.com/enable-https-blogger-blog-custom-domain/

[petebacondarwin]

@IgorMinar - yes we need to setup the DNS to CNAME to point to blogger and then set up the blog on Blogger to accept incoming connections from that domain - I believe there is now a setting that allows HTTPS. I don't have admin rights to do either of these two.

The alternative is to change blog.angularjs.org to point directly to blog.angular.io, which is not hosted on blogger but something else...

[IgorMinar]

I was thinking about this more and I think the lowest risk will be to just setup a redirect from blog.angularjs.org to blogger. The current DNS setup is messed up anyway and most urls on the web point to either https://blog.angularjs.org or http://angularjs.blogspot.com/<somepath>.html.

[petebacondarwin]

@IgorMinar that would be fine by me but I think it will need, yet another, Firebase project setup for the subdomain, right?

[IgorMinar]

all but ci.angularjs.org are switched now. @petebacondarwin we need to decide what to do with the jenkins instance. without the global proxy we no longer have a signed certificate for it. Could you look into getting a cert from lets encrypt and installing it on the box?

[IgorMinar]

haha.. so much for "we need to decide". above is my proposal. the only two alternatives is to use a self-signed cert (bad ux) or switch to http (bad security). @petebacondarwin if there are other options please propose them.

[petebacondarwin]

Can we continue to use the IP address directly for Jenkins, @IgorMinar?

[IgorMinar]

@petebacondarwin we can. but that's far from ideal :)

I update the DNS to point to the new ip address which uses self-signed cert. You can only bypass the cert if you visit the domain with a browser that previously hasn't visited the site because of angularjs.org previously serving Strict-Transport-Security: max-age=16070400; includeSubDomains; preload header.

So we'll either fix the cert or wait half a year for us to be able to bypass the cert warning.

If you don't care, we can just document this and move on...

[IgorMinar]

I updated the hosting doc. let's move on.. :)

[IgorMinar]

@IgorMinar that would be fine by me but I think it will need, yet another, Firebase project setup for the subdomain, right?

@petebacondarwin was this referring to dashboard.angularjs.org or something else?

[petebacondarwin]

@IgorMinar - I was referring to setting up a "redirect" from blog.angularjs.org to blogger. A HTTP redirect (rather than a DNS based solution) would require a configured server.

[petebacondarwin]

If you don't care, we can just document this and move on...

I don't care :-)

[IgorMinar]

On Thu, Jan 25, 2018 at 11:57 PM Pete Bacon Darwin [e-mail]
wrote:

@IgorMinar https://github.com/igorminar - I was referring to setting up
a "redirect" from blog.angularjs.org to blogger. A HTTP redirect (rather
than a DNS based solution) would require a configured server.

I see. This turned out not to be necessary. Blogger now has support for
vanity urls and https so it all worked out.

[petebacondarwin]

I have disabled the following jobs on ci.angularjs.org:

• angularjs.org - this was building and publishing the home page when it changed
• angular-sites - this was building and updating the nginx site config when it changed

[Narretz]

Something that's not in the list is that we are no longer publishing the build artefacts to the code.angularjs.org github repository, since we deploy to Firebase directly to Travis. I mentioned this before, but I just wanted to make sure that we are on the same page about this.

[IgorMinar]

there issues with code.angularjs.org robots.txt:

The following urls/patterns are not configured correctly:

• https://code.angularjs.org/0.9.7/docs-0.9.7/ - not covered by robots.txt Disallow: /*docs/ pattern
• https://code.angularjs.org/snapshot/docs/js/all-versions-data.js - should be allowed otherwise the js-enabled crawler can't index docs.angularjs.org

[IgorMinar]

I updated the checklist above with the summary of new issues that need followup.

[Narretz]

@IgorMinar

• you mentioned that we need to allow https://code.angularjs.org/snapshot/docs/js/all-versions-data.js in the code.angularjs.org folders, but do we actually want to index the versions on code? Don't we just want to index docs.angularjs.org? (which probably needs the same fix)
• I assume it's not only all-versions-data.js (the first file alphabetically), it's all javascript assets?
• wrt the sitemap.xml, we haven't had one for a long time, see https://github.com/angular/angular.js/issues/8740. Obviously now is a good time to add one. @petebacondarwin is there something in dgeni that we can use to create a sitemap?

[Narretz]

I've copied the sitemap generation script from angular.io, and it looks good. I just need to filter out the example files.

[petebacondarwin]

I don't think we want to index any of code.angularjs.org

[Narretz]

Yeah, I realized versions-data.js is needed because it is loaded by docs.angularjs.org, so that it always has the most up to date list of versions.

[IgorMinar]

yup. you got it. sorry about forgetting to mention the reason - indeed the
file with version number is being loaded across the domains from
docs.angularjs.org and that's why it needs to be allowed.

On Mon, Feb 5, 2018 at 3:25 AM Martin Staffa [e-mail]
wrote:

Yeah, I realized versions-data.js is needed because it is loaded by
docs.angularjs.org, so that it always has the most up to date list of
versions.

—
You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub
https://github.com/angular/angular.js/issues/16417#issuecomment-363057205,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AANM6KobUbhFX0agp3h2-NgJkJk9f6evks5tRuUsgaJpZM4RlWE3
.

[Narretz]

The robots.txt at code and docs have been updated:
https://docs.angularjs.org/robots.txt
https://code.angularjs.org/robots.txt

Sitemap has been added to docs:
https://docs.angularjs.org/sitemap.xml

[IgorMinar]

are we sure that disallowing /partials/ and /img/ in docs.angularjs.org is
the right thing to do?

keep in mind that for google to be able to crawl docs.angularjs.org it
needs to be able to access all resources via both the non-js crawler (this
is why I'm concerned about /partials/) and js-crawler (this is why I'm
concerned about /img/).

all of this stuff is black magic as far as I'm concerned, so I don't know
for sure if these concerns are valid or not. we'll need to test this in
prod to see the results.

On Mon, Feb 5, 2018 at 1:37 PM Martin Staffa [e-mail]
wrote:

The robots.txt at code and docs have been updated:
https://docs.angularjs.org/robots.txt
https://code.angularjs.org/robots.txt

Sitemap has been added to docs:
https://docs.angularjs.org/sitemap.xml

—
You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub
https://github.com/angular/angular.js/issues/16417#issuecomment-363229751,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AANM6CBbo4I281pjAm8b0qVdBwNVZYXJks5tR3SugaJpZM4RlWE3
.

[petebacondarwin]

If we allow the partials then they become indexed, which means that the raw partial is visible in searches. I don't think it matters what we do with images - we should probably allow them so that people can search for images in the guides.