Amphtml: `gulp dist` is timing out on Travis

Created on 13 Mar 2018 · 16Comments · Source: ampproject/amphtml

When gulp dist was last instrumented, it used to take ~13 mins to run on Travis during master builds. Since then, the running time went up to ~18 minutes, and as of today, it takes ~22 minutes. This is causing it to frequently time out on Travis, resulting in redness on master.

See https://travis-ci.org/ampproject/amphtml/jobs/352664009#L632 for a recent failure
See https://travis-ci.org/ampproject/amphtml/jobs/352471805#L639 for a build where gulp dist took ~27 mins to complete

Related to #10277

High Priority Bug infra

Source

rsimha

Most helpful comment

More extensions? Shouldn't nearly double the running time though. Separately, having our own metrics would be great to see trends over time.

I think @choumx is correct. Some recent extensions (amp-mathml.js, amp-fx-collection.js) are taking nearly a minute a piece.

For temporary relief, can we increase the timeout?

@erwinmombay perhaps we can do a deep dive here with a few folks to debug?

kristoferbaxter on 13 Mar 2018

👍2

All 16 comments

/cc @cramforce @choumx @aghassemi @lannka @kristoferbaxter for visibility, in case someone know why this is happening.

rsimha on 13 Mar 2018

More extensions? Shouldn't nearly double the running time though. Separately, having our own metrics would be great to see trends over time.

choumx on 13 Mar 2018

Here's a trace of gulp dist on my 6 core workstation. This should give an idea of the time for which each task remains queued. I see several that are queued for almost a minute, which should explain why gulp dist takes so long on a single core Travis VM.

rsimha@bobdylan (master):~/src/amphtml$ gulp dist --fortesting
[12:03:22] Using gulpfile ~/src/amphtml/gulpfile.js
[12:03:22] Starting 'update-packages'...
[12:03:23] All packages in node_modules are up to date.
[12:03:23] Finished 'update-packages' after 473 ms
[12:03:23] Starting 'dist'...
[12:03:23] Running dist. Press Ctrl + C to cancel...
[12:03:23] Building the runtime for local testing with the prod AMP config.
[12:03:23] ⤷ Use --config={canary|prod} with your gulp dist --fortesting command to specify which config to apply.
[12:03:23] Building all AMP extensions.
[12:03:23] ⤷ Use --noextensions to skip building extensions.
[12:03:23] ⤷ Use --extensions=amp-foo,amp-bar to choose which extensions to build.
[12:03:23] ⤷ Use --extensions=minimal_set to build just the extensions needed to load article.amp.html.
[12:03:23] Recompiled CSS in amp.css (239 ms)
[12:03:23] Recompiled CSS in amp-access (293 ms)
[12:03:23] Recompiled CSS in amp-access-scroll (292 ms)
[12:03:23] Recompiled CSS in amp-ad (291 ms)
[12:03:23] Recompiled CSS in amp-apester-media (292 ms)
[12:03:23] Recompiled CSS in amp-byside-content (290 ms)
[12:03:23] Recompiled CSS in amp-consent (287 ms)
[12:03:23] Recompiled CSS in amp-document-recommendations (287 ms)
[12:03:23] Recompiled CSS in amp-fit-text (287 ms)
[12:03:23] Recompiled CSS in amp-form (287 ms)
[12:03:23] Recompiled CSS in amp-fx-flying-carpet (286 ms)
[12:03:23] Recompiled CSS in amp-gwd-animation (286 ms)
[12:03:23] Recompiled CSS in amp-image-lightbox (286 ms)
[12:03:23] Recompiled CSS in amp-instagram (284 ms)
[12:03:23] Recompiled CSS in amp-lightbox (284 ms)
[12:03:23] Recompiled CSS in amp-live-list (286 ms)
[12:03:23] Recompiled CSS in amp-mathml (286 ms)
[12:03:23] Recompiled CSS in amp-pinterest (285 ms)
[12:03:23] Recompiled CSS in amp-sidebar (283 ms)
[12:03:23] Recompiled CSS in amp-selector (273 ms)
[12:03:23] Recompiled CSS in amp-web-push (272 ms)
[12:03:23] Recompiled CSS in amp-image-viewer (270 ms)
[12:03:23] Recompiled CSS in amp-user-notification (270 ms)
[12:03:23] Recompiled CSS in amp-viz-vega (270 ms)
[12:03:23] Recompiled CSS in amp-access-laterpay (297 ms)
[12:03:23] Recompiled CSS in amp-app-banner (296 ms)
[12:03:23] Recompiled CSS in amp-carousel (295 ms)
[12:03:23] Recompiled CSS in amp-playbuzz (286 ms)
[12:03:23] Recompiled CSS in amp-sticky-ad (285 ms)
[12:03:23] Recompiled CSS in amp-subscriptions (272 ms)
[12:03:23] Recompiled CSS in amp-lightbox-gallery (289 ms)
[12:03:23] Recompiled CSS in amp-social-share (300 ms)
[12:03:23] Recompiled CSS in amp-date-picker (435 ms)
[12:03:24] Recompiled CSS in amp-story (701 ms)
[12:03:24] Recompiled CSS in amp-access (221 ms)
[12:03:24] Recompiled CSS in amp-access-scroll (221 ms)
[12:03:24] Recompiled CSS in amp-ad (221 ms)
[12:03:24] Recompiled CSS in amp-apester-media (221 ms)
[12:03:24] Recompiled CSS in amp-byside-content (218 ms)
[12:03:24] Recompiled CSS in amp-consent (217 ms)
[12:03:24] Recompiled CSS in amp-document-recommendations (217 ms)
[12:03:24] Recompiled CSS in amp-fit-text (217 ms)
[12:03:24] Recompiled CSS in amp-form (217 ms)
[12:03:24] Recompiled CSS in amp-fx-flying-carpet (217 ms)
[12:03:24] Recompiled CSS in amp-gwd-animation (216 ms)
[12:03:24] Recompiled CSS in amp-image-lightbox (216 ms)
[12:03:24] Recompiled CSS in amp-instagram (216 ms)
[12:03:24] Recompiled CSS in amp-lightbox (216 ms)
[12:03:24] Recompiled CSS in amp-live-list (218 ms)
[12:03:24] Recompiled CSS in amp-mathml (218 ms)
[12:03:24] Recompiled CSS in amp-pinterest (218 ms)
[12:03:24] Recompiled CSS in amp-sidebar (216 ms)
[12:03:24] Recompiled CSS in amp-selector (208 ms)
[12:03:24] Recompiled CSS in amp-web-push (208 ms)
[12:03:24] Recompiled CSS in amp-image-viewer (207 ms)
[12:03:24] Recompiled CSS in amp-user-notification (207 ms)
[12:03:24] Recompiled CSS in amp-viz-vega (206 ms)
[12:03:24] Recompiled CSS in amp-access-laterpay (226 ms)
[12:03:24] Recompiled CSS in amp-app-banner (224 ms)
[12:03:24] Recompiled CSS in amp-carousel (223 ms)
[12:03:24] Recompiled CSS in amp-playbuzz (219 ms)
[12:03:24] Recompiled CSS in amp-sticky-ad (218 ms)
[12:03:24] Recompiled CSS in amp-subscriptions (209 ms)
[12:03:24] Recompiled CSS in amp-lightbox-gallery (221 ms)
[12:03:24] Processed 3p/nameframe.max.html (609 ms)
[12:03:24] Processed 3p/frame.max.html (623 ms)
[12:03:24] Recompiled CSS in amp-social-share (620 ms)
[12:03:25] Recompiled CSS in amp-date-picker (737 ms)
[12:03:25] Recompiled CSS in amp-story (975 ms)
[12:03:25] Copied build/css/v0.css to dist/v0.css (987 ms)
[12:03:34] Minified ampcontext-lib.js (10.317 s)
[12:03:34] Minified iframe-transport-client-lib.js (10.509 s)
[12:03:36] Minified integration.js (11.891 s)
[12:03:38] Minified amp.js (14.103 s)
[12:03:45] Minified inabox-host.js (21.311 s)
[12:03:48] Minified install-alp.js (24.339 s)
[12:03:48] Minified amp-shadow.js (24.457 s)
[12:03:48] Minified amp-inabox.js (24.460 s)
[12:03:52] Minified examiner.js (27.941 s)
[12:03:55] Minified kill.js (31.686 s)
[12:03:56] Minified shell.js (32.013 s)
[12:03:56] Minified core.js (32.180 s)
[12:04:01] Minified web-worker.js (37.437 s)
[12:04:05] Minified amp-accordion.js (40.858 s)
[12:04:05] Minified amp-3q-player.js (41.469 s)
[12:04:06] Minified amp-ad-network-adsense-impl.js (41.893 s)
[12:04:12] Minified amp-ad-network-adzerk-impl.js (48.203 s)
[12:04:15] Minified amp-ad-network-doubleclick-impl.js (51.449 s)
[12:04:16] Minified amp-ad-network-fake-impl.js (52.480 s)
[12:04:17] Minified amp-ad-network-triplelift-impl.js (53.199 s)
[12:04:23] Minified amp-ad-network-cloudflare-impl.js (59.206 s)
[12:04:25] Minified amp-ad-exit.js (1.706 s)
[12:04:26] Minified amp-ad-network-gmossp-impl.js (1.858 s)
[12:04:28] Minified amp-analytics.js (3.764 s)
[12:04:31] Minified amp-anim.js (7.528 s)
[12:04:34] Minified amp-audio.js (10.320 s)
[12:04:36] Minified amp-animation.js (12.476 s)
[12:04:37] Minified amp-auto-ads.js (13.384 s)
[12:04:43] Minified amp-bind.js (19.300 s)
[12:04:44] Minified amp-brid-player.js (20.127 s)
[12:04:45] Minified amp-brightcove.js (20.986 s)
[12:04:46] Minified amp-kaltura-player.js (21.996 s)
[12:04:52] Minified amp-compare-slider.js (27.941 s)
[12:04:52] Minified amp-call-tracking.js (28.217 s)
[12:04:53] Minified amp-crypto-polyfill.js (29.177 s)
[12:04:56] Minified amp-dailymotion.js (31.977 s)
[12:05:00] Minified amp-dynamic-css-classes.js (36.199 s)
[12:05:00] Minified amp-experiment.js (36.527 s)
[12:05:02] Minified amp-facebook.js (37.758 s)
[12:05:05] Minified amp-facebook-comments.js (41.428 s)
[12:05:09] Minified amp-facebook-page.js (45.170 s)
[12:05:10] Minified amp-facebook-like.js (46.039 s)
[12:05:10] Minified amp-font.js (46.179 s)
[12:05:14] Minified amp-fx-collection.js (50.017 s)
[12:05:18] Minified amp-hulu.js (54.273 s)
[12:05:18] Minified amp-gfycat.js (54.688 s)
[12:05:19] Minified amp-gist.js (55.657 s)
[12:05:24] Minified amp-iframe.js (59.738 s)
[12:05:27] Minified amp-imgur.js (2.831 s)
[12:05:28] Minified amp-ima-video.js (4.092 s)
[12:05:28] Minified amp-install-serviceworker.js (4.673 s)
[12:05:32] Minified amp-izlesene.js (8.449 s)
[12:05:34] Minified amp-jwplayer.js (10.601 s)
[12:05:37] Minified amp-list.js (13.312 s)
[12:05:38] Minified amp-mustache.js (13.874 s)
[12:05:42] Minified amp-nexxtv-player.js (18.017 s)
[12:05:43] Minified amp-o2-player.js (18.922 s)
[12:05:46] Minified amp-reach-player.js (22.121 s)
[12:05:47] Minified amp-ooyala-player.js (22.959 s)
[12:05:51] Minified amp-reddit.js (27.136 s)
[12:05:51] Minified amp-riddle-quiz.js (27.423 s)
[12:05:54] Minified amp-share-tracking.js (30.609 s)
[12:05:55] Minified amp-soundcloud.js (31.099 s)
[12:05:58] Minified amp-story-auto-ads.js (34.319 s)
[12:05:59] Minified amp-springboard-player.js (35.150 s)
[12:06:04] Minified amp-position-observer.js (40.236 s)
[12:06:04] Minified amp-wistia-player.js (40.553 s)
[12:06:08] Minified amp-slides.js (43.926 s)
[12:06:08] Minified amp-subscriptions-google.js (44.164 s)
[12:06:13] Minified amp-timeago.js (49.383 s)
[12:06:14] Minified amp-twitter.js (50.212 s)
[12:06:16] Minified amp-vimeo.js (52.345 s)
[12:06:16] Minified amp-vine.js (52.535 s)
[12:06:22] Minified amp-google-vrview-image.js (57.941 s)
[12:06:22] Minified amp-viewer-integration.js (58.385 s)
[12:06:25] Minified amp-vk.js (1.016 s)
[12:06:26] Minified amp-video.js (2.426 s)
[12:06:32] Minified amp-access.js (7.916 s)
[12:06:32] Minified amp-youtube.js (8.271 s)
[12:06:34] Minified amp-access-scroll.js (9.666 s)
[12:06:37] Minified amp-ad.js (12.958 s)
[12:06:41] Minified amp-apester-media.js (16.799 s)
[12:06:41] Minified amp-byside-content.js (16.949 s)
[12:06:42] Minified amp-consent.js (18.189 s)
[12:06:50] Minified amp-fit-text.js (26.041 s)
[12:06:50] Minified amp-document-recommendations.js (26.336 s)
[12:06:50] Minified amp-form.js (26.418 s)
[12:06:51] Minified amp-fx-flying-carpet.js (27.024 s)
[12:06:59] Minified amp-gwd-animation.js (35.296 s)
[12:07:00] Minified amp-instagram.js (35.587 s)
[12:07:00] Minified amp-image-lightbox.js (35.774 s)
[12:07:00] Minified amp-lightbox.js (36.210 s)
[12:07:09] Minified amp-pinterest.js (44.875 s)
[12:07:09] Minified amp-live-list.js (45.167 s)
[12:07:09] Minified amp-mathml.js (45.263 s)
[12:07:09] Minified amp-sidebar.js (45.341 s)
[12:07:18] Minified amp-selector.js (53.999 s)
[12:07:19] Minified amp-user-notification.js (54.558 s)
[12:07:19] Minified amp-image-viewer.js (54.856 s)
[12:07:19] Minified amp-web-push.js (54.966 s)
[12:07:27] Minified amp-viz-vega.js (3.117 s)
[12:07:27] Minified amp-access-laterpay.js (3.287 s)
[12:07:28] Minified amp-app-banner.js (3.674 s)
[12:07:28] Minified amp-carousel.js (4.281 s)
[12:07:36] Minified amp-sticky-ad.js (11.919 s)
[12:07:36] Minified amp-playbuzz.js (12.097 s)
[12:07:37] Minified amp-subscriptions.js (12.671 s)
[12:07:38] Minified amp-lightbox-gallery.js (13.562 s)
[12:07:44] Minified amp-social-share.js (19.779 s)
[12:07:46] Minified experiments.max.js (21.929 s)
[12:07:49] Minified amp-story.js (24.513 s)
[12:07:50] Minified amp-date-picker.js (25.069 s)
[12:07:52] Minified amp-web-push-helper-frame.js (27.261 s)
[12:07:54] Minified amp-web-push-permission-dialog.js (28.799 s)
[12:07:55] Minified amp-login-done-0.1.max.js (30.006 s)
[12:07:55] No configs found in dist/v0.js
[12:07:55] Enabled local development mode in dist/v0.js
[12:07:55] Wrote prod AMP config to dist/v0.js
[12:07:55] No configs found in dist.3p/current-min/f.js
[12:07:55] Enabled local development mode in dist.3p/current-min/f.js
[12:07:55] Wrote prod AMP config to dist.3p/current-min/f.js
[12:07:55] Finished 'dist' after 4.53 min

rsimha on 13 Mar 2018

@rsimha Is there any historical data to compare against?

kristoferbaxter on 13 Mar 2018

@kristoferbaxter See the linked issue #10277 at the top. There are several linked Travis builds. You can also look at the past few days worth of Travis push builds on master at https://travis-ci.org/ampproject/amphtml/builds (however far back it will let you go)

rsimha on 13 Mar 2018

Thanks! I'll take a look.

kristoferbaxter on 13 Mar 2018

More extensions? Shouldn't nearly double the running time though. Separately, having our own metrics would be great to see trends over time.

I think @choumx is correct. Some recent extensions (amp-mathml.js, amp-fx-collection.js) are taking nearly a minute a piece.

For temporary relief, can we increase the timeout?

@erwinmombay perhaps we can do a deep dive here with a few folks to debug?

kristoferbaxter on 13 Mar 2018

👍2

Deep dive done earlier today. @erwinmombay is working on the closure compiler upgrade in #12558, which should set us on the path to a single invocation of the closure compiler. Any other work than can be done on the side will be spun off into separate issues.

rsimha on 13 Mar 2018

👍1

This is a high priority issue but it hasn't been updated in awhile. @kristoferbaxter Do you have any updates?