Contao: Crawler stops between 95 and 98%.

Created on 19 Feb 2020 · 25Comments · Source: contao/contao

Affected version(s)

4.9

Description

On my website the new crawler stops between 95 and 98%.
The following error message can be found in the log files.

[2020-02-19 09:02:23] request.CRITICAL: Uncaught PHP Exception InvalidArgumentException: "Unable to parse URI: http://" at /www/htdocs/vendor/nyholm/psr7/src/Uri.php line 51 {"exception":"[object] (InvalidArgumentException(code: 0): Unable to parse URI: http:// at /www/htdocs//vendor/nyholm/psr7/src/Uri.php:51)"} []

How to reproduce

I can show you my installation.

bug

Source

christianbarkowsky

All 25 comments

Can you paste the complete stack trace?

Toflar on 19 Feb 2020

Uncaught` PHP Exception InvalidArgumentException: "Unable to parse URI: http://" at /www/htdocs/vendor/nyholm/psr7/src/Uri.php line 51
Hide context    Hide trace

[▼
  "exception" => InvalidArgumentException {#2097 ▼
    #message: "Unable to parse URI: http://"
    #code: 0
    #file: "/www/htdocs/vendor/nyholm/psr7/src/Uri.php"
    #line: 51
    trace: {▼
    /www/htdocs/vendor/nyholm/psr7/src/Uri.php:51 {▶}
      /www/htdocs/vendor/terminal42/escargot/src/Subscriber/HtmlCrawlerSubscriber.php:55 {▼
        Terminal42\Escargot\Subscriber\HtmlCrawlerSubscriber->onLastChunk(CrawlUri $crawlUri, ResponseInterface $response, ChunkInterface $chunk): void …
        › $link = new Link($node, (string) $crawlUri->getUri()->withPath('')->withQuery('')->withFragment(''));
        › $uri = new Uri($link->getUri());
        › 
        arguments: {▶}
      }
      /www/htdocs/vendor/terminal42/escargot/src/Escargot.php:449 {▼
        Terminal42\Escargot\Escargot->processResponseChunk(ResponseInterface $response, ChunkInterface $chunk): void …
        › if (SubscriberInterface::DECISION_NEGATIVE !== $needsContentDecision) {
                ›     $subscriber->onLastChunk($crawlUri, $response, $chunk);
        › }
        arguments: {▼
          $crawlUri: Terminal42\Escargot\CrawlUri {#829 …}
                    $response: Symfony\Component\HttpClient\Response\CurlResponse {#734 …}
                        $chunk: Symfony\Component\HttpClient\Chunk\LastChunk {#744 …}
                        }
      }
      /www/htdocs/vendor/terminal42/escargot/src/Escargot.php:407 {▼
        Terminal42\Escargot\Escargot->processResponses(array $responses): void …
        › foreach ($this->getClient()->stream($responses) as $response => $chunk) {
                            ›     $this->processResponseChunk($response, $chunk);
        › }
        arguments: {▼
          $response: Symfony\Component\HttpClient\Response\CurlResponse {#734 …}
                                $chunk: Symfony\Component\HttpClient\Chunk\LastChunk {#744 …}
                                }
      }
      /www/htdocs/vendor/terminal42/escargot/src/Escargot.php:315 {▼
        Terminal42\Escargot\Escargot->crawl(): void …
        › 
        ›     $this->processResponses($responses);
        › }
        arguments: {▼
          $responses: [ …5]
        }
      }
      /www/htdocs/vendor/contao/core-bundle/src/Resources/contao/classes/Crawl.php:171 {▼
        Contao\Crawl->run() …
        › // Start crawling
        › $escargot->crawl();
        › 
      }
      /www/htdocs/vendor/contao/core-bundle/src/Resources/contao/modules/ModuleMaintenance.php:49 {▼
        Contao\ModuleMaintenance->compile() …
        › 
        › $buffer = $this->$callback->run();
        › 
      }
      /www/htdocs/vendor/contao/core-bundle/src/Resources/contao/classes/BackendModule.php:92 {▼
        Contao\BackendModule->generate() …
        › $this->Template = new BackendTemplate($this->strTemplate);
        › $this->compile();
        › 
      }
      /www/htdocs/vendor/contao/core-bundle/src/Resources/contao/classes/Backend.php:434 {▼
        Contao\Backend->getBackendModule($module, PickerInterface $picker = null) …
        › 
        › \t$this->Template->main .= $objCallback->generate();
        › }
      }
      /www/htdocs/vendor/contao/core-bundle/src/Resources/contao/controllers/BackendMain.php:155 {▼
        Contao\BackendMain->run() …
        › 
        › $this->Template->main .= $this->getBackendModule(Input::get('do'), $picker);
        › $this->Template->title = $this->Template->headline;
        arguments: {▼
          $module: "maintenance"
          $picker: null
        }
      }
      /www/htdocs/vendor/contao/core-bundle/src/Controller/BackendController.php:48 {▼
        Contao\CoreBundle\Controller\BackendController->mainAction(): Response …
        › 
        ›     return $controller->run();
        › }
      }
      /www/htdocs/vendor/symfony/http-kernel/HttpKernel.php:146 {▼
        Symfony\Component\HttpKernel\HttpKernel->handleRaw(Request $request, int $type = self::MASTER_REQUEST): Response …
        › // call controller
        › $response = $controller(...$arguments);
        › 
      }
      /www/htdocs/vendor/symfony/http-kernel/HttpKernel.php:68 {▼
        Symfony\Component\HttpKernel\HttpKernel->handle(Request $request, $type = HttpKernelInterface::MASTER_REQUEST, $catch = true) …
        › try {
                        ›     return $this->handleRaw($request, $type);
        › } catch (\Exception $e) {
                        arguments: {▼
          $request: Symfony\Component\HttpFoundation\Request {#6 …}
                                $type: 1
        }
      }
                        /www/htdocs/vendor/symfony/http-kernel/Kernel.php:201 {▼
        Symfony\Component\HttpKernel\Kernel->handle(Request $request, $type = HttpKernelInterface::MASTER_REQUEST, $catch = true) …
        › try {
                                ›     return $this->getHttpKernel()->handle($request, $type, $catch);
        › } finally {
                                arguments: {▼
          $request: Symfony\Component\HttpFoundation\Request {#6 …}
                                        $type: 1
          $catch: true
        }
      }
                                /www/htdocs/web/index.php:31 {▼
        require …
        › 
        › $response = $kernel->handle($request);
        › $response->send();
        arguments: {▶}
      }
      /www/htdocs/web/app.php:4 {▼
        › // Backwards compatibility
        › require __DIR__.'/index.php';
        › 
        arguments: {▼
          "/www/htdocs/web/index.php"
        }
      }
    }
  }
]

{▼
/www/htdocs/vendor/nyholm/psr7/src/Uri.php:51 {▼
    Nyholm\Psr7\Uri->__construct(string $uri = '') …
    › if (false === $parts = \parse_url($uri)) {
        ›     throw new \InvalidArgumentException("Unable to parse URI: $uri");
    › }
  }
  /www/htdocs/vendor/terminal42/escargot/src/Subscriber/HtmlCrawlerSubscriber.php:55 {▼
    Terminal42\Escargot\Subscriber\HtmlCrawlerSubscriber->onLastChunk(CrawlUri $crawlUri, ResponseInterface $response, ChunkInterface $chunk): void …
    › $link = new Link($node, (string) $crawlUri->getUri()->withPath('')->withQuery('')->withFragment(''));
    › $uri = new Uri($link->getUri());
    › 
    arguments: {▼
      $uri: "http://"
    }
  }
  /www/htdocs/vendor/terminal42/escargot/src/Escargot.php:449 {▼
    Terminal42\Escargot\Escargot->processResponseChunk(ResponseInterface $response, ChunkInterface $chunk): void …
    › if (SubscriberInterface::DECISION_NEGATIVE !== $needsContentDecision) {
        ›     $subscriber->onLastChunk($crawlUri, $response, $chunk);
    › }
    arguments: {▼
      $crawlUri: Terminal42\Escargot\CrawlUri {#829 …}
            $response: Symfony\Component\HttpClient\Response\CurlResponse {#734 …}
                $chunk: Symfony\Component\HttpClient\Chunk\LastChunk {#744 …}
                }
  }
  /www/htdocs/vendor/terminal42/escargot/src/Escargot.php:407 {▼
    Terminal42\Escargot\Escargot->processResponses(array $responses): void …
    › foreach ($this->getClient()->stream($responses) as $response => $chunk) {
                    ›     $this->processResponseChunk($response, $chunk);
    › }
    arguments: {▼
      $response: Symfony\Component\HttpClient\Response\CurlResponse {#734 …}
                        $chunk: Symfony\Component\HttpClient\Chunk\LastChunk {#744 …}
                        }
  }
  /www/htdocs/vendor/terminal42/escargot/src/Escargot.php:315 {▼
    Terminal42\Escargot\Escargot->crawl(): void …
    › 
    ›     $this->processResponses($responses);
    › }
    arguments: {▼
      $responses: [ …5]
    }
  }
  /www/htdocs/vendor/contao/core-bundle/src/Resources/contao/classes/Crawl.php:171 {▼
    Contao\Crawl->run() …
    › // Start crawling
    › $escargot->crawl();
    › 
  }
  /www/htdocs/vendor/contao/core-bundle/src/Resources/contao/modules/ModuleMaintenance.php:49 {▼
    Contao\ModuleMaintenance->compile() …
    › 
    › $buffer = $this->$callback->run();
    › 
  }
  /www/htdocs/vendor/contao/core-bundle/src/Resources/contao/classes/BackendModule.php:92 {▼
    Contao\BackendModule->generate() …
    › $this->Template = new BackendTemplate($this->strTemplate);
    › $this->compile();
    › 
  }
  /www/htdocs/vendor/contao/core-bundle/src/Resources/contao/classes/Backend.php:434 {▼
    Contao\Backend->getBackendModule($module, PickerInterface $picker = null) …
    › 
    › \t$this->Template->main .= $objCallback->generate();
    › }
  }
  /www/htdocs/vendor/contao/core-bundle/src/Resources/contao/controllers/BackendMain.php:155 {▼
    Contao\BackendMain->run() …
    › 
    › $this->Template->main .= $this->getBackendModule(Input::get('do'), $picker);
    › $this->Template->title = $this->Template->headline;
    arguments: {▼
      $module: "maintenance"
      $picker: null
    }
  }
  /www/htdocs/vendor/contao/core-bundle/src/Controller/BackendController.php:48 {▼
    Contao\CoreBundle\Controller\BackendController->mainAction(): Response …
    › 
    ›     return $controller->run();
    › }
  }
  /www/htdocs/vendor/symfony/http-kernel/HttpKernel.php:146 {▼
    Symfony\Component\HttpKernel\HttpKernel->handleRaw(Request $request, int $type = self::MASTER_REQUEST): Response …
    › // call controller
    › $response = $controller(...$arguments);
    › 
  }
  /www/htdocs/vendor/symfony/http-kernel/HttpKernel.php:68 {▼
    Symfony\Component\HttpKernel\HttpKernel->handle(Request $request, $type = HttpKernelInterface::MASTER_REQUEST, $catch = true) …
    › try {
                ›     return $this->handleRaw($request, $type);
    › } catch (\Exception $e) {
                arguments: {▼
      $request: Symfony\Component\HttpFoundation\Request {#6 …}
                        $type: 1
    }
  }
                /www/htdocs/vendor/symfony/http-kernel/Kernel.php:201 {▼
    Symfony\Component\HttpKernel\Kernel->handle(Request $request, $type = HttpKernelInterface::MASTER_REQUEST, $catch = true) …
    › try {
                        ›     return $this->getHttpKernel()->handle($request, $type, $catch);
    › } finally {
                        arguments: {▼
      $request: Symfony\Component\HttpFoundation\Request {#6 …}
                                $type: 1
      $catch: true
    }
  }
                        /www/htdocs/web/index.php:31 {▼
    require …
    › 
    › $response = $kernel->handle($request);
    › $response->send();
    arguments: {▼
      $request: Symfony\Component\HttpFoundation\Request {#6 …}
                                }
  }
  /www/htdocs/web/app.php:4 {▶}

christianbarkowsky on 19 Feb 2020

You have an empty <a href="http://"></a> link somewhere on your page which is invalid. Can you try to update all dependencies so terminal42/escargot is updated to 0.5.2. It should be fixed with https://github.com/terminal42/escargot/commit/3b16bad749fb87d26035cf25ddb1cb3ef97ffa00. The debug log should then also tell you where that link was found and that it couldn't be added to the queue :)

Toflar on 20 Feb 2020

👍1

After the update comes a new error.

[2020-02-20 17:19:55] request.CRITICAL: Uncaught PHP Exception Doctrine\DBAL\Exception\DriverException: "An exception occurred while executing 'SELECT uri, level, processed, found_on, tags FROM tl_crawl_queue WHERE (job_id = ?) AND (processed = ?) ORDER BY id ASC LIMIT 1 OFFSET 508' with params ["c423a14e-233d-48a6-b291-155429e27422", 0]: SQLSTATE[HY000]: General error: 2006 MySQL server has gone away" at /www/htdocs/vendor/doctrine/dbal/lib/Doctrine/DBAL/Driver/AbstractMySQLDriver.php line 106 {"exception":"[object] (Doctrine\\DBAL\\Exception\\DriverException(code: 0): An exception occurred while executing 'SELECT uri, level, processed, found_on, tags FROM tl_crawl_queue WHERE (job_id = ?) AND (processed = ?) ORDER BY id ASC LIMIT 1 OFFSET 508' with params [\"c423a14e-233d-48a6-b291-155429e27422\", 0]:\n\nSQLSTATE[HY000]: General error: 2006 MySQL server has gone away at /www/htdocs/vendor/doctrine/dbal/lib/Doctrine/DBAL/Driver/AbstractMySQLDriver.php:106, Doctrine\\DBAL\\Driver\\PDOException(code: HY000): SQLSTATE[HY000]: General error: 2006 MySQL server has gone away at /www/htdocs/vendor/doctrine/dbal/lib/Doctrine/DBAL/Driver/PDOStatement.php:123, PDOException(code: HY000): SQLSTATE[HY000]: General error: 2006 MySQL server has gone away at /www/htdocs/vendor/doctrine/dbal/lib/Doctrine/DBAL/Driver/PDOStatement.php:121)"} []