Pgloader: Something wrong with AWS RDS URLs

[bkmeneguello]

• [ ] pgloader --version

  3.6.2~devel
  

• [ ] did you test a fresh compile from the source tree?

  No, I've used latest Docker image

• [ ] did you search for other similar issues?

• [ ] how can I reproduce the bug?

  pgloader -v mysql://xxxxx@xxxxxxx/xxxx postgresql://xxxxx.jadj77asdyha7.us-east-1.rds.amazonaws.com:postgres@postgres:5432/postgres

• [ ] pgloader output you obtain

  KABOOM!
  FATAL error: Failed to connect to pgsql at "xxxxx.jadj77asdyha7.us-eeast-11.rds.amazonaws.com" (port 5432) as user "postgres": Database error: Name service error in "getaddrinfo": -2 (Name or service not known)

Something gets weird with "us-east-1", it becomes "us-eeast-11". I dunno if it's just a log issue, but aliasing the URL to a shorter one, like just "postgres", it worked.

• [ ] data that is being loaded, if relevant

• [ ] How the data is different from what you expected, if relevant

[deepfryed]

Yeah got the same issue, long host names trips pgloaded. Works fine if I replace them with shorter aliases or IP

root@de0925db70e2:/# pgloader --debug migration.load 
pgloader version 3.6.2~devel
compiled with SBCL 1.4.16.debian
sb-impl::*default-external-format* :ANSI_X3.4-1968
tmpdir: #P"/tmp/pgloader/"
2020-02-15T00:22:35.012000Z NOTICE Starting pgloader, log system is ready.
2020-02-15T00:22:35.019000Z INFO Starting monitor
2020-02-15T00:22:35.022000Z LOG pgloader version "3.6.2~devel"
2020-02-15T00:22:35.043000Z INFO Parsed command:
LOAD DATABASE
  FROM mysql://testfeeds:****@testfeeds-db-china.chzjggrxsdis.rds.cn-north-1.amazonaws.com.cn:3306/testfeeds
  INTO postgresql://testfeeds:****@testfeeds-db-china-v2.chzjggrxsdis.rds.cn-north-1.amazonaws.com.cn:5432/testfeeds

WITH preserve index names,
     batch rows = 5000,
     prefetch rows = 5000,
     workers = 2,
     concurrency = 1 

BEFORE LOAD DO
$$ create schema if not exists public; $$;

KABOOM!
FATAL error: Failed to connect to pgsql at "testfeeds-ddb-cchina-vv2.chzjggrxsdis.rds.cn-nnorth-11.amazonaws.com.cn" (port 5432) as user "testfeeds": Database error: Name service error in "getaddrinfo": -2 (Name or service not known)
Date/time: 2020-02-15-00:22!

[svantevonerichsen6906]

Sorry if I just point out the obvious: it seems that any character after a dash in the URL gets doubled.

[deepfryed]

Yeah looks like it.

root@tardis:/# pgloader mysql://[email protected]/test postgres://[email protected]/test
2020-02-16T23:29:41.015000Z LOG pgloader version "3.6.2~devel"
2020-02-16T23:29:41.039000Z LOG Migrating from #<MYSQL-CONNECTION mysql://[email protected]:3306/test {1005EA4F93}>
2020-02-16T23:29:41.039000Z LOG Migrating into #<PGSQL-CONNECTION pgsql://[email protected]:5432/test {1005FE72E3}>
2020-02-16T23:29:41.076000Z ERROR mysql: Failed to connect to mysql at "some-hhost-nname.local" (port 3306) as user "dev": Condition USOCKET:NS-HOST-NOT-FOUND-ERROR was signalled.
2020-02-16T23:29:41.076000Z LOG report summary reset
       table name     errors       rows      bytes      total time
-----------------  ---------  ---------  ---------  --------------
  fetch meta data          0          0                     0.000s
-----------------  ---------  ---------  ---------  --------------
-----------------  ---------  ---------  ---------  --------------

[mrbschroeder]

I don't know enough lisp to fix it, but it looks like the but was introduced in d4da906

[svantevonerichsen6906]

Yep, that's it.

PARSER> (esrap:parse 'network-label-with-hyphen "abc-def")
"abc-ddef"
NIL
T

[svantevonerichsen6906]

You could extract the production for the positive lookahead:

(defrule hyphen-before-letter-or-digit
    (and #\- (& network-label-letters-digit))
  (:constant #\-))

(defrule network-label-with-hyphen
    (and network-label-letters-digit
         (+ (or hyphen-before-letter-or-digit
                network-label-letters-digit))
         (! #\-))
  (:text t))

I'm not sure where to put a test for this, though.

[svantevonerichsen6906]

Or avoid the lookahead entirely:

(defrule network-label-with-hyphen
    (and network-label-letters-digit
         (+ (or (and #\- network-label-letters-digit)
                network-label-letters-digit)))
  (:text t))

[dimitri]

Thanks @svantevonerichsen6906 for the fix! Do you want to have contributor access to the pgloader repository?

[svantevonerichsen6906]

Heh, thanks, but credit also goes to @mrbschroeder for pinning down the location of the problem.

As for access, I'm not sure whether I would actually exercise it.

[dimitri]

Still invited you in, we need to grow a team, all contributions are welcome!