Pgloader: KABOOM! on DBF load.

[richardkmichael]

• [x] pgloader --version

pgloader version "3.5.1~devel"
compiled with SBCL 1.4.7

• [x] did you test a fresh compile from the source tree?

No. There are only a few new commits on master after my version of pgloader. None seem related to this issue, so I did not build from source.

• [x] did you search for other similar issues?
  Yes, none seem related.

• [x] how can I reproduce the bug?

I am trying to load this DBF file, Easement.dbf.

I created the database myself with $ createdb easements.

I read the DBF section of the pgloader tutorial, and copied the LOAD file, with modification as follows:

LOAD DBF
    FROM ./Easement.dbf
    INTO postgresql:///easements
    WITH truncate, create table
    SET client_encoding TO 'latin1';

• [x] pgloader output you obtain

I ran $ pgloader dbf.load and obtained:

2018-05-31T20:02:35.022000+01:00 LOG Data errors in '/private/tmp/pgloader/'
2018-05-31T20:02:35.024000+01:00 LOG Parsing commands from file #P"/Users/testuser/easements/dbf.load"
2018-05-31T20:02:35.094000+01:00 WARNING pgloader always talk to PostgreSQL in utf-8, client_encoding has been forced to 'utf8'.
KABOOM!
FATAL error: NIL fell through ETYPECASE expression.
             Wanted one of (STRING CONS PGLOADER.CATALOG:TABLE).
An unhandled error condition has been signalled:
   NIL fell through ETYPECASE expression.
   Wanted one of (STRING CONS PGLOADER.CATALOG:TABLE).




What I am doing here?

NIL fell through ETYPECASE expression.
Wanted one of (STRING CONS PGLOADER.CATALOG:TABLE).

• [x] How the data is different from what you expected, if relevant

I expected a table to be created with 2 rows.

To verify the DBF file is valid, I used pgdbf, which emits Postgres commands and SQL:

$ pgdbf Easement.dbf

BEGIN;
SET statement_timeout=60000; DROP TABLE IF EXISTS easement; SET statement_timeout=0;
CREATE TABLE easement (pin NUMERIC(11), designator VARCHAR(254), planno VARCHAR(30), admin_code VARCHAR(20), remarks VARCHAR(254), pcl_state VARCHAR(254), reg_pin VARCHAR(254));
\COPY easement FROM STDIN
1182172 SEWER R/W       06336   NO PLAN FOR PROPOSED SEWER EASEMENT (SEE SKETCH IN REG #14400)  ACTIVE  402525487
1077542 R/W 53114 CLSR ON   06336       ACTIVE  402525486
\.
COMMIT;

If I use this output with psql, pgdbf Easement.dbf | psql easements, the table is created with 2 rows, as expected.

[dimitri]

This should be fixed now, please try again and report!

[richardkmichael]

@dimitri

Thanks for the fast reply. :)

I am being lazy, trying to build HEAD using homebrew, but I get a "missing dependency" error.

I realize the error probably due to the homebrew formula, but I'll fix it if I can understand which dependency is missing.

Any hints from this output?

$ brew install --HEAD pgloader

# ..snip..

Last 15 lines from /Users/testuser/Library/Logs/Homebrew/pgloader/01.make:
pgloader-standalone
BUILDAPP=buildapp

buildapp    --require sb-posix --require sb-bsd-sockets --require sb-rotate-byte                        \
                       --sbcl sbcl                            \
                       --load-system pgloader               \
                       --load src/hooks.lisp                   \
                       --entry pgloader:main                   \
                       --dynamic-space-size 4096         \
                       --compress-core                    \
                       --output build/bin/pgloader
;; loading system "pgloader"
Fatal MISSING-DEPENDENCY:
  Component #:ZS3 not found, required by #<SYSTEM "pgloader">
make: *** [pgloader-standalone] Error 1

READ THIS: https://docs.brew.sh/Troubleshooting

Thanks!

[dimitri]

That must be https://github.com/xach/zs3. Thanks!

[richardkmichael]

Yes this fixes the problem, thanks! (Aside, I'll PR a pgloader brew formula updated to include zs3.)

[szydan]

For anyone hitting

Component #:ZS3 not found, required by #<SYSTEM "pgloader">

For me, it happens when I try to install with --HEAD, without --HEAD all is fine

[richardkmichael]

@szydan Yes, expected because it's recent versions of pgloader which have RedShift support. See: https://github.com/Homebrew/homebrew-core/pull/28602