Kafka-connect-jdbc: Postgresql sink constructs invalid upsert sql when pk.fields include all fields

[caili314]

io.confluent.connect.jdbc.sink.dialect.PostgreSqlDialect.getUpsertQuery()

change

builder.append(") ON CONFLICT (");
joinToBuilder(builder, ",", keyCols, escaper());
builder.append(") DO UPDATE SET ");
joinToBuilder(
    builder,
    ",",
    cols,
    new StringBuilderUtil.Transform<String>() {
      @Override
      public void apply(StringBuilder builder, String col) {
        builder.append(escaped(col)).append("=EXCLUDED.").append(escaped(col));
      }
    }
);

to

builder.append(") ON CONFLICT (");
joinToBuilder(builder, ",", keyCols, escaper());
if (cols.size() > 0) {
  builder.append(") DO UPDATE SET ");
  joinToBuilder(
      builder,
      ",",
      cols,
      new StringBuilderUtil.Transform<String>() {
        @Override
        public void apply(StringBuilder builder, String col) {
          builder.append(escaped(col)).append("=EXCLUDED.").append(escaped(col));
        }
      });
} else {
  builder.append(") DO NOTHING");
}

[rhauch]

@caili314 thanks. any interest in contributing a pull request?

[hashhar]

@rhauch I can take a shot at this since we are facing this issue internally. Will update once I have something working.

[hashhar]

You can track my progress at https://github.com/hashhar/kafka-connect-jdbc/tree/fix-401.

[hashhar]

I am unable to get the build working even after following the steps in https://github.com/confluentinc/kafka-connect-jdbc/wiki/FAQ.

https://github.com/confluentinc/common fails to build with the following error

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] assembly-plugin-boilerplate ........................ SUCCESS [ 14.187 s]
[INFO] Build Tools ........................................ SUCCESS [ 12.950 s]
[INFO] common ............................................. SUCCESS [02:01 min]
[INFO] utils .............................................. SUCCESS [ 10.356 s]
[INFO] metrics ............................................ SUCCESS [ 10.492 s]
[INFO] config ............................................. SUCCESS [  5.404 s]
[INFO] common-logging ..................................... FAILURE [ 19.570 s]
[INFO] confluent-log4j-extensions ......................... SKIPPED
[INFO] confluent-log4j2-extensions ........................ SKIPPED
[INFO] package ............................................ SKIPPED
[INFO] Common ............................................. SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 03:37 min
[INFO] Finished at: 2019-02-13T16:23:56+05:30
[INFO] Final Memory: 28M/300M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project common-logging: Could not resolve dependencies for
project io.confluent:common-logging:jar:5.3.0-SNAPSHOT: The following artifacts could not be
resolved: org.apache.kafka:connect-api:jar:2.3.0-SNAPSHOT, org.apache.kafka:connect-json:jar:2.3.0-SNAPSHOT:
Could not find artifact org.apache.kafka:connect-api:jar:2.3.0-SNAPSHOT in confluent (http://packages.confluent.io/maven/) -> [Help 1]

[OneCricketeer]

SNAPSHOT builds are not available in http://packages.confluent.io/maven.

Have you built Kafka source code successfully for the 2.3.0-SNAPSHOT?

[hashhar]

Yeah, Kafka builds successfully from the trunk branch and I can see that gradle.properties indeed contains 2.3.0-SNAPSHOT as the version. Do I have to do something after ./gradlew installAll to install the artifacts into my local maven repo?

[hashhar]

Got it to work. Needed to nuke my local maven repo first. I'll be sending a PR shortly.

[hashhar]

Should I add a separate test function for it or include the tests within the upsert() test at https://github.com/confluentinc/kafka-connect-jdbc/blob/master/src/test/java/io/confluent/connect/jdbc/dialect/PostgreSqlDatabaseDialectTest.java#L229 ?

[hashhar]

@rhauch I have the PR at https://github.com/confluentinc/kafka-connect-jdbc/pull/605.

[hashhar]

@cyrusv @mohnishbasha @rhauch Did anyone get a chance to look at this?

[mohnishbasha]

AK 2.3.0 is not released yet.

[cyrusv]

Resolved with https://github.com/confluentinc/kafka-connect-jdbc/pull/605

[reneveyj]

Any workaround before this comes to production?