Shardingsphere: Parameterized ON DUPLICATE KEY UPDATE in insert statement doesn't work on MySQL proxy

[sandynz]

Bug Report

Which version of ShardingSphere did you use?

5.0.0-RC1-SNAPSHOT

Which project did you use? ShardingSphere-JDBC or ShardingSphere-Proxy?

ShardingSphere-Proxy

Expected behavior

Sql could be executed correctly, just like MySQL 5.7 server.

Actual behavior

Exception thrown:

Exception in thread "main" java.sql.SQLException: Statement parameter 5 not set.
    at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:965)
    at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:898)
    at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:887)
    at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:861)
    at com.mysql.jdbc.ServerPreparedStatement.serverExecute(ServerPreparedStatement.java:1174)
    at com.mysql.jdbc.ServerPreparedStatement.executeInternal(ServerPreparedStatement.java:787)
    at com.mysql.jdbc.PreparedStatement.executeUpdateInternal(PreparedStatement.java:2133)
    at com.mysql.jdbc.PreparedStatement.executeUpdateInternal(PreparedStatement.java:2067)
    at com.mysql.jdbc.PreparedStatement.executeLargeUpdate(PreparedStatement.java:5175)
    at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2052)

Reason analyze (If you can)

mysql-connector-java-5.1.48
ServerPreparedStatement.java, serverPrepare(String sql)
                Buffer prepareResultPacket = mysql.sendCommand(MysqlDefs.COM_PREPARE, sql, null, false, characterEncoding, 0);

                if (this.connection.versionMeetsMinimum(4, 1, 1)) {
                    // 4.1.1 and newer use the first byte as an 'ok' or 'error' flag, so move the buffer pointer past it to start reading the statement id.
                    prepareResultPacket.setPosition(1);
                } else {
                    // 4.1.0 doesn't use the first byte as an 'ok' or 'error' flag
                    prepareResultPacket.setPosition(0);
                }

                this.serverStatementId = prepareResultPacket.readLong();
                this.fieldCount = prepareResultPacket.readInt();
                this.parameterCount = prepareResultPacket.readInt();
                this.parameterBindings = new BindValue[this.parameterCount];
this.parameterCount is 5, which is from proxy server side, but it's 4 in fact

org.apache.shardingsphere.proxy.frontend.mysql.command.query.binary.prepare.MySQLComStmtPrepareExecutor.java, execute()
        SQLStatement sqlStatement = schema.getRuntimeContext().getSqlParserEngine().parse(packet.getSql(), true);
        if (!MySQLComStmtPrepareChecker.isStatementAllowed(sqlStatement)) {
            result.add(new MySQLErrPacket(++currentSequenceId, MySQLServerErrorCode.ER_UNSUPPORTED_PS));
            return result;
        }
        int parametersCount = sqlStatement.getParameterCount();
parametersCount is 5 after parsing

Steps to reproduce the behavior, such as: SQL to execute, sharding rule configuration, when exception occur etc.

Run sql via jdbc:

jdbcUrl:
jdbc:mysql://127.0.0.1:3307/sharding_db?serverTimezone=UTC&useSSL=false&useServerPrepStmts=true&cachePrepStmts=true

try (PreparedStatement statement = connection.prepareStatement(
        "insert into t_order (order_id, user_id, status) values (?, ?, ?) ON DUPLICATE KEY UPDATE status = ?")) {
    statement.setInt(1, orderId);
    statement.setInt(2, userId);
    statement.setString(3, status);
    statement.setString(4, "DUPLICATED");
    statement.executeUpdate();
}

Example codes for reproduce this issue (such as a github link).

[strongduanmu]

@sandynz Hi, Can you provide your table schema and sharding configuration? I have used the master branch to test, the INSERT INTO ... ON DUPLICATE KEY UPDATE ... statement can be executed normally.

[sandynz]

@sandynz Hi, Can you provide your table schema and sharding configuration? I have used the master branch to test, the INSERT INTO ... ON DUPLICATE KEY UPDATE ... statement can be executed normally.

Hi @strongduanmu , I tested it on lastest master branch again, commit: bfb1486549e9908f8784210fe2100f1c5fdd212d (2020-08-19), run org.apache.shardingsphere.proxy.Bootstrap, it doesn't work.

My config-sharding.yaml

schemaName: sharding_db

dataSourceCommon:
  username: root
  password: test
  connectionTimeoutMilliseconds: 30000
  idleTimeoutMilliseconds: 60000
  maxLifetimeMilliseconds: 1800000
  maxPoolSize: 50
  minPoolSize: 1
  maintenanceIntervalMilliseconds: 30000

dataSources:
  ds:
    url: jdbc:mysql://localhost:3306/demo_ds_1?serverTimezone=UTC&useSSL=false&useUnicode=true&characterEncoding=UTF-8

rules:
  - !SHARDING
    tables:
      t_order:
        actualDataNodes: ds.t_order_${0..1}
        tableStrategy:
          standard:
            shardingColumn: order_id
            shardingAlgorithmName: t_order_inline
        keyGenerateStrategy:
          column: order_id
          keyGeneratorName: snowflake
      t_order_item:
        actualDataNodes: ds.t_order_item_${0..1}
        tableStrategy:
          standard:
            shardingColumn: order_id
            shardingAlgorithmName: t_order_item_inline
        keyGenerateStrategy:
          column: order_item_id
          keyGeneratorName: snowflake
    bindingTables:
      - t_order,t_order_item
    broadcastTables:
      - t_address

    shardingAlgorithms:
      t_order_inline:
        type: INLINE
        props:
          algorithm.expression: t_order_${order_id % 2}
      t_order_item_inline:
        type: INLINE
        props:
          algorithm.expression: t_order_item_${order_id % 2}

    keyGenerators:
      snowflake:
        type: SNOWFLAKE
        props:
          worker.id: 123

My database:

mysql> use demo_ds_1
Database changed

CREATE TABLE IF NOT EXISTS t_address (address_id BIGINT NOT NULL, address_name VARCHAR(100) NOT NULL, PRIMARY KEY (address_id));
CREATE TABLE IF NOT EXISTS `t_order_0` (`order_id` INT NOT NULL, `user_id` INT NOT NULL, `status` VARCHAR(45) NULL, PRIMARY KEY (`order_id`));
CREATE TABLE IF NOT EXISTS `t_order_1` (`order_id` INT NOT NULL, `user_id` INT NOT NULL, `status` VARCHAR(45) NULL, PRIMARY KEY (`order_id`));
CREATE TABLE IF NOT EXISTS `t_order_item_0` (`order_item_id` INT NOT NULL, `order_id` INT NOT NULL, `user_id` INT NOT NULL, `status` VARCHAR(45) NULL, PRIMARY KEY (`order_item_id`));
CREATE TABLE IF NOT EXISTS `t_order_item_1` (`order_item_id` INT NOT NULL, `order_id` INT NOT NULL, `user_id` INT NOT NULL, `status` VARCHAR(45) NULL, PRIMARY KEY (`order_item_id`));

mysql> show tables;
+---------------------+
| Tables_in_demo_ds_1 |
+---------------------+
| t_address           |
| t_order_0           |
| t_order_1           |
| t_order_item_0      |
| t_order_item_1      |
+---------------------+
5 rows in set (0.00 sec)

My client test code snippet:

public static void main(String[] args) throws Exception {
    final boolean useServerPrepStmts = true;
    try (Connection connection = DriverManager.getConnection("jdbc:mysql://127.0.0.1:3307/sharding_db?serverTimezone=UTC&useSSL=false" + (useServerPrepStmts ? "&useServerPrepStmts=true&cachePrepStmts=true" : ""), "root", "root")) {
        int orderId = 1, userId = 2;
        String status = "PAID";
        try (PreparedStatement statement = connection.prepareStatement("insert into t_order (order_id, user_id, status) values (?, ?, ?) ON DUPLICATE KEY UPDATE status = ?")) {
            statement.setInt(1, orderId);
            statement.setInt(2, userId);
            statement.setString(3, status);
            statement.setString(4, "DUPLICATED");
            statement.executeUpdate();
        }
    }
}

[strongduanmu]

@sandynz Thank you for your reply, I will test it again!

[strongduanmu]

@sandynz Thank you for the example, and I have reproduced this exception. The reason for this problem is that the value in the on duplicate key statement is parsed twice, therefore the parametersCount is greater than the actual value by 1.

@Override
public ASTNode visitOnDuplicateKeyClause(final OnDuplicateKeyClauseContext ctx) {
    Collection<AssignmentSegment> columns = new LinkedList<>();
    for (AssignmentContext each : ctx.assignment()) {
        columns.add((AssignmentSegment) visit(each));
        visit(each.assignmentValue());
    }
    return new OnDuplicateKeyColumnsSegment(ctx.getStart().getStartIndex(), ctx.getStop().getStopIndex(), columns);
}

I will fix this problem as soon as possible.