Presto: Release notes for 0.209

[dain]

Andrii Rosa

• [x] all checked
• 08-23 07:26:06 20cffc4ef6 Add wrapped Boolean benchmarks
• 08-23 07:26:06 a64563ec1f Implement equality comparisons projection benchmark
• 08-23 08:05:14 37aaf49785 Use local variable instead of dup instruction in InCodeGenerator
• 08-23 12:04:51 e16cb85cbf Change EQUAL and NOT_EQUAL implementation signatures
• 08-23 12:04:52 226c9541f5 Implement nullable EQUAL and NOT_EQUAL for ARRAY type
• 08-23 12:04:52 2afb4c27ab Change equals semantincs for null in MAP
• 08-23 12:04:52 9f4cd60522 Implement nullable EQUAL and NOT_EQUAL for ROW type

• 08-23 12:11:05 b6fb1451f8 Support IN predicate for complex type values with nulls

  Dain Sundstrom

• [x] all checked

• 08-09 11:30:06 19b272185b Add NoOpTransactionManager for workers
• 08-09 11:30:06 a627f4230f Extract TransactionManager interface

• 08-14 19:41:17 de4679af77 Use field names to map field to underlying data for structs in ORC

  David Phillips

• [x] all checked

• 08-13 12:40:17 b600533621 Fix formatting in PushAggregationThroughOuterJoin
• 08-13 12:50:18 9e674fb1f6 Update to Airbase 83
• 08-13 12:50:18 cf79df3966 Simplify annotations
• 08-13 12:50:18 dbc25b6343 Add fall through comments for switch statement
• 08-13 12:50:42 4a4ab61fe0 Move stats calculator binding to separate module
• 08-13 12:51:37 b7482630c7 Only bind stats calculator on coordinator
• 08-13 13:01:06 68283b0f67 Cleanup statement resource executors on shutdown
• 08-13 13:01:06 89dba434b9 Remove unused statement resource thread pools from workers
• 08-13 13:08:24 5d211f4f3c Cleanup binding of failure detector
• 08-13 13:24:37 c1287aa36c Only bind discovery server on coordinator
• 08-19 21:46:50 91524c2351 Update to ASM 6.2.1 and bytecode 1.1
• 08-20 08:25:27 25d1ecd0c4 Completely remove SHOW PARTITIONS
• 08-20 10:26:56 a28fcad239 Add workaround for Mac JDK IPv6 bug for CLI and JDBC
• 08-20 23:00:29 3911f79541 Fix data loss when writing sorted Hive tables
• 08-20 23:00:29 46a36da31b Cleanup all temporary files when writing sorted Hive tables
• 08-23 14:18:35 67992400e7 Allow alternate implementations of HiveMetadata

• 08-24 11:13:42 ec2f4db199 Fix handling of thread interruption in JDBC driver

  Gerlou Shyy

• [x] all checked

• 08-13 16:42:42 a06780d815 Add FunctionInvokerProvider for requesting invocation convention

• 08-13 16:49:39 fcf66b6134 Rename pre-existing FunctionInvoker to InterpretedFunctionInvoker

  Grzegorz Kokosiński

• [x] all checked

• 08-08 22:47:28 08e7c37df6 Remove unnecessary copies in PruneUnreferencedOutputs
• 08-14 22:17:34 ff115ce115 Add Kudu connector
• 08-18 12:47:49 384ae61c64 Remove CoefficientBasedStatsCalculator
• 08-20 03:49:08 fbd4cb2647 Simplify assertions in SHOW TABLES test

• 08-22 04:02:00 d7ccd894f8 Remove TableScanNode originalConstraint

  Haozhun Jin

• [x] all checked

• 08-09 17:29:08 8fcb2a393d Add UnknownType.writeBoolean because some logic relies on it
• 08-09 17:29:08 ca6cf5743e Remove special handling of UNKNOWN in min_by/max_by
• 08-14 11:07:39 28d830dc2c Rename SqlScalarFunctionBuilder to PolymorphicScalarFunctionBuilder
• 08-14 11:07:39 738fa51841 Remove unused predicate feature from PolymorphicScalarFunction
• 08-15 14:44:22 81f28fd75b Move TestBlockAndPositionNullConvention to scalar package
• 08-15 14:44:23 762a117a17 Fix handling of Object type in annotation-based scalar
• 08-15 14:44:23 a998082c81 Add constructor dependency sanity check to ParametricScalarImplementation
• 08-15 14:46:06 99545bf618 Fix delay in sending noMoreSplits for scan node in colocated join
• 08-16 10:55:20 c12f84e917 Fix EmptySplit handling during bucket-by-bucket execution
• 08-18 15:43:42 8ccfcacfbe Minor refactor in SqlTaskExecution
• 08-18 15:43:42 b35d611adc Improve abstraction of SourcePartitionedScheduler
• 08-18 15:43:42 e1fbcb01f5 Remove unused method in SourcePartitionedScheduler
• 08-19 00:12:46 f0f2730783 Determine grouped execution on an individual scan node basis
• 08-19 00:12:47 020b1ce004 Reduce method overloads in PlanPrinter
• 08-19 00:12:47 beee9e848f Print grouped execution decision for individual Scan node in EXPLAIN

• 08-20 20:46:40 a9ace9e65f Fix truncated print for ProjectionNode in PlanPrinter

  Jiexi Lin

• [x] all checked

• 08-24 15:31:14 7546f9198c Add per table column_ranges system table in Raptor

  Karol Sobczak

• [x] all checked

• 08-09 05:50:34 5053352096 Remove redundent is decimal check
• 08-09 05:50:34 92f05c6be9 Simplify decimal type test
• 08-17 08:27:26 20d222e2e9 Add WorkProcessor#flatten method
• 08-17 08:27:26 8dc337ebaa Add Transformation#process javadoc
• 08-17 08:27:26 8e1852839d Remove redundant "else"
• 08-21 02:12:55 2696592f92 Better estimate NDVs and range in subtractColumnStats
• 08-21 02:12:55 3834510653 Use lambda instead of expression
• 08-21 02:12:55 6c0c84bbf6 Make tests stats fields in TestFilterStatsCalculator
• 08-21 02:12:55 707871aa58 Prevent nulls fraction to be negative in subtractColumnStats
• 08-21 02:12:55 8d004628cc Mark subtractColumnStats as @Deprecated as it semantics is undefined
• 08-21 08:17:12 1dc4ecc122 Dispose resources in WorkProcessorUtils when they are no longer needed

• 08-21 08:17:12 e1ab5bf7d5 Add WorkProcessor#transformProcessor method

  Maria Basmanova

• [x] all checked

• 08-09 08:13:05 c8ebb19972 Add the geometry_union_agg spatial aggregation function
• 08-10 10:49:51 2c6ba08e37 Check MarkDistinctSymbols#distinctSymbols is not empty
• 08-10 10:49:51 575a9a6be5 Remove unnecessary remote exchange from scalar correlated subquery plan
• 08-15 17:29:50 731eb9a97c Add convex_hull_agg function
• 08-16 08:27:46 dbe03b43bd Document geometry_union_agg function
• 08-16 13:21:40 cff67783a7 Move BenchmarkSpatialJoin to presto-geospatial

• 08-16 16:48:33 7ec5210b77 Add bing_tiles_around variant taking radius in kilometers

  Martin Traverso

• [x] all checked

• 08-13 11:51:58 662af7a9f1 Remove explicit grouping sets list from aggregation node
• 08-13 12:06:20 4bafb48856 Parse grouping columns as generic expressions
• 08-13 17:10:01 6ae9e8cf4a Compute covariance, correlation and regression in numerically stable manner
• 08-17 14:19:46 f9de0bd032 Fix incorrect predicate pushdown through empty grouping set
• 08-22 11:28:36 14f2ee4e7d Tighten assertion for parsing failure in TestSqlEnvironmentConfig
• 08-22 11:28:36 20f5ec2963 Rename singleExpression to standaloneExpression
• 08-22 11:28:36 30d455ec98 Implement recursion-free ATN simulator
• 08-22 11:28:36 8e04402761 Log raised during error handling
• 08-22 11:28:36 999e0aa185 Improve parsing error message
• 08-22 11:28:36 f18d0e9c5c Handle end of stream at the top-level rule
• 08-22 11:28:36 f5790cec97 Update to antlr 4.7.1

• 08-22 11:28:36 f95ed89a10 Use EOF-terminated rule for path specification

  Nezih Yigitbasi

• [x] all checked

• 08-09 10:17:52 10efb40e09 Don't poll/update the coordinator memory pools if not necessary
• 08-09 11:33:46 d47b9dc24b Move tagged memory allocations on memory pool update
• 08-09 14:04:19 7c1842671b Make moveQuery package private
• 08-09 14:04:19 e2e98ebbb3 Update tagged memory allocation API to make tags required
• 08-10 14:54:37 c25ab366bd Fix the race between close and destroy in the Driver
• 08-10 14:55:22 7bb2f0d6d3 Fix information schema predicate pushdown for views
• 08-10 17:08:58 70abfd42bc Fix live plan and stage kill button URL
• 08-10 17:10:59 88e4c1e935 Fix various typos
• 08-16 14:57:35 049fdce8d5 Add MockConnectorFactory.Builder to simplify construction

• 08-16 15:52:26 b4bbeedac0 Update docs with new memory-related configs

  Piotr Findeisen

• [x] all checked

• 08-11 12:55:45 005bc1fbdc Remove duplicate method
• 08-11 13:09:18 0895e77536 Report SemanticException for invalid TIME literal
• 08-13 23:49:24 1f2182c4a1 Update documentation about link to query's JSON
• 08-16 07:39:35 0964662615 Use Optionals in KafkaTopicDescription
• 08-16 07:39:35 54df0a482e Add basic Avro decoder for Kafka
• 08-16 07:39:35 9045ce4ce0 Add data schema to the decoder params
• 08-16 07:39:35 b93d25ad89 Add tests for Avro schema evolution in Kafka connector
• 08-16 07:39:35 cbdd36f1a3 Add documentation for Avro support in Kafka connector
• 08-16 07:39:35 dcb6dd83f7 Add TestingEnvironment
• 08-16 07:39:35 edabfc58ad Add Avro decoder to Redis
• 08-16 07:39:35 ff38359a1a Add support for structural types in Avro decoder for Kafka
• 08-20 00:27:52 656c2fcd3e Respect X-Forwarded-Proto header in / endpoint
• 08-22 00:27:10 687baebdd5 Ensure access control classes implement all methods
• 08-23 03:27:18 bac5959f4f Make HiveClientConfig.resourceConfigFiles non-nullable

• 08-27 02:08:15 abd65e37cc Replace usage of deprecated TreeTraverser

  Raghav Sethi

• [x] all checked

• 08-21 17:23:33 afe04dfba3 Add ManagedQueryExecution as root execution interface
• 08-21 18:07:14 59d297913f Fix error message in QuerySessionSupplier

• 08-24 16:17:43 e8f962d304 Separate EXCEEDED_MEMORY_LIMIT error into local and global

  Rebecca Schlussel

• [x] all checked

• 08-22 13:55:45 65ea3ca9d7 Don't check column access for aliases
• 08-22 13:55:45 f79aa806cd Track origin column for fields
• 08-27 07:51:19 0d20887425 Simplify formatFragment in PlanPrinter
• 08-27 07:51:19 16ab0c36e4 Show cost estimates in QueryMonitor
• 08-27 07:51:19 1b47fb3df7 Add exchange type as a field to RemoteSourceNode

• 08-27 07:51:19 789753c218 Propagate stats/costs across fragments in EXPLAIN

  Shixuan Fan

• [x] all checked

• 08-14 16:16:05 60de647d8d Implement EXPLAIN (TYPE IO, FORMAT JSON) statement

• 08-15 14:12:50 b813033e0a Use catalog field instead of hardcoded catalog

  Wenlei Xie

• [x] all checked

• 08-10 16:06:51 b3d9e15503 Log queryId when a small stripe get MAX_BYTES flush
• 08-17 09:35:45 f8ee82d868 Add block compact test cases
• 08-22 11:40:03 cdd2f673c2 Raise requried Java version to 8u151
• 08-22 14:27:05 05b66c0419 Avoid flushing small stripes due to DICTIONARY_FULL
• 08-22 14:27:05 40af0e803c Log queryId when a small stripe get DICTIONARY_FULL flush
• 08-23 18:11:13 c92cfef154 Add Hive connector procedure call to create empty partition
• 08-24 15:34:37 51ebbd09b8 Fix Hive smoke test when running in parallel
• 08-26 17:28:34 47355c83b3 Fix test failures due to ExceededMemoryLimitException message

• 08-26 22:20:11 900b011531 Fix more test failures due to ExceededMemoryLimitException message

  yingsu00

• [x] all checked

• 08-13 12:10:44 21841870e3 Refactor BooleanInputStream
• 08-13 12:27:03 97552478c9 Add memory tracking for StreamReader local buffers

[tdcmeehan]

--------
* Add the geometry_union_agg spatial aggregation function to union the input geometries.

[mbasmanova]

-------
* Optimized scalar correlated subquery to eliminate unnecessary data shuffle

[nezihyigitbasi]

Web UI Changes
--------------
* Fix the kill query button in the live plan and stage performance UIs.

[martint]

* Improve numerical stability for :func:`covar_samp`, :func:`corr`, :func:`regr_slope` and :func:`regr_intercept`.

[shixuan-fan]

-------
* Support EXPLAIN (TYPE IO, FORMAT JSON) that shows input tables with constraints and output table in JSON format.

[kokosing]

Kudu Connector
----------------
 * Introduce Apache Kudu Connector

[yuyongyang800]

General
---------
 * Add the convex_hull_agg spatial aggregation function to compute the convex hull of the input geometries.

[anusudarsan]

Kafka connector
----------------
* Support for Avro formatted Kafka messages.
* Support for backward compatible Avro schema evolution.

https://github.com/prestodb/presto/pull/11108

[avinashidnani]

---------
 * Add bing_tiles_around variant taking radius in kilometers

[martint]

* Fix incorrect predicate pushdown when grouping sets contain the empty grouping set (#11296).

[findepi]

* Fix ``X-Forwarded-Proto`` header handling for ``/`` path

https://github.com/prestodb/presto/pull/11290

[shreyasjoshis]

Just want to make sure this isn't missed:

----
* ORC struct columns are now mapped by name rather than ordinal. This correctly handles missing/extra struct fields in the ORC file.

cc @dain #11001

[kokosing]

--------
 * Remove table scan original constraint information from EXPLAIN queries

[wenleix]

General

• Raise required Java version to 8u151. This avoids correctness issues for map to map cast when running under some lower version of JVM (e.g 8u92)

Hive

• Add Hive connector procedure call to create empty partition
• Reduces the number of cases where tiny ORC stripes will be written when some columns are highly dictionary compressed.

[rschlussel]

```

General

• Fix AllowAllAccessControl and FileBasedAccessControl to default to allowing access for checkCanCreateSchema, checkCanDropSchema, and checkCanRenameSchema.
• Fix a bug where we would send column aliases in the list of columns whose access permissions we were checking.

[raghavsethi]

General

• Change exceeded memory limit exceptions to have distinct error codes for global and per-node limits.

[rschlussel]

* Print statistic and cost estimates in EXPLAIN (TYPE DISTRIBUTED) and EXPLAIN ANALYZE plans

[arhimondr]

-------
* Support ROW/ARRAY/MAP values with NULLs in EQUAL and NOT_EQUAL operators

[jessesleeping]

--------
* Every Raptor table now has a hidden system table which contains min/max values of every 
BIGINT, DATE, TIMESTAMP and BOOLEAN column. A table named `example` will have a 
column range system table named `example$column_ranges`. 

[nezihyigitbasi]

Web UI Changes
--------------
* Fix kill query button in live plan and stage UI.

[sopel39]

-------
* Improve statistics estimation for expressions that involve NOT/OR
* Fix potential negative nulls fraction estimates for expressions that involve NOT/OR 

[yingsu00]

-------
* Improve memory accounting when reading ORC files. Previously, local buffers 
and object overhead of StreamReader was not tracked.

[haozhun]

```

• Fix regression in 0.208 that results in execution failure when
  at least one of the arguments to :func:min_by and :func:max_by
  is a constant null.
• Fix execution failure of queries using bucket-by-bucket execution when
  some buckets are completely filtered out.
• Fix execution failure of queries due to planning deficiency involving
  complex nested joins where a join that isn't eligible for bucket-by-bucket execution
  feeds into the build side of a join that is eligible.

SPI

• Allow using Object as parameter type and return type of SQL functions
  when the correponding SQL type is an unbounded generic.
  Previously, Object can be used only in limited circumstances, and wasn't intentional.