Presto: java.lang.UnsupportedOperationException: com.facebook.presto.spi.type.DoubleType

Created on 5 Dec 2018 · 7Comments · Source: prestodb/presto

I'm using latest Presto Server Version presto-server-0.214...

Hive Create Table Query:

create external table some_table_name(col1 double,col2 double) partitioned by (col3 string)  stored as parquet location 's3://<some location>';

The parquet file's schema is:

       {
        "field_name": "col1",
        "name": "col1",
        "numpy_type": "float64",
        "pandas_type": "float64"
    }, {
        "field_name": "col2",
        "name": "col2",
        "numpy_type": "float64",
        "pandas_type": "float64"
    }

Hive table created successfully but Presto throws below error,

java.lang.UnsupportedOperationException: com.facebook.presto.spi.type.DoubleType

Also tried setting hive.parquet.use-column-names=true but no luck...

Source

santosh-kore

👍3

Most helpful comment

@santosh-kore Most likely the error is caused by schema mismatch between table schema and parquet schema. you can try add this patch to get a more user friendly error message. https://github.com/prestodb/presto/pull/11515

qqibrow on 7 Dec 2018

👍5

All 7 comments

@santosh-kore Could you paste the full stack trace?

mbasmanova on 7 Dec 2018

@mbasmanova Thanks....please find full stack trace...

java.lang.UnsupportedOperationException: com.facebook.presto.spi.type.DoubleType
    at com.facebook.presto.spi.type.AbstractType.writeLong(AbstractType.java:111)
    at com.facebook.presto.parquet.reader.LongColumnReader.readValue(LongColumnReader.java:32)
    at com.facebook.presto.parquet.reader.PrimitiveColumnReader.lambda$readValues$1(PrimitiveColumnReader.java:184)
    at com.facebook.presto.parquet.reader.PrimitiveColumnReader.processValues(PrimitiveColumnReader.java:204)
    at com.facebook.presto.parquet.reader.PrimitiveColumnReader.readValues(PrimitiveColumnReader.java:183)
    at com.facebook.presto.parquet.reader.PrimitiveColumnReader.readPrimitive(PrimitiveColumnReader.java:171)
    at com.facebook.presto.parquet.reader.ParquetReader.readPrimitive(ParquetReader.java:209)
    at com.facebook.presto.parquet.reader.ParquetReader.readColumnChunk(ParquetReader.java:259)
    at com.facebook.presto.parquet.reader.ParquetReader.readBlock(ParquetReader.java:242)
    at com.facebook.presto.hive.parquet.ParquetPageSource$ParquetBlockLoader.load(ParquetPageSource.java:246)
    at com.facebook.presto.hive.parquet.ParquetPageSource$ParquetBlockLoader.load(ParquetPageSource.java:224)
    at com.facebook.presto.spi.block.LazyBlock.assureLoaded(LazyBlock.java:269)
    at com.facebook.presto.spi.block.LazyBlock.getLoadedBlock(LazyBlock.java:260)
    at com.facebook.presto.spi.Page.getLoadedPage(Page.java:261)
    at com.facebook.presto.operator.TableScanOperator.getOutput(TableScanOperator.java:245)
    at com.facebook.presto.operator.Driver.processInternal(Driver.java:379)
    at com.facebook.presto.operator.Driver.lambda$processFor$8(Driver.java:283)
    at com.facebook.presto.operator.Driver.tryWithLock(Driver.java:675)
    at com.facebook.presto.operator.Driver.processFor(Driver.java:276)
    at com.facebook.presto.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:1065)
    at com.facebook.presto.execution.executor.PrioritizedSplitRunner.process(PrioritizedSplitRunner.java:162)
    at com.facebook.presto.execution.executor.TaskExecutor$TaskRunner.run(TaskExecutor.java:483)
    at com.facebook.presto.$gen.Presto_0_214____20181205_114848_1.run(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

santosh-kore on 7 Dec 2018

@santosh-kore Looks like float64 columns are processed as long columns. What's the output of desc some_table_name? What are the column types in that output?

CC: @zhenxiao

mbasmanova on 7 Dec 2018

@mbasmanova in presto desc some_table_name output is,

                 Column                 |  Type   |     Extra     | Comment 
-----------------------------+---------+---------------+--------------------    
 col1                                   | double  |               |         
 col2                                   | double  |               |

santosh-kore on 7 Dec 2018

qqibrow on 7 Dec 2018

👍5

@santosh-kore This is a schema mismatch issue. You can confirm that with the parquet-tools and compare the schema in the parquet file and the table schema (that you get with DESC).