Presto: java.net.SocketTimeoutException: Read timed out when querying hive

Created on 20 Nov 2019 · 4Comments · Source: prestodb/presto

i'm using presto 0206 and i get read timeout when trying to query hive table after about 35 seconds, querying the same from hive cli works
this only happens on tables with a lot of partitions

presto:default> select * from hive.default.passenger_demand limit 5;
Query 20191120_183710_00036_a5shx failed: 10.80.1.91: java.net.SocketTimeoutException: Read timed out
com.facebook.presto.spi.PrestoException: 10.80.1.91: java.net.SocketTimeoutException: Read timed out
    at com.facebook.presto.hive.metastore.thrift.ThriftHiveMetastore.getPartitionColumnStatistics(ThriftHiveMetastore.java:273)
    at com.facebook.presto.hive.metastore.thrift.BridgingHiveMetastore.getPartitionColumnStatistics(BridgingHiveMetastore.java:90)
    at com.facebook.presto.hive.metastore.CachingHiveMetastore.loadPartitionColumnStatistics(CachingHiveMetastore.java:341)
    at com.facebook.presto.hive.metastore.CachingHiveMetastore.access$100(CachingHiveMetastore.java:62)
    at com.facebook.presto.hive.metastore.CachingHiveMetastore$2.loadAll(CachingHiveMetastore.java:156)
    at com.google.common.cache.CacheLoader$1.loadAll(CacheLoader.java:201)
    at com.google.common.cache.LocalCache.loadAll(LocalCache.java:4242)
    at com.google.common.cache.LocalCache.getAll(LocalCache.java:4205)
    at com.google.common.cache.LocalCache$LocalLoadingCache.getAll(LocalCache.java:5161)
    at com.facebook.presto.hive.metastore.CachingHiveMetastore.getAll(CachingHiveMetastore.java:224)
    at com.facebook.presto.hive.metastore.CachingHiveMetastore.getPartitionColumnStatistics(CachingHiveMetastore.java:314)
    at com.facebook.presto.hive.metastore.CachingHiveMetastore.loadPartitionColumnStatistics(CachingHiveMetastore.java:341)
    at com.facebook.presto.hive.metastore.CachingHiveMetastore.access$100(CachingHiveMetastore.java:62)
    at com.facebook.presto.hive.metastore.CachingHiveMetastore$2.loadAll(CachingHiveMetastore.java:156)
    at com.google.common.cache.CacheLoader$1.loadAll(CacheLoader.java:201)
    at com.google.common.cache.LocalCache.loadAll(LocalCache.java:4242)
    at com.google.common.cache.LocalCache.getAll(LocalCache.java:4205)
    at com.google.common.cache.LocalCache$LocalLoadingCache.getAll(LocalCache.java:5161)
    at com.facebook.presto.hive.metastore.CachingHiveMetastore.getAll(CachingHiveMetastore.java:224)
    at com.facebook.presto.hive.metastore.CachingHiveMetastore.getPartitionColumnStatistics(CachingHiveMetastore.java:314)
    at com.facebook.presto.hive.metastore.SemiTransactionalHiveMetastore.getPartitionColumnStatistics(SemiTransactionalHiveMetastore.java:197)
    at com.facebook.presto.hive.statistics.MetastoreHiveStatisticsProvider.getPartitionsStatistics(MetastoreHiveStatisticsProvider.java:259)
    at com.facebook.presto.hive.statistics.MetastoreHiveStatisticsProvider.getPartitionsStatistics(MetastoreHiveStatisticsProvider.java:246)
    at com.facebook.presto.hive.statistics.MetastoreHiveStatisticsProvider.getTableStatistics(MetastoreHiveStatisticsProvider.java:76)
    at com.facebook.presto.hive.HiveMetadata.getTableStatistics(HiveMetadata.java:400)
    at com.facebook.presto.spi.connector.classloader.ClassLoaderSafeConnectorMetadata.getTableStatistics(ClassLoaderSafeConnectorMetadata.java:176)
    at com.facebook.presto.metadata.MetadataManager.getTableStatistics(MetadataManager.java:354)
    at com.facebook.presto.cost.CoefficientBasedCostCalculator$Visitor.visitTableScanWithPredicate(CoefficientBasedCostCalculator.java:189)
    at com.facebook.presto.cost.CoefficientBasedCostCalculator$Visitor.visitTableScan(CoefficientBasedCostCalculator.java:182)
    at com.facebook.presto.cost.CoefficientBasedCostCalculator$Visitor.visitTableScan(CoefficientBasedCostCalculator.java:84)
    at com.facebook.presto.sql.planner.plan.TableScanNode.accept(TableScanNode.java:135)
    at com.facebook.presto.cost.CoefficientBasedCostCalculator$Visitor.lambda$visitSources$4(CoefficientBasedCostCalculator.java:270)
    at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
    at java.util.Collections$2.tryAdvance(Collections.java:4717)
    at java.util.Collections$2.forEachRemaining(Collections.java:4725)
    at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
    at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
    at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
    at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
    at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
    at com.facebook.presto.cost.CoefficientBasedCostCalculator$Visitor.visitSources(CoefficientBasedCostCalculator.java:271)
    at com.facebook.presto.cost.CoefficientBasedCostCalculator$Visitor.visitSource(CoefficientBasedCostCalculator.java:276)
    at com.facebook.presto.cost.CoefficientBasedCostCalculator$Visitor.visitLimit(CoefficientBasedCostCalculator.java:248)
    at com.facebook.presto.cost.CoefficientBasedCostCalculator$Visitor.visitLimit(CoefficientBasedCostCalculator.java:84)
    at com.facebook.presto.sql.planner.plan.LimitNode.accept(LimitNode.java:87)
    at com.facebook.presto.cost.CoefficientBasedCostCalculator$Visitor.lambda$visitSources$4(CoefficientBasedCostCalculator.java:270)
    at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
    at java.util.Collections$2.tryAdvance(Collections.java:4717)
    at java.util.Collections$2.forEachRemaining(Collections.java:4725)
    at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
    at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
    at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
    at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
    at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
    at com.facebook.presto.cost.CoefficientBasedCostCalculator$Visitor.visitSources(CoefficientBasedCostCalculator.java:271)
    at com.facebook.presto.cost.CoefficientBasedCostCalculator$Visitor.visitExchange(CoefficientBasedCostCalculator.java:161)
    at com.facebook.presto.cost.CoefficientBasedCostCalculator$Visitor.visitExchange(CoefficientBasedCostCalculator.java:84)
    at com.facebook.presto.sql.planner.plan.ExchangeNode.accept(ExchangeNode.java:196)
    at com.facebook.presto.cost.CoefficientBasedCostCalculator$Visitor.lambda$visitSources$4(CoefficientBasedCostCalculator.java:270)
    at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
    at java.util.Collections$2.tryAdvance(Collections.java:4717)
    at java.util.Collections$2.forEachRemaining(Collections.java:4725)
    at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
    at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
    at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
    at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
    at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
    at com.facebook.presto.cost.CoefficientBasedCostCalculator$Visitor.visitSources(CoefficientBasedCostCalculator.java:271)
    at com.facebook.presto.cost.CoefficientBasedCostCalculator$Visitor.visitExchange(CoefficientBasedCostCalculator.java:161)
    at com.facebook.presto.cost.CoefficientBasedCostCalculator$Visitor.visitExchange(CoefficientBasedCostCalculator.java:84)
    at com.facebook.presto.sql.planner.plan.ExchangeNode.accept(ExchangeNode.java:196)
    at com.facebook.presto.cost.CoefficientBasedCostCalculator$Visitor.lambda$visitSources$4(CoefficientBasedCostCalculator.java:270)
    at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
    at java.util.Collections$2.tryAdvance(Collections.java:4717)
    at java.util.Collections$2.forEachRemaining(Collections.java:4725)
    at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
    at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
    at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
    at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
    at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
    at com.facebook.presto.cost.CoefficientBasedCostCalculator$Visitor.visitSources(CoefficientBasedCostCalculator.java:271)
    at com.facebook.presto.cost.CoefficientBasedCostCalculator$Visitor.visitSource(CoefficientBasedCostCalculator.java:276)
    at com.facebook.presto.cost.CoefficientBasedCostCalculator$Visitor.visitLimit(CoefficientBasedCostCalculator.java:248)
    at com.facebook.presto.cost.CoefficientBasedCostCalculator$Visitor.visitLimit(CoefficientBasedCostCalculator.java:84)
    at com.facebook.presto.sql.planner.plan.LimitNode.accept(LimitNode.java:87)
    at com.facebook.presto.cost.CoefficientBasedCostCalculator$Visitor.lambda$visitSources$4(CoefficientBasedCostCalculator.java:270)
    at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
    at java.util.Collections$2.tryAdvance(Collections.java:4717)
    at java.util.Collections$2.forEachRemaining(Collections.java:4725)
    at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
    at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
    at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
    at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
    at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
    at com.facebook.presto.cost.CoefficientBasedCostCalculator$Visitor.visitSources(CoefficientBasedCostCalculator.java:271)
    at com.facebook.presto.cost.CoefficientBasedCostCalculator$Visitor.visitSource(CoefficientBasedCostCalculator.java:276)
    at com.facebook.presto.cost.CoefficientBasedCostCalculator$Visitor.copySourceCost(CoefficientBasedCostCalculator.java:262)
    at com.facebook.presto.cost.CoefficientBasedCostCalculator$Visitor.visitOutput(CoefficientBasedCostCalculator.java:114)
    at com.facebook.presto.cost.CoefficientBasedCostCalculator$Visitor.visitOutput(CoefficientBasedCostCalculator.java:84)
    at com.facebook.presto.sql.planner.plan.OutputNode.accept(OutputNode.java:82)
    at com.facebook.presto.cost.CoefficientBasedCostCalculator.calculateCostForPlan(CoefficientBasedCostCalculator.java:80)
    at com.facebook.presto.sql.planner.LogicalPlanner.plan(LogicalPlanner.java:143)
    at com.facebook.presto.sql.planner.LogicalPlanner.plan(LogicalPlanner.java:122)
    at com.facebook.presto.execution.SqlQueryExecution.doAnalyzeQuery(SqlQueryExecution.java:363)
    at com.facebook.presto.execution.SqlQueryExecution.analyzeQuery(SqlQueryExecution.java:342)
    at com.facebook.presto.execution.SqlQueryExecution.start(SqlQueryExecution.java:274)
    at com.facebook.presto.execution.QueuedExecution.lambda$start$1(QueuedExecution.java:62)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.thrift.transport.TTransportException: 10.80.1.91: java.net.SocketTimeoutException: Read timed out
    at com.facebook.presto.hive.metastore.thrift.Transport.rewriteException(Transport.java:91)
    at com.facebook.presto.hive.metastore.thrift.Transport.access$000(Transport.java:29)
    at com.facebook.presto.hive.metastore.thrift.Transport$TTransportWrapper.readAll(Transport.java:168)
    at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:362)
    at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:284)
    at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:191)
    at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
    at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_partitions_statistics_req(ThriftHiveMetastore.java:3000)
    at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_partitions_statistics_req(ThriftHiveMetastore.java:2987)
    at com.facebook.presto.hive.metastore.thrift.ThriftHiveMetastoreClient.getPartitionColumnStatistics(ThriftHiveMetastoreClient.java:153)
    at com.facebook.presto.hive.metastore.thrift.ThriftHiveMetastore.lambda$getPartitionColumnStatistics$8(ThriftHiveMetastore.java:260)
    at com.facebook.presto.hive.metastore.thrift.HiveMetastoreApiStats.lambda$wrap$0(HiveMetastoreApiStats.java:42)
    at com.facebook.presto.hive.RetryDriver.run(RetryDriver.java:138)
    at com.facebook.presto.hive.metastore.thrift.ThriftHiveMetastore.getPartitionColumnStatistics(ThriftHiveMetastore.java:258)
    ... 109 more
Caused by: java.net.SocketTimeoutException: Read timed out
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
    at java.net.SocketInputStream.read(SocketInputStream.java:171)
    at java.net.SocketInputStream.read(SocketInputStream.java:141)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
    at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
    at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
    at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
    at com.facebook.presto.hive.metastore.thrift.Transport$TTransportWrapper.readAll(Transport.java:165)
    ... 120 more

Source

dinal

Most helpful comment

It is not in the doc unfortunately. Here is the property definition in the
source code:
https://github.com/prestodb/presto/blob/master/presto-hive-metastore/src/main/java/com/facebook/presto/hive/MetastoreClientConfig.java#L63

Feel free to open a PR updating the documentation.

On Wed, Jul 29, 2020 at 5:17 AM HanHui notifications@github.com wrote:

@arhimondr https://github.com/arhimondr i can't find
hive.metastore-timeout from document from there
https://prestodb.io/docs/current/connector/hive.html. can you provide
the doc, thx~

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/prestodb/presto/issues/13728#issuecomment-665545187,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABKQDLFKUKQFLME7LY4ZDYDR57SMLANCNFSM4JPYHH7A
.

arhimondr on 30 Jul 2020

❤3

All 4 comments

Seems to be related to column stats. @arhimondr Do you want to take a look?

shixuan-fan on 21 Nov 2019

It looks like you are trying to query too many partitions at the same time.

There are 2 solutions that you can try:

Try to decrease the partition sample size by setting the partition_statistics_sample_size session property to something lower than 100, that is set by default.
Try to increase the metastore request timeout by setting the hive.metastore-timeout configuration property in the hive connector configuration property file

arhimondr on 22 Nov 2019

@arhimondr i can't find hive.metastore-timeout from document from there. can you provide the doc, thx~

inkinworld on 29 Jul 2020

Feel free to open a PR updating the documentation.

On Wed, Jul 29, 2020 at 5:17 AM HanHui notifications@github.com wrote:

@arhimondr https://github.com/arhimondr i can't find
hive.metastore-timeout from document from there
https://prestodb.io/docs/current/connector/hive.html. can you provide
the doc, thx~

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/prestodb/presto/issues/13728#issuecomment-665545187,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABKQDLFKUKQFLME7LY4ZDYDR57SMLANCNFSM4JPYHH7A
.