Presto: Cassandra connector IN query very slow planning on large list
A query like -
select col1
from table
where col2 in (<long list of integers>)
and col3 in (<long list of string>)
and col4 in (<another long list of integers>)
and col1 is not null
group by col1;
takes more than 5 minutes just planning. My cassandra table being queried has a lot of partitions and list length for IN query I was experimenting with was anywhere between 50 to 200.
<col2, col3, col4> together form the partition keys so I don't imagine a full table scan to take place during planning or execution.
Any ideas?
aandis
All 34 comments
Using Presto on EMR, v0.194.
aandis
on 24 May 2018
FYI: @kokosing
What is type of col2, col3 and col4 columns?
sopel39
on 25 May 2018
@sopel39 just plain integers and strings. col2, col4 are int, col3 is text.
aandis
on 25 May 2018
Let me know if you guys need any info from my side to help debug this. I am curious myself and would like to know the reason this is happening and possibly help fix it.
aandis
on 25 May 2018
~It appears that the bottleneck is col3 which is a long list of string. Making it small makes the query execute instantaneously even if col2 and col4 are large integer lists.~
It looks like making any of the lists smaller helps.
aandis
on 25 May 2018
Setting iterative_optimizer_enabled to false helps but planning time is still around a couple of minutes.
aandis
on 27 May 2018
https://github.com/prestodb/presto/issues/9990 seems related.
@aandis are you able to profile/sample the coordinator while it's planning your query to see where most of the time is spent?
findepi
on 27 May 2018
@findepi I have used async-profiler for profiling a sample query like above. Plan time was around 5 minutes. Output is in this gist. Let me know if that helps.
aandis
on 28 May 2018
Setting iterative_optimizer_enabled to false helps but planning time is still around a couple of minutes.
It is very interesting, can you also profile this case and share the output?
Have you considered the update, notice there were some changes which could help in your case:
Support predicate pushdown for the
IN predicate where values in the values list require casting to match the type of column.
- https://prestodb.io/docs/current/release/release-0.199.html
Improve predicate evaluation performance during predicate pushdown in planning.
Does removing IN predicate on string column help? Can you test the below query?
select col1
from table
where col2 in (<long list of integers>)
and col4 in (<another long list of integers>)
and col1 is not null
group by col1;
kokosing
on 28 May 2018
can you also profile this case and share the output?
@kokosing here's the profiler output with iterative optimizer disabled.
I will try the update version 0.199 and see if that helps.
Does removing IN predicate on string column help?
It does. Query starts instantaneously. But since <col2, col3, col4> form the partition key of the table, I would imaging that presto would do a full table scan if col3 isn't specified. So query execution takes a long time in this case because table is very large.
Note my comment that making any of the list smaller also helps.
aandis
on 28 May 2018
Using 0.199 didn't help. Still seeing the same behavior.
aandis
on 28 May 2018
It looks like CassandraPartitionManager is the bottleneck. I ran the server in debug mode and saw this
2018-05-28T07:53:36.962Z DEBUG query-execution-0 com.facebook.presto.execution.QueryStateMachine Query 20180528_075336_00001_j36cu is PLANNING
[Loaded sun.reflect.GeneratedMethodAccessor135 from __JVM_DefineClass__]
[Loaded sun.reflect.GeneratedMethodAccessor136 from __JVM_DefineClass__]
[Loaded sun.reflect.GeneratedMethodAccessor137 from __JVM_DefineClass__]
[Loaded sun.reflect.GeneratedMethodAccessor138 from __JVM_DefineClass__]
[Loaded sun.reflect.GeneratedMethodAccessor139 from __JVM_DefineClass__]
2018-05-28T07:54:36.482Z DEBUG Query-20180528_075336_00001_j36cu-166 com.facebook.presto.cassandra.CassandraPartitionManager <keyspace.table> #partitions: 20201
2018-05-28T07:55:35.541Z DEBUG Query-20180528_075336_00001_j36cu-166 com.facebook.presto.cassandra.CassandraPartitionManager <keyspace.table> #partitions: 20201
[Loaded com.facebook.presto.sql.planner.iterative.rule.PushProjectionThroughExchange$$Lambda$1000/586980830 from com.facebook.presto.sql.planner.iterative.rule.PushProjectionThroughExchange]
[Loaded com.facebook.presto.spi.block.LongArrayBlockBuilder from file:/home/hadoop/presto/presto-server-0.199/lib/presto-spi-0.199.jar]
[Loaded com.facebook.presto.spi.block.LongArrayBlock from file:/home/hadoop/presto/presto-server-0.199/lib/presto-spi-0.199.jar]
2018-05-28T07:56:34.056Z DEBUG Query-20180528_075336_00001_j36cu-166 com.facebook.presto.cassandra.CassandraPartitionManager <keyspace.table> #partitions: 20201
[Loaded com.facebook.presto.sql.planner.optimizations.PreferredProperties from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.sql.planner.optimizations.PreferredProperties$Builder from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.sql.planner.optimizations.PreferredProperties$Global from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.sql.planner.optimizations.PreferredProperties$PartitioningProperties from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter$$Lambda$1001/1214932022 from com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter]
[Loaded com.facebook.presto.sql.planner.optimizations.PreferredProperties$$Lambda$1002/948364710 from com.facebook.presto.sql.planner.optimizations.PreferredProperties]
[Loaded com.facebook.presto.sql.planner.optimizations.LocalProperties from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.sql.planner.optimizations.LocalProperties$$Lambda$1003/1636340279 from com.facebook.presto.sql.planner.optimizations.LocalProperties]
[Loaded com.facebook.presto.spi.LocalProperty from file:/home/hadoop/presto/presto-server-0.199/lib/presto-spi-0.199.jar]
[Loaded com.facebook.presto.sql.planner.optimizations.LocalProperties$$Lambda$1004/1043796413 from com.facebook.presto.sql.planner.optimizations.LocalProperties]
[Loaded com.facebook.presto.spi.GroupingProperty from file:/home/hadoop/presto/presto-server-0.199/lib/presto-spi-0.199.jar]
[Loaded com.facebook.presto.sql.planner.optimizations.PreferredProperties$$Lambda$1005/189359888 from com.facebook.presto.sql.planner.optimizations.PreferredProperties]
[Loaded com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter$$Lambda$1006/225781053 from com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter]
[Loaded com.facebook.presto.spi.predicate.TupleDomain$$Lambda$1007/835911188 from com.facebook.presto.spi.predicate.TupleDomain]
[Loaded com.facebook.presto.spi.predicate.TupleDomain$$Lambda$1008/250736365 from com.facebook.presto.spi.predicate.TupleDomain]
[Loaded com.facebook.presto.spi.predicate.Domain$$Lambda$1009/1321232836 from com.facebook.presto.spi.predicate.Domain]
[Loaded com.facebook.presto.spi.predicate.Domain$$Lambda$1010/1790501356 from com.facebook.presto.spi.predicate.Domain]
[Loaded com.facebook.presto.spi.predicate.Domain$$Lambda$1011/1279033585 from com.facebook.presto.spi.predicate.Domain]
[Loaded com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter$$Lambda$1012/1304135965 from com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter]
[Loaded com.facebook.presto.sql.planner.optimizations.AddExchanges$LayoutConstraintEvaluator from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter$$Lambda$1013/1443920136 from com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter]
[Loaded com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter$$Lambda$1014/1757856342 from com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter]
2018-05-28T07:57:32.654Z DEBUG Query-20180528_075336_00001_j36cu-166 com.facebook.presto.cassandra.CassandraPartitionManager <keyspace.table> #partitions: 20201
[Loaded com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter$$Lambda$1015/54218617 from com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter]
[Loaded com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter$$Lambda$1016/1527043797 from com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter]
[Loaded com.facebook.presto.sql.planner.optimizations.AddExchanges$PlanWithProperties from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter$$Lambda$1017/1845377395 from com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter]
[Loaded com.facebook.presto.sql.planner.optimizations.PropertyDerivations from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.sql.planner.optimizations.PropertyDerivations$Visitor from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.sql.planner.optimizations.ActualProperties from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.sql.planner.optimizations.ActualProperties$Builder from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.sql.planner.optimizations.ActualProperties$Global from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.spi.predicate.TupleDomain$$Lambda$1018/338747667 from com.facebook.presto.spi.predicate.TupleDomain]
[Loaded com.facebook.presto.spi.predicate.TupleDomain$$Lambda$1019/967522159 from com.facebook.presto.spi.predicate.TupleDomain]
[Loaded com.facebook.presto.spi.predicate.TupleDomain$$Lambda$1020/820530708 from com.facebook.presto.spi.predicate.TupleDomain]
[Loaded com.facebook.presto.sql.planner.optimizations.PropertyDerivations$Visitor$$Lambda$1021/1135821355 from com.facebook.presto.sql.planner.optimizations.PropertyDerivations$Visitor]
[Loaded com.facebook.presto.sql.planner.optimizations.PropertyDerivations$Visitor$$Lambda$1022/1356665895 from com.facebook.presto.sql.planner.optimizations.PropertyDerivations$Visitor]
[Loaded com.facebook.presto.sql.planner.optimizations.PropertyDerivations$Visitor$$Lambda$1023/574060742 from com.facebook.presto.sql.planner.optimizations.PropertyDerivations$Visitor]
[Loaded com.facebook.presto.sql.planner.optimizations.PropertyDerivations$Visitor$$Lambda$1024/428111346 from com.facebook.presto.sql.planner.optimizations.PropertyDerivations$Visitor]
[Loaded com.facebook.presto.sql.planner.optimizations.PropertyDerivations$Visitor$$Lambda$1025/1090506483 from com.facebook.presto.sql.planner.optimizations.PropertyDerivations$Visitor]
[Loaded com.facebook.presto.sql.planner.optimizations.PropertyDerivations$Visitor$$Lambda$1026/2025555687 from com.facebook.presto.sql.planner.optimizations.PropertyDerivations$Visitor]
[Loaded com.facebook.presto.spi.ConnectorTablePartitioning from file:/home/hadoop/presto/presto-server-0.199/lib/presto-spi-0.199.jar]
[Loaded com.facebook.presto.metadata.TableLayout$TablePartitioning from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.metadata.TableLayout$$Lambda$1027/282225947 from com.facebook.presto.metadata.TableLayout]
[Loaded com.facebook.presto.spi.ConstantProperty from file:/home/hadoop/presto/presto-server-0.199/lib/presto-spi-0.199.jar]
[Loaded com.facebook.presto.sql.planner.optimizations.PropertyDerivations$Visitor$$Lambda$1028/716103634 from com.facebook.presto.sql.planner.optimizations.PropertyDerivations$Visitor]
[Loaded com.facebook.presto.sql.planner.optimizations.PropertyDerivations$Visitor$$Lambda$1029/1590046 from com.facebook.presto.sql.planner.optimizations.PropertyDerivations$Visitor]
[Loaded com.google.common.collect.Iterators$PeekingImpl from file:/home/hadoop/presto/presto-server-0.199/lib/guava-24.1-jre.jar]
[Loaded com.facebook.presto.sql.planner.optimizations.ActualProperties$$Lambda$1030/246227864 from com.facebook.presto.sql.planner.optimizations.ActualProperties]
[Loaded com.facebook.presto.sql.planner.Partitioning from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.sql.planner.optimizations.PropertyDerivations$$Lambda$1031/1476580622 from com.facebook.presto.sql.planner.optimizations.PropertyDerivations]
[Loaded com.facebook.presto.sql.planner.optimizations.PropertyDerivations$$Lambda$1032/1779726091 from com.facebook.presto.sql.planner.optimizations.PropertyDerivations]
[Loaded com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter$$Lambda$1033/1066087300 from com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter]
[Loaded com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter$$Lambda$1034/1911499742 from com.facebook.presto.sql.planner.optimizations.AddExchanges$Rewriter]
[Loaded com.facebook.presto.sql.planner.SystemPartitioningHandle$SystemPartitioning from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.sql.planner.SystemPartitioningHandle$SystemPartitionFunction from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.sql.planner.SystemPartitioningHandle$SystemPartitionFunction$1 from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.sql.planner.SystemPartitioningHandle$SystemPartitionFunction$2 from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.sql.planner.SystemPartitioningHandle$SystemPartitionFunction$3 from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.sql.planner.SystemPartitioningHandle$SystemPartitionFunction$4 from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.sql.planner.SystemPartitioningHandle$SystemPartitionFunction$5 from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.spi.BucketFunction from file:/home/hadoop/presto/presto-server-0.199/lib/presto-spi-0.199.jar]
[Loaded com.facebook.presto.operator.HashGenerator from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.sql.planner.Partitioning$$Lambda$1035/995193715 from com.facebook.presto.sql.planner.Partitioning]
[Loaded com.facebook.presto.sql.planner.Partitioning$$Lambda$1036/649328884 from com.facebook.presto.sql.planner.Partitioning]
[Loaded com.facebook.presto.sql.planner.Partitioning$$Lambda$1037/1898184236 from com.facebook.presto.sql.planner.Partitioning]
[Loaded com.facebook.presto.sql.planner.PartitioningScheme$$Lambda$1038/1893918543 from com.facebook.presto.sql.planner.PartitioningScheme]
[Loaded com.facebook.presto.sql.planner.plan.ExchangeNode$$Lambda$1039/1236256255 from com.facebook.presto.sql.planner.plan.ExchangeNode]
[Loaded com.facebook.presto.sql.planner.optimizations.PropertyDerivations$Visitor$$Lambda$1040/663978104 from com.facebook.presto.sql.planner.optimizations.PropertyDerivations$Visitor]
[Loaded com.facebook.presto.sql.planner.optimizations.PropertyDerivations$Visitor$$Lambda$1041/561746992 from com.facebook.presto.sql.planner.optimizations.PropertyDerivations$Visitor]
[Loaded com.facebook.presto.sql.planner.optimizations.ActualProperties$$Lambda$1042/1187305423 from com.facebook.presto.sql.planner.optimizations.ActualProperties]
[Loaded com.facebook.presto.sql.planner.optimizations.ActualProperties$Global$$Lambda$1043/1255718083 from com.facebook.presto.sql.planner.optimizations.ActualProperties$Global]
[Loaded com.facebook.presto.sql.planner.optimizations.ActualProperties$Global$$Lambda$1044/939490236 from com.facebook.presto.sql.planner.optimizations.ActualProperties$Global]
[Loaded com.facebook.presto.sql.planner.optimizations.PropertyDerivations$Visitor$$Lambda$1045/1474672675 from com.facebook.presto.sql.planner.optimizations.PropertyDerivations$Visitor]
[Loaded com.facebook.presto.sql.planner.optimizations.PropertyDerivations$Visitor$$Lambda$1046/1030360512 from com.facebook.presto.sql.planner.optimizations.PropertyDerivations$Visitor]
[Loaded com.facebook.presto.sql.planner.optimizations.PropertyDerivations$1 from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.sql.planner.plan.IndexJoinNode$Type from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.sql.planner.optimizations.PropertyDerivations$Visitor$$Lambda$1047/121060233 from com.facebook.presto.sql.planner.optimizations.PropertyDerivations$Visitor]
[Loaded com.facebook.presto.sql.planner.Partitioning$ArgumentBinding$$Lambda$1048/1345256086 from com.facebook.presto.sql.planner.Partitioning$ArgumentBinding]
[Loaded com.facebook.presto.sql.planner.optimizations.PropertyDerivations$Visitor$$Lambda$1049/154994188 from com.facebook.presto.sql.planner.optimizations.PropertyDerivations$Visitor]
[Loaded com.facebook.presto.spi.GroupingProperty$$Lambda$1050/1965612029 from com.facebook.presto.spi.GroupingProperty]
[Loaded com.facebook.presto.spi.GroupingProperty$$Lambda$1051/1266503074 from com.facebook.presto.spi.GroupingProperty]
[Loaded com.facebook.presto.sql.planner.optimizations.ActualProperties$Global$$Lambda$1052/223122159 from com.facebook.presto.sql.planner.optimizations.ActualProperties$Global]
[Loaded com.facebook.presto.sql.planner.optimizations.PropertyDerivations$Visitor$$Lambda$1053/1049122696 from com.facebook.presto.sql.planner.optimizations.PropertyDerivations$Visitor]
[Loaded com.facebook.presto.sql.planner.optimizations.UnaliasSymbolReferences$Rewriter$$Lambda$1054/1045764028 from com.facebook.presto.sql.planner.optimizations.UnaliasSymbolReferences$Rewriter]
[Loaded com.facebook.presto.sql.planner.optimizations.UnaliasSymbolReferences$Rewriter$$Lambda$1055/1170372105 from com.facebook.presto.sql.planner.optimizations.UnaliasSymbolReferences$Rewriter]
[Loaded com.facebook.presto.sql.planner.Partitioning$$Lambda$1056/1105669069 from com.facebook.presto.sql.planner.Partitioning]
[Loaded com.facebook.presto.sql.planner.optimizations.PruneUnreferencedOutputs$Rewriter$$Lambda$1057/1489103283 from com.facebook.presto.sql.planner.optimizations.PruneUnreferencedOutputs$Rewriter]
[Loaded com.facebook.presto.sql.planner.optimizations.PruneUnreferencedOutputs$Rewriter$$Lambda$1058/1312812512 from com.facebook.presto.sql.planner.optimizations.PruneUnreferencedOutputs$Rewriter]
[Loaded com.facebook.presto.sql.planner.optimizations.StreamPreferredProperties from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.sql.planner.optimizations.StreamPreferredProperties$$Lambda$1059/931246148 from com.facebook.presto.sql.planner.optimizations.StreamPreferredProperties]
[Loaded com.facebook.presto.sql.planner.optimizations.AddLocalExchanges$PlanWithProperties from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.sql.planner.optimizations.AddLocalExchanges$Rewriter$$Lambda$1060/2031112337 from com.facebook.presto.sql.planner.optimizations.AddLocalExchanges$Rewriter]
[Loaded com.facebook.presto.sql.planner.optimizations.AddLocalExchanges$Rewriter$$Lambda$1061/1490227850 from com.facebook.presto.sql.planner.optimizations.AddLocalExchanges$Rewriter]
[Loaded com.facebook.presto.sql.planner.optimizations.AddLocalExchanges$Rewriter$$Lambda$1062/254850939 from com.facebook.presto.sql.planner.optimizations.AddLocalExchanges$Rewriter]
[Loaded com.facebook.presto.sql.planner.optimizations.StreamPropertyDerivations$StreamProperties$StreamDistribution from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.sql.planner.optimizations.StreamPreferredProperties$$Lambda$1063/1046862355 from com.facebook.presto.sql.planner.optimizations.StreamPreferredProperties]
[Loaded com.facebook.presto.sql.planner.optimizations.AddLocalExchanges$Rewriter$$Lambda$1064/878983627 from com.facebook.presto.sql.planner.optimizations.AddLocalExchanges$Rewriter]
[Loaded com.facebook.presto.sql.planner.optimizations.StreamPropertyDerivations$StreamProperties from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.sql.planner.optimizations.AddLocalExchanges$Rewriter$$Lambda$1065/821497584 from com.facebook.presto.sql.planner.optimizations.AddLocalExchanges$Rewriter]
[Loaded com.facebook.presto.sql.planner.optimizations.StreamPropertyDerivations from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.sql.planner.optimizations.StreamPropertyDerivations$Visitor from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.sql.planner.optimizations.StreamPropertyDerivations$$Lambda$1066/914204419 from com.facebook.presto.sql.planner.optimizations.StreamPropertyDerivations]
[Loaded com.facebook.presto.sql.planner.optimizations.StreamPropertyDerivations$Visitor$$Lambda$1067/2124753524 from com.facebook.presto.sql.planner.optimizations.StreamPropertyDerivations$Visitor]
[Loaded com.facebook.presto.sql.planner.optimizations.StreamPropertyDerivations$Visitor$$Lambda$1068/1131865398 from com.facebook.presto.sql.planner.optimizations.StreamPropertyDerivations$Visitor]
[Loaded com.facebook.presto.sql.planner.optimizations.StreamPropertyDerivations$Visitor$$Lambda$1069/409470948 from com.facebook.presto.sql.planner.optimizations.StreamPropertyDerivations$Visitor]
[Loaded com.facebook.presto.sql.planner.optimizations.StreamPropertyDerivations$StreamProperties$$Lambda$1070/806742979 from com.facebook.presto.sql.planner.optimizations.StreamPropertyDerivations$StreamProperties]
[Loaded com.facebook.presto.sql.planner.optimizations.StreamPropertyDerivations$$Lambda$1071/1739886911 from com.facebook.presto.sql.planner.optimizations.StreamPropertyDerivations]
[Loaded com.facebook.presto.sql.planner.optimizations.StreamPropertyDerivations$$Lambda$1072/329226154 from com.facebook.presto.sql.planner.optimizations.StreamPropertyDerivations]
[Loaded com.facebook.presto.sql.planner.plan.ExchangeNode$$Lambda$1073/1259398426 from com.facebook.presto.sql.planner.plan.ExchangeNode]
[Loaded com.facebook.presto.sql.planner.optimizations.PropertyDerivations$Visitor$$Lambda$1074/1422148680 from com.facebook.presto.sql.planner.optimizations.PropertyDerivations$Visitor]
[Loaded com.facebook.presto.sql.planner.optimizations.PropertyDerivations$Visitor$$Lambda$1075/389683846 from com.facebook.presto.sql.planner.optimizations.PropertyDerivations$Visitor]
[Loaded com.facebook.presto.sql.planner.optimizations.StreamPropertyDerivations$1 from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.sql.planner.optimizations.StreamPropertyDerivations$Visitor$$Lambda$1076/1108544481 from com.facebook.presto.sql.planner.optimizations.StreamPropertyDerivations$Visitor]
[Loaded com.facebook.presto.sql.planner.optimizations.StreamPropertyDerivations$Visitor$$Lambda$1077/96126735 from com.facebook.presto.sql.planner.optimizations.StreamPropertyDerivations$Visitor]
[Loaded com.facebook.presto.sql.planner.optimizations.StreamPropertyDerivations$StreamProperties$$Lambda$1078/1122161977 from com.facebook.presto.sql.planner.optimizations.StreamPropertyDerivations$StreamProperties]
[Loaded com.facebook.presto.sql.planner.optimizations.StreamPropertyDerivations$Visitor$$Lambda$1079/879432549 from com.facebook.presto.sql.planner.optimizations.StreamPropertyDerivations$Visitor]
[Loaded com.facebook.presto.sql.planner.optimizations.StreamPropertyDerivations$Visitor$$Lambda$1080/1152019583 from com.facebook.presto.sql.planner.optimizations.StreamPropertyDerivations$Visitor]
[Loaded com.facebook.presto.sql.planner.plan.AggregationNode$$Lambda$1081/1543729099 from com.facebook.presto.sql.planner.plan.AggregationNode]
[Loaded com.facebook.presto.sql.planner.plan.AggregationNode$$Lambda$1082/511703989 from com.facebook.presto.sql.planner.plan.AggregationNode]
[Loaded com.facebook.presto.sql.planner.plan.AggregationNode$$Lambda$1083/1927492388 from com.facebook.presto.sql.planner.plan.AggregationNode]
[Loaded com.facebook.presto.sql.planner.plan.AggregationNode$$Lambda$1084/1698521235 from com.facebook.presto.sql.planner.plan.AggregationNode]
[Loaded com.facebook.presto.sql.planner.plan.AggregationNode$$Lambda$1085/570894024 from com.facebook.presto.sql.planner.plan.AggregationNode]
[Loaded com.facebook.presto.sql.planner.plan.AggregationNode$$Lambda$1086/1128584046 from com.facebook.presto.sql.planner.plan.AggregationNode]
[Loaded com.facebook.presto.sql.planner.plan.AggregationNode$$Lambda$1087/676700721 from com.facebook.presto.sql.planner.plan.AggregationNode]
[Loaded com.facebook.presto.sql.planner.plan.AggregationNode$$Lambda$1088/652375632 from com.facebook.presto.sql.planner.plan.AggregationNode]
[Loaded com.facebook.presto.operator.aggregation.ParametricAggregation$$Lambda$1089/1492278870 from com.facebook.presto.operator.aggregation.ParametricAggregation]
[Loaded com.facebook.presto.operator.aggregation.state.StateCompiler from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.operator.aggregation.state.StateCompiler$1 from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.spi.function.AccumulatorStateMetadata from file:/home/hadoop/presto/presto-server-0.199/lib/presto-spi-0.199.jar]
[Loaded io.airlift.bytecode.Access from file:/home/hadoop/presto/presto-server-0.199/lib/bytecode-1.0.jar]
[Loaded com.facebook.presto.util.CompilerUtils from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded java.time.format.DateTimeFormatterBuilder$WeekBasedFieldPrinterParser from /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.171-7.b10.37.amzn1.x86_64/jre/lib/rt.jar]
[Loaded com.facebook.presto.util.CompilerUtils$$Lambda$1090/1549543633 from com.facebook.presto.util.CompilerUtils]
[Loaded java.time.Clock$SystemClock from /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.171-7.b10.37.amzn1.x86_64/jre/lib/rt.jar]
[Loaded java.time.temporal.WeekFields from /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.171-7.b10.37.amzn1.x86_64/jre/lib/rt.jar]
[Loaded java.time.DayOfWeek from /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.171-7.b10.37.amzn1.x86_64/jre/lib/rt.jar]
[Loaded java.time.temporal.WeekFields$ComputedDayOfField from /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.171-7.b10.37.amzn1.x86_64/jre/lib/rt.jar]
[Loaded java.time.Month from /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.171-7.b10.37.amzn1.x86_64/jre/lib/rt.jar]
[Loaded java.time.Month$1 from /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.171-7.b10.37.amzn1.x86_64/jre/lib/rt.jar]
[Loaded io.airlift.bytecode.BytecodeUtils from file:/home/hadoop/presto/presto-server-0.199/lib/bytecode-1.0.jar]
[Loaded io.airlift.bytecode.BytecodeUtils$$Lambda$1091/1469833579 from io.airlift.bytecode.BytecodeUtils]
[Loaded io.airlift.bytecode.expression.SetFieldBytecodeExpression from file:/home/hadoop/presto/presto-server-0.199/lib/bytecode-1.0.jar]
[Loaded io.airlift.bytecode.expression.SetArrayElementBytecodeExpression from file:/home/hadoop/presto/presto-server-0.199/lib/bytecode-1.0.jar]
[Loaded io.airlift.bytecode.expression.GetElementBytecodeExpression from file:/home/hadoop/presto/presto-server-0.199/lib/bytecode-1.0.jar]
[Loaded io.airlift.bytecode.expression.InvokeBytecodeExpression from file:/home/hadoop/presto/presto-server-0.199/lib/bytecode-1.0.jar]
[Loaded io.airlift.bytecode.expression.ArrayLengthBytecodeExpression from file:/home/hadoop/presto/presto-server-0.199/lib/bytecode-1.0.jar]
[Loaded io.airlift.bytecode.expression.GetFieldBytecodeExpression from file:/home/hadoop/presto/presto-server-0.199/lib/bytecode-1.0.jar]
[Loaded io.airlift.bytecode.expression.CastBytecodeExpression from file:/home/hadoop/presto/presto-server-0.199/lib/bytecode-1.0.jar]
[Loaded io.airlift.bytecode.expression.PopBytecodeExpression from file:/home/hadoop/presto/presto-server-0.199/lib/bytecode-1.0.jar]
[Loaded io.airlift.bytecode.expression.ReturnBytecodeExpression from file:/home/hadoop/presto/presto-server-0.199/lib/bytecode-1.0.jar]
[Loaded io.airlift.bytecode.MethodDefinition$$Lambda$1092/482189359 from io.airlift.bytecode.MethodDefinition]
[Loaded io.airlift.bytecode.MethodDefinition$$Lambda$1093/1176867680 from io.airlift.bytecode.MethodDefinition]
[Loaded io.airlift.bytecode.Variable$SetVariableBytecodeExpression from file:/home/hadoop/presto/presto-server-0.199/lib/bytecode-1.0.jar]
[Loaded io.airlift.bytecode.instruction.InstructionNode from file:/home/hadoop/presto/presto-server-0.199/lib/bytecode-1.0.jar]
[Loaded io.airlift.bytecode.instruction.InvokeInstruction from file:/home/hadoop/presto/presto-server-0.199/lib/bytecode-1.0.jar]
[Loaded io.airlift.bytecode.OpCode from file:/home/hadoop/presto/presto-server-0.199/lib/bytecode-1.0.jar]
[Loaded com.facebook.presto.operator.aggregation.state.StateCompiler$StateField from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded io.airlift.bytecode.Comment from file:/home/hadoop/presto/presto-server-0.199/lib/bytecode-1.0.jar]
[Loaded com.facebook.presto.sql.gen.Binding from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.sql.gen.Bootstrap from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded io.airlift.bytecode.expression.CastBytecodeExpression$1 from file:/home/hadoop/presto/presto-server-0.199/lib/bytecode-1.0.jar]
[Loaded io.airlift.bytecode.expression.CastBytecodeExpression$TypeKind from file:/home/hadoop/presto/presto-server-0.199/lib/bytecode-1.0.jar]
[Loaded io.airlift.bytecode.instruction.TypeInstruction from file:/home/hadoop/presto/presto-server-0.199/lib/bytecode-1.0.jar]
[Loaded io.airlift.bytecode.expression.BytecodeExpression$$Lambda$1094/223898114 from io.airlift.bytecode.expression.BytecodeExpression]
2018-05-28T07:57:33.224Z DEBUG Query-20180528_075336_00001_j36cu-166 com.facebook.presto.util.CompilerUtils Defining class: com/facebook/presto/$gen/LongStateSerializer_20180528_075733_1
[Loaded io.airlift.bytecode.ClassGenerator from file:/home/hadoop/presto/presto-server-0.199/lib/bytecode-1.0.jar]
[Loaded java.lang.NegativeArraySizeException from /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.171-7.b10.37.amzn1.x86_64/jre/lib/rt.jar]
[Loaded java.lang.VerifyError from /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.171-7.b10.37.amzn1.x86_64/jre/lib/rt.jar]
[Loaded org.objectweb.asm.ClassVisitor from file:/home/hadoop/presto/presto-server-0.199/lib/asm-6.0.jar]
[Loaded org.objectweb.asm.ClassWriter from file:/home/hadoop/presto/presto-server-0.199/lib/asm-6.0.jar]
[Loaded io.airlift.bytecode.SmartClassWriter from file:/home/hadoop/presto/presto-server-0.199/lib/bytecode-1.0.jar]
[Loaded io.airlift.bytecode.AddFakeLineNumberClassVisitor from file:/home/hadoop/presto/presto-server-0.199/lib/bytecode-1.0.jar]
[Loaded org.objectweb.asm.util.Printer from file:/home/hadoop/presto/presto-server-0.199/lib/asm-util-6.0.jar]
[Loaded org.objectweb.asm.util.Textifier from file:/home/hadoop/presto/presto-server-0.199/lib/asm-util-6.0.jar]
[Loaded org.objectweb.asm.util.TraceClassVisitor from file:/home/hadoop/presto/presto-server-0.199/lib/asm-util-6.0.jar]
[Loaded com.google.common.io.CharStreams from file:/home/hadoop/presto/presto-server-0.199/lib/guava-24.1-jre.jar]
[Loaded com.google.common.io.CharStreams$NullWriter from file:/home/hadoop/presto/presto-server-0.199/lib/guava-24.1-jre.jar]
[Loaded com.google.common.io.AppendableWriter from file:/home/hadoop/presto/presto-server-0.199/lib/guava-24.1-jre.jar]
[Loaded io.airlift.bytecode.ClassInfoLoader from file:/home/hadoop/presto/presto-server-0.199/lib/bytecode-1.0.jar]
[Loaded org.objectweb.asm.tree.ClassNode from file:/home/hadoop/presto/presto-server-0.199/lib/asm-tree-6.0.jar]
[Loaded org.objectweb.asm.util.CheckClassAdapter from file:/home/hadoop/presto/presto-server-0.199/lib/asm-util-6.0.jar]
[Loaded org.objectweb.asm.ModuleVisitor from file:/home/hadoop/presto/presto-server-0.199/lib/asm-6.0.jar]
[Loaded org.objectweb.asm.tree.ModuleNode from file:/home/hadoop/presto/presto-server-0.199/lib/asm-tree-6.0.jar]
[Loaded org.objectweb.asm.AnnotationVisitor from file:/home/hadoop/presto/presto-server-0.199/lib/asm-6.0.jar]
[Loaded org.objectweb.asm.tree.AnnotationNode from file:/home/hadoop/presto/presto-server-0.199/lib/asm-tree-6.0.jar]
[Loaded org.objectweb.asm.tree.TypeAnnotationNode from file:/home/hadoop/presto/presto-server-0.199/lib/asm-tree-6.0.jar]
[Loaded org.objectweb.asm.MethodVisitor from file:/home/hadoop/presto/presto-server-0.199/lib/asm-6.0.jar]
[Loaded org.objectweb.asm.tree.MethodNode from file:/home/hadoop/presto/presto-server-0.199/lib/asm-tree-6.0.jar]
[Loaded org.objectweb.asm.FieldVisitor from file:/home/hadoop/presto/presto-server-0.199/lib/asm-6.0.jar]
[Loaded org.objectweb.asm.tree.FieldNode from file:/home/hadoop/presto/presto-server-0.199/lib/asm-tree-6.0.jar]
[Loaded io.airlift.bytecode.ClassDefinition$$Lambda$1095/2021745200 from io.airlift.bytecode.ClassDefinition]
[Loaded io.airlift.bytecode.MethodDefinition$$Lambda$1096/349825765 from io.airlift.bytecode.MethodDefinition]
[Loaded org.objectweb.asm.tree.AbstractInsnNode from file:/home/hadoop/presto/presto-server-0.199/lib/asm-tree-6.0.jar]
[Loaded org.objectweb.asm.tree.LineNumberNode from file:/home/hadoop/presto/presto-server-0.199/lib/asm-tree-6.0.jar]
[Loaded org.objectweb.asm.tree.LocalVariableAnnotationNode from file:/home/hadoop/presto/presto-server-0.199/lib/asm-tree-6.0.jar]
[Loaded org.objectweb.asm.tree.MultiANewArrayInsnNode from file:/home/hadoop/presto/presto-server-0.199/lib/asm-tree-6.0.jar]
[Loaded org.objectweb.asm.tree.LookupSwitchInsnNode from file:/home/hadoop/presto/presto-server-0.199/lib/asm-tree-6.0.jar]
[Loaded org.objectweb.asm.tree.TableSwitchInsnNode from file:/home/hadoop/presto/presto-server-0.199/lib/asm-tree-6.0.jar]
[Loaded org.objectweb.asm.tree.IincInsnNode from file:/home/hadoop/presto/presto-server-0.199/lib/asm-tree-6.0.jar]
[Loaded org.objectweb.asm.tree.InvokeDynamicInsnNode from file:/home/hadoop/presto/presto-server-0.199/lib/asm-tree-6.0.jar]
[Loaded org.objectweb.asm.tree.FrameNode from file:/home/hadoop/presto/presto-server-0.199/lib/asm-tree-6.0.jar]
[Loaded org.objectweb.asm.tree.LabelNode from file:/home/hadoop/presto/presto-server-0.199/lib/asm-tree-6.0.jar]
[Loaded org.objectweb.asm.tree.JumpInsnNode from file:/home/hadoop/presto/presto-server-0.199/lib/asm-tree-6.0.jar]
[Loaded org.objectweb.asm.tree.LdcInsnNode from file:/home/hadoop/presto/presto-server-0.199/lib/asm-tree-6.0.jar]
[Loaded org.objectweb.asm.tree.TypeInsnNode from file:/home/hadoop/presto/presto-server-0.199/lib/asm-tree-6.0.jar]
[Loaded org.objectweb.asm.tree.IntInsnNode from file:/home/hadoop/presto/presto-server-0.199/lib/asm-tree-6.0.jar]
[Loaded org.objectweb.asm.tree.InsnNode from file:/home/hadoop/presto/presto-server-0.199/lib/asm-tree-6.0.jar]
[Loaded org.objectweb.asm.tree.FieldInsnNode from file:/home/hadoop/presto/presto-server-0.199/lib/asm-tree-6.0.jar]
[Loaded org.objectweb.asm.tree.MethodInsnNode from file:/home/hadoop/presto/presto-server-0.199/lib/asm-tree-6.0.jar]
[Loaded org.objectweb.asm.tree.VarInsnNode from file:/home/hadoop/presto/presto-server-0.199/lib/asm-tree-6.0.jar]
[Loaded org.objectweb.asm.tree.InsnList from file:/home/hadoop/presto/presto-server-0.199/lib/asm-tree-6.0.jar]
[Loaded io.airlift.bytecode.MethodGenerationContext from file:/home/hadoop/presto/presto-server-0.199/lib/bytecode-1.0.jar]
[Loaded io.airlift.bytecode.MethodGenerationContext$ScopeContext from file:/home/hadoop/presto/presto-server-0.199/lib/bytecode-1.0.jar]
[Loaded io.airlift.bytecode.instruction.LabelNode from file:/home/hadoop/presto/presto-server-0.199/lib/bytecode-1.0.jar]
[Loaded org.objectweb.asm.Label from file:/home/hadoop/presto/presto-server-0.199/lib/asm-6.0.jar]
[Loaded org.objectweb.asm.Type from file:/home/hadoop/presto/presto-server-0.199/lib/asm-6.0.jar]
[Loaded io.airlift.bytecode.instruction.VariableInstruction from file:/home/hadoop/presto/presto-server-0.199/lib/bytecode-1.0.jar]
[Loaded io.airlift.bytecode.instruction.VariableInstruction$LoadVariableInstruction from file:/home/hadoop/presto/presto-server-0.199/lib/bytecode-1.0.jar]
[Loaded io.airlift.bytecode.debug.DebugNode from file:/home/hadoop/presto/presto-server-0.199/lib/bytecode-1.0.jar]
[Loaded io.airlift.bytecode.debug.LocalVariableNode from file:/home/hadoop/presto/presto-server-0.199/lib/bytecode-1.0.jar]
[Loaded org.objectweb.asm.tree.LocalVariableNode from file:/home/hadoop/presto/presto-server-0.199/lib/asm-tree-6.0.jar]
[Loaded io.airlift.bytecode.instruction.InvokeInstruction$InvokeDynamicInstruction from file:/home/hadoop/presto/presto-server-0.199/lib/bytecode-1.0.jar]
[Loaded io.airlift.bytecode.instruction.InvokeInstruction$$Lambda$1097/927817130 from io.airlift.bytecode.instruction.InvokeInstruction]
[Loaded org.objectweb.asm.Handle from file:/home/hadoop/presto/presto-server-0.199/lib/asm-6.0.jar]
[Loaded io.airlift.bytecode.MethodDefinition$$Lambda$1098/1621488877 from io.airlift.bytecode.MethodDefinition]
[Loaded io.airlift.bytecode.instruction.VariableInstruction$StoreVariableInstruction from file:/home/hadoop/presto/presto-server-0.199/lib/bytecode-1.0.jar]
[Loaded org.objectweb.asm.ModuleWriter from file:/home/hadoop/presto/presto-server-0.199/lib/asm-6.0.jar]
[Loaded org.objectweb.asm.AnnotationWriter from file:/home/hadoop/presto/presto-server-0.199/lib/asm-6.0.jar]
[Loaded org.objectweb.asm.MethodWriter from file:/home/hadoop/presto/presto-server-0.199/lib/asm-6.0.jar]
[Loaded org.objectweb.asm.FieldWriter from file:/home/hadoop/presto/presto-server-0.199/lib/asm-6.0.jar]
[Loaded org.objectweb.asm.ByteVector from file:/home/hadoop/presto/presto-server-0.199/lib/asm-6.0.jar]
[Loaded org.objectweb.asm.Item from file:/home/hadoop/presto/presto-server-0.199/lib/asm-6.0.jar]
[Loaded org.objectweb.asm.Frame from file:/home/hadoop/presto/presto-server-0.199/lib/asm-6.0.jar]
[Loaded org.objectweb.asm.CurrentFrame from file:/home/hadoop/presto/presto-server-0.199/lib/asm-6.0.jar]
[Loaded io.airlift.bytecode.ClassGenerator$$Lambda$1099/1365730836 from io.airlift.bytecode.ClassGenerator]
[Loaded com.facebook.presto.$gen.LongStateSerializer_20180528_075733_1 from __JVM_DefineClass__]
[Loaded com.google.common.reflect.Reflection from file:/home/hadoop/presto/presto-server-0.199/lib/guava-24.1-jre.jar]
[Loaded io.airlift.bytecode.instruction.Constant from file:/home/hadoop/presto/presto-server-0.199/lib/bytecode-1.0.jar]
[Loaded io.airlift.bytecode.instruction.Constant$BoxedDoubleConstant from file:/home/hadoop/presto/presto-server-0.199/lib/bytecode-1.0.jar]
[Loaded io.airlift.bytecode.instruction.Constant$BoxedBooleanConstant from file:/home/hadoop/presto/presto-server-0.199/lib/bytecode-1.0.jar]
[Loaded io.airlift.bytecode.instruction.Constant$BoxedFloatConstant from file:/home/hadoop/presto/presto-server-0.199/lib/bytecode-1.0.jar]
[Loaded io.airlift.bytecode.instruction.Constant$BoxedLongConstant from file:/home/hadoop/presto/presto-server-0.199/lib/bytecode-1.0.jar]
[Loaded io.airlift.bytecode.instruction.Constant$BoxedIntegerConstant from file:/home/hadoop/presto/presto-server-0.199/lib/bytecode-1.0.jar]
[Loaded io.airlift.bytecode.instruction.Constant$StringConstant from file:/home/hadoop/presto/presto-server-0.199/lib/bytecode-1.0.jar]
[Loaded io.airlift.bytecode.instruction.Constant$IntConstant from file:/home/hadoop/presto/presto-server-0.199/lib/bytecode-1.0.jar]
[Loaded io.airlift.bytecode.instruction.Constant$BooleanConstant from file:/home/hadoop/presto/presto-server-0.199/lib/bytecode-1.0.jar]
[Loaded io.airlift.bytecode.instruction.Constant$FloatConstant from file:/home/hadoop/presto/presto-server-0.199/lib/bytecode-1.0.jar]
[Loaded io.airlift.bytecode.instruction.Constant$DoubleConstant from file:/home/hadoop/presto/presto-server-0.199/lib/bytecode-1.0.jar]
[Loaded io.airlift.bytecode.instruction.Constant$LongConstant from file:/home/hadoop/presto/presto-server-0.199/lib/bytecode-1.0.jar]
[Loaded io.airlift.bytecode.instruction.Constant$NullConstant from file:/home/hadoop/presto/presto-server-0.199/lib/bytecode-1.0.jar]
[Loaded io.airlift.bytecode.instruction.Constant$ClassConstant from file:/home/hadoop/presto/presto-server-0.199/lib/bytecode-1.0.jar]
[Loaded io.airlift.bytecode.instruction.InvokeInstruction$$Lambda$1100/1492870279 from io.airlift.bytecode.instruction.InvokeInstruction]
[Loaded io.airlift.bytecode.instruction.FieldInstruction from file:/home/hadoop/presto/presto-server-0.199/lib/bytecode-1.0.jar]
[Loaded io.airlift.bytecode.instruction.FieldInstruction$GetFieldInstruction from file:/home/hadoop/presto/presto-server-0.199/lib/bytecode-1.0.jar]
[Loaded io.airlift.bytecode.instruction.FieldInstruction$PutFieldInstruction from file:/home/hadoop/presto/presto-server-0.199/lib/bytecode-1.0.jar]
[Loaded io.airlift.bytecode.instruction.InvokeInstruction$$Lambda$1101/686433535 from io.airlift.bytecode.instruction.InvokeInstruction]
[Loaded io.airlift.bytecode.expression.BytecodeExpressions from file:/home/hadoop/presto/presto-server-0.199/lib/bytecode-1.0.jar]
[Loaded io.airlift.bytecode.expression.ConstantBytecodeExpression from file:/home/hadoop/presto/presto-server-0.199/lib/bytecode-1.0.jar]
[Loaded io.airlift.bytecode.expression.NegateBytecodeExpression from file:/home/hadoop/presto/presto-server-0.199/lib/bytecode-1.0.jar]
[Loaded io.airlift.bytecode.expression.InlineIfBytecodeExpression from file:/home/hadoop/presto/presto-server-0.199/lib/bytecode-1.0.jar]
[Loaded io.airlift.bytecode.expression.NotBytecodeExpression from file:/home/hadoop/presto/presto-server-0.199/lib/bytecode-1.0.jar]
[Loaded io.airlift.bytecode.expression.AndBytecodeExpression from file:/home/hadoop/presto/presto-server-0.199/lib/bytecode-1.0.jar]
[Loaded io.airlift.bytecode.expression.OrBytecodeExpression from file:/home/hadoop/presto/presto-server-0.199/lib/bytecode-1.0.jar]
[Loaded io.airlift.bytecode.expression.InvokeDynamicBytecodeExpression from file:/home/hadoop/presto/presto-server-0.199/lib/bytecode-1.0.jar]
[Loaded io.airlift.bytecode.expression.NewArrayBytecodeExpression from file:/home/hadoop/presto/presto-server-0.199/lib/bytecode-1.0.jar]
[Loaded io.airlift.bytecode.expression.NewInstanceBytecodeExpression from file:/home/hadoop/presto/presto-server-0.199/lib/bytecode-1.0.jar]
2018-05-28T07:57:33.272Z DEBUG Query-20180528_075336_00001_j36cu-166 com.facebook.presto.util.CompilerUtils Defining class: com/facebook/presto/$gen/SingleLongState_20180528_075733_2
[Loaded com.facebook.presto.$gen.SingleLongState_20180528_075733_2 from __JVM_DefineClass__]
[Loaded com.facebook.presto.spi.function.GroupedAccumulatorState from file:/home/hadoop/presto/presto-server-0.199/lib/presto-spi-0.199.jar]
[Loaded com.facebook.presto.operator.aggregation.state.AbstractGroupedAccumulatorState from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.operator.aggregation.GroupedAccumulator from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.array.LongBigArray from file:/home/hadoop/presto/presto-server-0.199/lib/presto-array-0.199.jar]
[Loaded io.airlift.bytecode.expression.BytecodeExpressions$$Lambda$1102/260212780 from io.airlift.bytecode.expression.BytecodeExpressions]
[Loaded io.airlift.bytecode.expression.ArithmeticBytecodeExpression from file:/home/hadoop/presto/presto-server-0.199/lib/bytecode-1.0.jar]
[Loaded io.airlift.bytecode.expression.ArithmeticBytecodeExpression$1 from file:/home/hadoop/presto/presto-server-0.199/lib/bytecode-1.0.jar]
2018-05-28T07:57:33.277Z DEBUG Query-20180528_075336_00001_j36cu-166 com.facebook.presto.util.CompilerUtils Defining class: com/facebook/presto/$gen/GroupedLongState_20180528_075733_3
[Loaded com.facebook.presto.$gen.GroupedLongState_20180528_075733_3 from __JVM_DefineClass__]
2018-05-28T07:57:33.280Z DEBUG Query-20180528_075336_00001_j36cu-166 com.facebook.presto.util.CompilerUtils Defining class: com/facebook/presto/$gen/LongStateFactory_20180528_075733_4
[Loaded com.facebook.presto.$gen.LongStateFactory_20180528_075733_4 from __JVM_DefineClass__]
[Loaded com.facebook.presto.operator.aggregation.ParametricAggregation$1 from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.operator.aggregation.AggregationMetadata$ParameterMetadata from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.operator.aggregation.ParametricAggregation$$Lambda$1103/1381000388 from com.facebook.presto.operator.aggregation.ParametricAggregation]
[Loaded com.facebook.presto.operator.aggregation.AggregationUtils from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.operator.aggregation.AggregationMetadata from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.operator.aggregation.AggregationMetadata$$Lambda$1104/1346358136 from com.facebook.presto.operator.aggregation.AggregationMetadata]
[Loaded com.facebook.presto.operator.aggregation.AggregationMetadata$1 from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.operator.aggregation.AggregationMetadata$$Lambda$1105/1744215814 from com.facebook.presto.operator.aggregation.AggregationMetadata]
[Loaded com.facebook.presto.operator.aggregation.AggregationMetadata$$Lambda$1106/1228733579 from com.facebook.presto.operator.aggregation.AggregationMetadata]
[Loaded com.facebook.presto.operator.aggregation.AggregationMetadata$$Lambda$1107/2043739460 from com.facebook.presto.operator.aggregation.AggregationMetadata]
[Loaded java.lang.invoke.LambdaForm$MH/576628293 from java.lang.invoke.LambdaForm]
[Loaded java.lang.invoke.LambdaForm$BMH/1904411559 from java.lang.invoke.LambdaForm]
[Loaded java.lang.invoke.LambdaForm$BMH/1601528248 from java.lang.invoke.LambdaForm]
[Loaded com.facebook.presto.operator.aggregation.LazyAccumulatorFactoryBinder from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.operator.aggregation.LazyAccumulatorFactoryBinder$$Lambda$1108/1174613469 from com.facebook.presto.operator.aggregation.LazyAccumulatorFactoryBinder]
[Loaded com.facebook.presto.sql.planner.iterative.rule.PushPartialAggregationThroughExchange$$Lambda$1109/1677040964 from com.facebook.presto.sql.planner.iterative.rule.PushPartialAggregationThroughExchange]
[Loaded com.facebook.presto.sql.planner.iterative.rule.PushPartialAggregationThroughExchange$$Lambda$1110/310036461 from com.facebook.presto.sql.planner.iterative.rule.PushPartialAggregationThroughExchange]
[Loaded com.facebook.presto.sql.planner.iterative.rule.PushPartialAggregationThroughExchange$1 from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.sql.planner.optimizations.SymbolMapper$Builder from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.sql.planner.optimizations.HashGenerationOptimizer$HashComputationSet from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.sql.planner.optimizations.HashGenerationOptimizer$HashComputation from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.sql.planner.optimizations.HashGenerationOptimizer$HashComputationSet$$Lambda$1111/698900157 from com.facebook.presto.sql.planner.optimizations.HashGenerationOptimizer$HashComputationSet]
[Loaded com.facebook.presto.sql.planner.optimizations.HashGenerationOptimizer$Rewriter$$Lambda$1112/1908339008 from com.facebook.presto.sql.planner.optimizations.HashGenerationOptimizer$Rewriter]
[Loaded com.facebook.presto.sql.planner.optimizations.HashGenerationOptimizer$Rewriter$$Lambda$1113/966955564 from com.facebook.presto.sql.planner.optimizations.HashGenerationOptimizer$Rewriter]
[Loaded com.facebook.presto.sql.planner.optimizations.HashGenerationOptimizer$Rewriter$$Lambda$1114/858872137 from com.facebook.presto.sql.planner.optimizations.HashGenerationOptimizer$Rewriter]
[Loaded com.facebook.presto.sql.planner.optimizations.HashGenerationOptimizer$HashComputationSet$$Lambda$1115/897817246 from com.facebook.presto.sql.planner.optimizations.HashGenerationOptimizer$HashComputationSet]
[Loaded com.facebook.presto.sql.planner.optimizations.HashGenerationOptimizer$HashComputationSet$$Lambda$1116/404573680 from com.facebook.presto.sql.planner.optimizations.HashGenerationOptimizer$HashComputationSet]
[Loaded com.facebook.presto.sql.planner.optimizations.HashGenerationOptimizer$HashComputationSet$$Lambda$1117/2109024792 from com.facebook.presto.sql.planner.optimizations.HashGenerationOptimizer$HashComputationSet]
[Loaded com.facebook.presto.sql.planner.optimizations.HashGenerationOptimizer$Rewriter$$Lambda$1118/1157091318 from com.facebook.presto.sql.planner.optimizations.HashGenerationOptimizer$Rewriter]
[Loaded com.facebook.presto.sql.planner.optimizations.HashGenerationOptimizer$Rewriter$$Lambda$1119/1829366428 from com.facebook.presto.sql.planner.optimizations.HashGenerationOptimizer$Rewriter]
[Loaded com.facebook.presto.sql.planner.optimizations.HashGenerationOptimizer$Rewriter$$Lambda$1120/529094656 from com.facebook.presto.sql.planner.optimizations.HashGenerationOptimizer$Rewriter]
[Loaded com.facebook.presto.sql.planner.optimizations.HashGenerationOptimizer$PlanWithProperties from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.sql.planner.optimizations.HashGenerationOptimizer$Rewriter$$Lambda$1121/1185544316 from com.facebook.presto.sql.planner.optimizations.HashGenerationOptimizer$Rewriter]
[Loaded com.facebook.presto.sql.planner.optimizations.HashGenerationOptimizer$PlanWithProperties$$Lambda$1122/1337468150 from com.facebook.presto.sql.planner.optimizations.HashGenerationOptimizer$PlanWithProperties]
[Loaded com.facebook.presto.sql.planner.plan.TableWriterNode$CreateHandle from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.sql.planner.plan.TableWriterNode$InsertHandle from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.sql.planner.optimizations.BeginTableWrite$Context from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.sql.planner.sanity.PlanSanityChecker$$Lambda$1123/644217010 from com.facebook.presto.sql.planner.sanity.PlanSanityChecker]
[Loaded com.facebook.presto.sql.planner.sanity.VerifyNoFilteredAggregations$$Lambda$1124/1015368556 from com.facebook.presto.sql.planner.sanity.VerifyNoFilteredAggregations]
[Loaded com.facebook.presto.sql.planner.sanity.VerifyNoFilteredAggregations$$Lambda$1125/661798241 from com.facebook.presto.sql.planner.sanity.VerifyNoFilteredAggregations]
[Loaded com.facebook.presto.sql.planner.sanity.VerifyNoFilteredAggregations$$Lambda$1126/559208295 from com.facebook.presto.sql.planner.sanity.VerifyNoFilteredAggregations]
[Loaded com.facebook.presto.sql.planner.sanity.VerifyNoFilteredAggregations$$Lambda$1127/924595722 from com.facebook.presto.sql.planner.sanity.VerifyNoFilteredAggregations]
[Loaded com.facebook.presto.sql.planner.sanity.ValidateAggregationsWithDefaultValues$Visitor$$Lambda$1128/1976840680 from com.facebook.presto.sql.planner.sanity.ValidateAggregationsWithDefaultValues$Visitor]
[Loaded com.facebook.presto.sql.planner.sanity.ValidateAggregationsWithDefaultValues$Visitor$$Lambda$1129/1916547034 from com.facebook.presto.sql.planner.sanity.ValidateAggregationsWithDefaultValues$Visitor]
[Loaded com.facebook.presto.sql.planner.sanity.ValidateAggregationsWithDefaultValues$SeenExchanges from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.sql.planner.InputExtractor from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.sql.planner.InputExtractor$Visitor from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.execution.Column from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.sql.planner.InputExtractor$$Lambda$1130/1681884663 from com.facebook.presto.sql.planner.InputExtractor]
[Loaded com.facebook.presto.execution.Input from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.sql.planner.OutputExtractor from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.sql.planner.OutputExtractor$Visitor from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.sql.planner.PlanFragmenter from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.sql.planner.PlanFragmenter$GroupedExecutionTagger from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.sql.planner.PlanFragmenter$Fragmenter from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.sql.planner.PlanFragmenter$FragmentProperties from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.sql.planner.PartitioningScheme$$Lambda$1131/233458371 from com.facebook.presto.sql.planner.PartitioningScheme]
[Loaded com.facebook.presto.sql.planner.PartitioningScheme$$Lambda$1132/1763308145 from com.facebook.presto.sql.planner.PartitioningScheme]
[Loaded java.lang.invoke.LambdaForm$DMH/968567287 from java.lang.invoke.LambdaForm]
[Loaded com.facebook.presto.sql.planner.PartitioningScheme$$Lambda$1133/130473759 from com.facebook.presto.sql.planner.PartitioningScheme]
[Loaded com.facebook.presto.sql.planner.plan.PlanFragmentId from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.sql.planner.PlanFragmenter$Fragmenter$$Lambda$1134/798145077 from com.facebook.presto.sql.planner.PlanFragmenter$Fragmenter]
[Loaded com.facebook.presto.sql.planner.PlanFragmenter$Fragmenter$$Lambda$1135/1582104624 from com.facebook.presto.sql.planner.PlanFragmenter$Fragmenter]
[Loaded com.facebook.presto.sql.planner.PlanFragmenter$Fragmenter$$Lambda$1136/491754840 from com.facebook.presto.sql.planner.PlanFragmenter$Fragmenter]
[Loaded com.facebook.presto.sql.planner.SymbolsExtractor$$Lambda$1137/290197376 from com.facebook.presto.sql.planner.SymbolsExtractor]
[Loaded com.facebook.presto.sql.planner.SchedulingOrderVisitor from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.sql.planner.SchedulingOrderVisitor$Visitor from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.sql.planner.SchedulingOrderVisitor$$Lambda$1138/75412827 from com.facebook.presto.sql.planner.SchedulingOrderVisitor]
[Loaded com.facebook.presto.operator.PipelineExecutionStrategy from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.sql.planner.PlanFragment$$Lambda$1139/705025222 from com.facebook.presto.sql.planner.PlanFragment]
[Loaded com.facebook.presto.sql.planner.PlanFragmenter$Fragmenter$$Lambda$1140/226355057 from com.facebook.presto.sql.planner.PlanFragmenter$Fragmenter]
[Loaded com.facebook.presto.sql.planner.PlanFragmenter$Fragmenter$$Lambda$1141/108190605 from com.facebook.presto.sql.planner.PlanFragmenter$Fragmenter]
[Loaded com.facebook.presto.sql.planner.PlanFragmenter$GroupedExecutionProperties from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.sql.planner.SubPlan$$Lambda$1142/2058062241 from com.facebook.presto.sql.planner.SubPlan]
[Loaded com.facebook.presto.sql.planner.SubPlan$$Lambda$1143/1537969285 from com.facebook.presto.sql.planner.SubPlan]
[Loaded com.google.common.collect.ImmutableMultiset$$Lambda$1144/2135112813 from com.google.common.collect.ImmutableMultiset]
[Loaded com.google.common.collect.ImmutableMultiset$$Lambda$1145/399977414 from com.google.common.collect.ImmutableMultiset]
[Loaded com.google.common.collect.ImmutableMultiset$$Lambda$1146/1896996101 from com.google.common.collect.ImmutableMultiset]
[Loaded com.google.common.collect.ImmutableMultiset$$Lambda$1147/846498302 from com.google.common.collect.ImmutableMultiset]
[Loaded com.google.common.collect.ImmutableMultiset$$Lambda$1148/899511696 from com.google.common.collect.ImmutableMultiset]
[Loaded com.facebook.presto.sql.planner.SubPlan$$Lambda$1149/465236478 from com.facebook.presto.sql.planner.SubPlan]
[Loaded com.facebook.presto.sql.planner.SubPlan$$Lambda$1150/1690107513 from com.facebook.presto.sql.planner.SubPlan]
[Loaded com.google.common.collect.ImmutableSet$Indexed$1 from file:/home/hadoop/presto/presto-server-0.199/lib/guava-24.1-jre.jar]
[Loaded com.facebook.presto.execution.SqlQueryExecution$PlanRoot from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.sql.planner.DistributedExecutionPlanner from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.sql.planner.DistributedExecutionPlanner$Visitor from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.spi.FixedSplitSource from file:/home/hadoop/presto/presto-server-0.199/lib/presto-spi-0.199.jar]
[Loaded com.facebook.presto.cassandra.util.HostAddressFactory from file:/mnt/var/lib/presto/data/plugin/cassandra/presto-cassandra-0.199.jar]
[Loaded com.facebook.presto.cassandra.NativeCassandraSession$$Lambda$1151/429858357 from com.facebook.presto.cassandra.NativeCassandraSession]
[Loaded com.facebook.presto.split.ConnectorAwareSplitSource from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.split.BufferingSplitSource from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.sql.planner.StageExecutionPlan from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.execution.SqlQueryExecution$$Lambda$1152/1305514645 from com.facebook.presto.execution.SqlQueryExecution]
[Loaded com.facebook.presto.execution.QueryStateMachine$QueryOutputManager$$Lambda$1153/1285098252 from com.facebook.presto.execution.QueryStateMachine$QueryOutputManager]
[Loaded com.facebook.presto.execution.QueryStateMachine$QueryOutputManager$$Lambda$1154/554866398 from com.facebook.presto.execution.QueryStateMachine$QueryOutputManager]
[Loaded com.facebook.presto.OutputBuffers$BufferType from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.client.ClientTypeSignature from file:/home/hadoop/presto/presto-server-0.199/lib/presto-client-0.199.jar]
[Loaded com.facebook.presto.execution.scheduler.SplitPlacementPolicy from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.client.ClientTypeSignatureParameter from file:/home/hadoop/presto/presto-server-0.199/lib/presto-client-0.199.jar]
[Loaded com.facebook.presto.client.ClientTypeSignature$$Lambda$1155/1862514016 from com.facebook.presto.client.ClientTypeSignature]
[Loaded com.facebook.presto.client.ClientTypeSignatureParameter$1 from file:/home/hadoop/presto/presto-server-0.199/lib/presto-client-0.199.jar]
[Loaded java.lang.invoke.LambdaForm$DMH/1361222841 from java.lang.invoke.LambdaForm]
[Loaded com.facebook.presto.execution.scheduler.SqlQueryScheduler$ExchangeLocationsConsumer from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.execution.scheduler.SqlQueryScheduler$$Lambda$1156/1062544795 from com.facebook.presto.execution.scheduler.SqlQueryScheduler]
[Loaded com.facebook.presto.execution.scheduler.SqlQueryScheduler$$Lambda$1157/892742110 from com.facebook.presto.execution.scheduler.SqlQueryScheduler]
[Loaded com.facebook.presto.execution.SqlStageExecution from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.execution.StageStateMachine from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.execution.StageState from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.execution.StageState$$Lambda$1158/1485614842 from com.facebook.presto.execution.StageState]
[Loaded com.facebook.presto.execution.StageStateMachine$$Lambda$1159/2100003229 from com.facebook.presto.execution.StageStateMachine]
[Loaded com.facebook.presto.execution.SqlStageExecution$ListenerManager from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.execution.scheduler.SqlQueryScheduler$$Lambda$1160/1035726996 from com.facebook.presto.execution.scheduler.SqlQueryScheduler]
[Loaded com.facebook.presto.execution.scheduler.NodeScheduler$$Lambda$1161/1287380586 from com.facebook.presto.execution.scheduler.NodeScheduler]
[Loaded com.facebook.presto.execution.scheduler.TopologyAwareNodeSelector from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.execution.scheduler.NodeScheduler$$Lambda$1162/409044 from com.facebook.presto.execution.scheduler.NodeScheduler]
[Loaded com.google.common.cache.LocalCache$StrongWriteEntry from file:/home/hadoop/presto/presto-server-0.199/lib/guava-24.1-jre.jar]
[Loaded com.google.common.cache.CacheLoader$1$1 from file:/home/hadoop/presto/presto-server-0.199/lib/guava-24.1-jre.jar]
[Loaded com.google.common.util.concurrent.ListenableFutureTask from file:/home/hadoop/presto/presto-server-0.199/lib/guava-24.1-jre.jar]
[Loaded com.google.common.util.concurrent.ExecutionList from file:/home/hadoop/presto/presto-server-0.199/lib/guava-24.1-jre.jar]
[Loaded com.google.common.cache.LocalCache$LoadingValueReference$1 from file:/home/hadoop/presto/presto-server-0.199/lib/guava-24.1-jre.jar]
[Loaded com.google.common.cache.LocalCache$Segment$1 from file:/home/hadoop/presto/presto-server-0.199/lib/guava-24.1-jre.jar]
[Loaded com.google.common.cache.RemovalCause from file:/home/hadoop/presto/presto-server-0.199/lib/guava-24.1-jre.jar]
[Loaded com.google.common.cache.RemovalCause$1 from file:/home/hadoop/presto/presto-server-0.199/lib/guava-24.1-jre.jar]
[Loaded com.google.common.cache.RemovalCause$2 from file:/home/hadoop/presto/presto-server-0.199/lib/guava-24.1-jre.jar]
[Loaded com.google.common.cache.RemovalCause$3 from file:/home/hadoop/presto/presto-server-0.199/lib/guava-24.1-jre.jar]
[Loaded com.google.common.cache.RemovalCause$4 from file:/home/hadoop/presto/presto-server-0.199/lib/guava-24.1-jre.jar]
[Loaded com.google.common.cache.RemovalCause$5 from file:/home/hadoop/presto/presto-server-0.199/lib/guava-24.1-jre.jar]
[Loaded java.lang.invoke.LambdaForm$DMH/304805249 from java.lang.invoke.LambdaForm]
[Loaded com.facebook.presto.execution.scheduler.NodeScheduler$$Lambda$1163/1208498873 from com.facebook.presto.execution.scheduler.NodeScheduler]
[Loaded java.lang.invoke.LambdaForm$DMH/1289560817 from java.lang.invoke.LambdaForm]
[Loaded java.lang.invoke.LambdaForm$MH/118364338 from java.lang.invoke.LambdaForm]
[Loaded com.facebook.presto.execution.scheduler.NodeScheduler$$Lambda$1164/1112372015 from com.facebook.presto.execution.scheduler.NodeScheduler]
[Loaded com.facebook.presto.sql.planner.SystemPartitioningHandle$$Lambda$1165/432698353 from com.facebook.presto.sql.planner.SystemPartitioningHandle]
[Loaded java.util.stream.IntPipeline$$Lambda$1166/1850683482 from java.util.stream.IntPipeline]
[Loaded java.util.stream.Nodes from /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.171-7.b10.37.amzn1.x86_64/jre/lib/rt.jar]
[Loaded java.util.stream.Node from /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.171-7.b10.37.amzn1.x86_64/jre/lib/rt.jar]
[Loaded java.util.stream.Nodes$EmptyNode from /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.171-7.b10.37.amzn1.x86_64/jre/lib/rt.jar]
[Loaded java.util.stream.Nodes$EmptyNode$OfRef from /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.171-7.b10.37.amzn1.x86_64/jre/lib/rt.jar]
[Loaded java.util.stream.Node$OfPrimitive from /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.171-7.b10.37.amzn1.x86_64/jre/lib/rt.jar]
[Loaded java.util.stream.Node$OfInt from /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.171-7.b10.37.amzn1.x86_64/jre/lib/rt.jar]
[Loaded java.util.stream.Nodes$EmptyNode$OfInt from /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.171-7.b10.37.amzn1.x86_64/jre/lib/rt.jar]
[Loaded java.util.stream.Node$OfLong from /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.171-7.b10.37.amzn1.x86_64/jre/lib/rt.jar]
[Loaded java.util.stream.Nodes$EmptyNode$OfLong from /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.171-7.b10.37.amzn1.x86_64/jre/lib/rt.jar]
[Loaded java.util.stream.Node$OfDouble from /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.171-7.b10.37.amzn1.x86_64/jre/lib/rt.jar]
[Loaded java.util.stream.Nodes$EmptyNode$OfDouble from /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.171-7.b10.37.amzn1.x86_64/jre/lib/rt.jar]
[Loaded java.util.stream.Node$Builder from /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.171-7.b10.37.amzn1.x86_64/jre/lib/rt.jar]
[Loaded java.util.stream.Node$Builder$OfInt from /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.171-7.b10.37.amzn1.x86_64/jre/lib/rt.jar]
[Loaded java.util.stream.Nodes$IntArrayNode from /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.171-7.b10.37.amzn1.x86_64/jre/lib/rt.jar]
[Loaded java.util.stream.Nodes$IntFixedNodeBuilder from /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.171-7.b10.37.amzn1.x86_64/jre/lib/rt.jar]
[Loaded com.facebook.presto.execution.scheduler.StageScheduler from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.execution.scheduler.FixedCountScheduler from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.execution.scheduler.FixedCountScheduler$$Lambda$1167/375578736 from com.facebook.presto.execution.scheduler.FixedCountScheduler]
[Loaded com.facebook.presto.execution.scheduler.SqlQueryScheduler$$Lambda$1168/420454797 from com.facebook.presto.execution.scheduler.SqlQueryScheduler]
[Loaded com.facebook.presto.execution.scheduler.DynamicSplitPlacementPolicy from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.execution.scheduler.SqlQueryScheduler$$Lambda$1169/1445576732 from com.facebook.presto.execution.scheduler.SqlQueryScheduler]
[Loaded com.facebook.presto.execution.scheduler.SourcePartitionedScheduler from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.execution.scheduler.SourcePartitionedScheduler$State from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.execution.Lifespan from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.spi.connector.ConnectorPartitionHandle from file:/home/hadoop/presto/presto-server-0.199/lib/presto-spi-0.199.jar]
[Loaded com.facebook.presto.spi.connector.NotPartitionedPartitionHandle from file:/home/hadoop/presto/presto-server-0.199/lib/presto-spi-0.199.jar]
[Loaded com.facebook.presto.execution.scheduler.SourcePartitionedScheduler$ScheduleGroup from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.execution.scheduler.SourcePartitionedScheduler$ScheduleGroupState from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.execution.scheduler.SqlQueryScheduler$$Lambda$1170/2063594309 from com.facebook.presto.execution.scheduler.SqlQueryScheduler]
[Loaded com.facebook.presto.execution.scheduler.SqlQueryScheduler$StageLinkage from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.execution.scheduler.OutputBufferManager from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.execution.scheduler.SqlQueryScheduler$StageLinkage$$Lambda$1171/450645723 from com.facebook.presto.execution.scheduler.SqlQueryScheduler$StageLinkage]
[Loaded com.facebook.presto.execution.scheduler.SqlQueryScheduler$StageLinkage$$Lambda$1172/2057379262 from com.facebook.presto.execution.scheduler.SqlQueryScheduler$StageLinkage]
[Loaded com.facebook.presto.execution.scheduler.PartitionedOutputBufferManager from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.execution.scheduler.SqlQueryScheduler$StageLinkage$$Lambda$1173/1843438490 from com.facebook.presto.execution.scheduler.SqlQueryScheduler$StageLinkage]
[Loaded com.facebook.presto.execution.SqlStageExecution$$Lambda$1174/151000889 from com.facebook.presto.execution.SqlStageExecution]
[Loaded com.facebook.presto.execution.scheduler.SqlQueryScheduler$$Lambda$1175/1877391481 from com.facebook.presto.execution.scheduler.SqlQueryScheduler]
[Loaded com.facebook.presto.execution.scheduler.SqlQueryScheduler$$Lambda$1176/1846394029 from com.facebook.presto.execution.scheduler.SqlQueryScheduler]
[Loaded com.facebook.presto.execution.scheduler.SqlQueryScheduler$$Lambda$1177/1915813108 from com.facebook.presto.execution.scheduler.SqlQueryScheduler]
[Loaded com.facebook.presto.execution.QueryStateMachine$$Lambda$1178/581196813 from com.facebook.presto.execution.QueryStateMachine]
2018-05-28T07:57:33.475Z DEBUG query-execution-2 com.facebook.presto.execution.QueryStateMachine Query 20180528_075336_00001_j36cu is STARTING
[Loaded com.facebook.presto.execution.scheduler.SqlQueryScheduler$$Lambda$1179/2075143714 from com.facebook.presto.execution.scheduler.SqlQueryScheduler]
[Loaded com.google.common.collect.ImmutableList$$Lambda$1180/886123959 from com.google.common.collect.ImmutableList]
[Loaded com.facebook.presto.execution.scheduler.SqlQueryScheduler$$Lambda$1181/640794953 from com.facebook.presto.execution.scheduler.SqlQueryScheduler]
[Loaded com.facebook.presto.execution.scheduler.SqlQueryScheduler$$Lambda$1182/150610486 from com.facebook.presto.execution.scheduler.SqlQueryScheduler]
[Loaded com.facebook.presto.execution.scheduler.SqlQueryScheduler$$Lambda$1183/960532284 from com.facebook.presto.execution.scheduler.SqlQueryScheduler]
[Loaded com.facebook.presto.execution.scheduler.AllAtOnceExecutionSchedule from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.execution.SqlStageExecution$$Lambda$1184/79798582 from com.facebook.presto.execution.SqlStageExecution]
[Loaded com.facebook.presto.execution.SqlStageExecution$$Lambda$1185/743576430 from com.facebook.presto.execution.SqlStageExecution]
[Loaded com.facebook.presto.execution.scheduler.AllAtOnceExecutionSchedule$$Lambda$1186/1192395591 from com.facebook.presto.execution.scheduler.AllAtOnceExecutionSchedule]
[Loaded com.facebook.presto.execution.SqlStageExecution$$Lambda$1187/1977646912 from com.facebook.presto.execution.SqlStageExecution]
[Loaded com.facebook.presto.execution.scheduler.AllAtOnceExecutionSchedule$$Lambda$1188/989212192 from com.facebook.presto.execution.scheduler.AllAtOnceExecutionSchedule]
[Loaded com.facebook.presto.execution.scheduler.AllAtOnceExecutionSchedule$$Lambda$1189/1746747832 from com.facebook.presto.execution.scheduler.AllAtOnceExecutionSchedule]
[Loaded com.facebook.presto.execution.StageStats from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.execution.scheduler.AllAtOnceExecutionSchedule$$Lambda$1190/415044568 from com.facebook.presto.execution.scheduler.AllAtOnceExecutionSchedule]
[Loaded com.facebook.presto.execution.scheduler.AllAtOnceExecutionSchedule$$Lambda$1191/1265017651 from com.facebook.presto.execution.scheduler.AllAtOnceExecutionSchedule]
[Loaded com.facebook.presto.execution.scheduler.AllAtOnceExecutionSchedule$$Lambda$1192/1123588725 from com.facebook.presto.execution.scheduler.AllAtOnceExecutionSchedule]
[Loaded io.airlift.stats.QuantileDigest$2 from file:/home/hadoop/presto/presto-server-0.199/lib/stats-0.167.jar]
[Loaded com.google.common.base.MoreObjects$ToStringHelper from file:/home/hadoop/presto/presto-server-0.199/lib/guava-24.1-jre.jar]
[Loaded com.google.common.base.MoreObjects$ToStringHelper$ValueHolder from file:/home/hadoop/presto/presto-server-0.199/lib/guava-24.1-jre.jar]
[Loaded java.lang.invoke.LambdaForm$DMH/1653352639 from java.lang.invoke.LambdaForm]
[Loaded io.airlift.stats.QuantileDigest$$Lambda$1193/217840173 from io.airlift.stats.QuantileDigest]
[Loaded io.airlift.stats.QuantileDigest$$Lambda$1194/1937353634 from io.airlift.stats.QuantileDigest]
[Loaded com.facebook.presto.execution.scheduler.AllAtOnceExecutionSchedule$Visitor from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.spi.eventlistener.StageGcStatistics from file:/home/hadoop/presto/presto-server-0.199/lib/presto-spi-0.199.jar]
[Loaded com.facebook.presto.execution.scheduler.AllAtOnceExecutionSchedule$Visitor$$Lambda$1195/1395492093 from com.facebook.presto.execution.scheduler.AllAtOnceExecutionSchedule$Visitor]
[Loaded com.facebook.presto.execution.scheduler.AllAtOnceExecutionSchedule$Visitor$$Lambda$1196/1364933049 from com.facebook.presto.execution.scheduler.AllAtOnceExecutionSchedule$Visitor]
[Loaded com.google.common.collect.Ordering$IncomparableValueException from file:/home/hadoop/presto/presto-server-0.199/lib/guava-24.1-jre.jar]
[Loaded com.facebook.presto.execution.scheduler.SqlQueryScheduler$$Lambda$1197/1239756467 from com.facebook.presto.execution.scheduler.SqlQueryScheduler]
[Loaded com.facebook.presto.execution.scheduler.AllAtOnceExecutionSchedule$$Lambda$1198/1396377948 from com.facebook.presto.execution.scheduler.AllAtOnceExecutionSchedule]
[Loaded com.facebook.presto.execution.scheduler.AllAtOnceExecutionSchedule$$Lambda$1199/1537565811 from com.facebook.presto.execution.scheduler.AllAtOnceExecutionSchedule]
[Loaded com.facebook.presto.execution.StageInfo$$Lambda$1200/2004854934 from com.facebook.presto.execution.StageInfo]
[Loaded com.facebook.presto.split.BufferingSplitSource$GetNextBatch from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.execution.QueryStateMachine$$Lambda$1201/1526515644 from com.facebook.presto.execution.QueryStateMachine]
[Loaded com.facebook.presto.spi.ConnectorSplitSource$ConnectorSplitBatch from file:/home/hadoop/presto/presto-server-0.199/lib/presto-spi-0.199.jar]
[Loaded com.facebook.presto.execution.QueryStateMachine$$Lambda$1202/615814977 from com.facebook.presto.execution.QueryStateMachine]
[Loaded io.airlift.concurrent.MoreFutures$$Lambda$1203/1053071312 from io.airlift.concurrent.MoreFutures]
[Loaded java.lang.invoke.LambdaForm$DMH/508314343 from java.lang.invoke.LambdaForm]
2018-05-28T07:57:33.488Z DEBUG query-execution-3 com.facebook.presto.execution.StageStateMachine Stage 20180528_075336_00001_j36cu.2 is SCHEDULING
[Loaded java.lang.invoke.LambdaForm$MH/12219471 from java.lang.invoke.LambdaForm]
[Loaded com.facebook.presto.execution.QueryStateMachine$$Lambda$1204/1153738194 from com.facebook.presto.execution.QueryStateMachine]
[Loaded io.airlift.concurrent.MoreFutures$$Lambda$1205/1265562377 from io.airlift.concurrent.MoreFutures]
[Loaded com.facebook.presto.split.SplitSource$SplitBatch from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.split.ConnectorAwareSplitSource$$Lambda$1206/1819386412 from com.facebook.presto.split.ConnectorAwareSplitSource]
[Loaded com.facebook.presto.split.BufferingSplitSource$GetNextBatch$$Lambda$1207/2085911299 from com.facebook.presto.split.BufferingSplitSource$GetNextBatch]
[Loaded com.facebook.presto.split.BufferingSplitSource$GetNextBatch$$Lambda$1208/1160313617 from com.facebook.presto.split.BufferingSplitSource$GetNextBatch]
[Loaded com.facebook.presto.execution.scheduler.SourcePartitionedScheduler$1 from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.execution.scheduler.NodeAssignmentStats from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.google.common.util.concurrent.ExecutionList$RunnableExecutorPair from file:/home/hadoop/presto/presto-server-0.199/lib/guava-24.1-jre.jar]
[Loaded com.facebook.presto.execution.scheduler.NodeAssignmentStats$$Lambda$1209/1690839080 from com.facebook.presto.execution.scheduler.NodeAssignmentStats]
[Loaded com.facebook.presto.execution.scheduler.NodeAssignmentStats$$Lambda$1210/786904097 from com.facebook.presto.execution.scheduler.NodeAssignmentStats]
[Loaded com.facebook.presto.execution.scheduler.SourcePartitionedScheduler$$Lambda$1211/1528905381 from com.facebook.presto.execution.scheduler.SourcePartitionedScheduler]
[Loaded com.google.common.collect.AbstractMapBasedMultimap$Itr from file:/home/hadoop/presto/presto-server-0.199/lib/guava-24.1-jre.jar]
[Loaded com.google.common.collect.AbstractMapBasedMultimap$1 from file:/home/hadoop/presto/presto-server-0.199/lib/guava-24.1-jre.jar]
[Loaded com.google.common.collect.Iterators$EmptyModifiableIterator from file:/home/hadoop/presto/presto-server-0.199/lib/guava-24.1-jre.jar]
[Loaded com.google.common.collect.Maps$KeySet from file:/home/hadoop/presto/presto-server-0.199/lib/guava-24.1-jre.jar]
[Loaded com.google.common.collect.AbstractMapBasedMultimap$KeySet from file:/home/hadoop/presto/presto-server-0.199/lib/guava-24.1-jre.jar]
[Loaded com.google.common.collect.AbstractMapBasedMultimap$KeySet$1 from file:/home/hadoop/presto/presto-server-0.199/lib/guava-24.1-jre.jar]
[Loaded com.facebook.presto.execution.SqlStageExecution$$Lambda$1212/969840117 from com.facebook.presto.execution.SqlStageExecution]
[Loaded com.google.common.collect.AbstractMapBasedMultimap$$Lambda$1213/1724777987 from com.google.common.collect.AbstractMapBasedMultimap]
[Loaded com.facebook.presto.execution.NodeTaskMap$NodeTasks$TaskPartitionedSplitCountTracker from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded java.lang.invoke.LambdaForm$DMH/1072778948 from java.lang.invoke.LambdaForm]
[Loaded com.facebook.presto.execution.NodeTaskMap$NodeTasks$$Lambda$1214/700072778 from com.facebook.presto.execution.NodeTaskMap$NodeTasks]
[Loaded com.facebook.presto.execution.NodeTaskMap$NodeTasks$$Lambda$1215/270633307 from com.facebook.presto.execution.NodeTaskMap$NodeTasks]
[Loaded com.facebook.presto.util.FinalizerService$FinalizerReference from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.server.remotetask.HttpRemoteTask from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.server.remotetask.SimpleHttpResponseCallback from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.server.remotetask.RequestErrorTracker from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.server.remotetask.Backoff from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.server.remotetask.Backoff$$Lambda$1216/1801167690 from com.facebook.presto.server.remotetask.Backoff]
[Loaded java.util.stream.LongPipeline$$Lambda$1217/668988093 from java.util.stream.LongPipeline]
[Loaded java.util.stream.Node$Builder$OfLong from /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.171-7.b10.37.amzn1.x86_64/jre/lib/rt.jar]
[Loaded java.util.stream.Nodes$LongArrayNode from /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.171-7.b10.37.amzn1.x86_64/jre/lib/rt.jar]
[Loaded java.util.stream.Nodes$LongFixedNodeBuilder from /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.171-7.b10.37.amzn1.x86_64/jre/lib/rt.jar]
[Loaded com.facebook.presto.ScheduledSplit from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.server.remotetask.HttpRemoteTask$$Lambda$1218/729717019 from com.facebook.presto.server.remotetask.HttpRemoteTask]
[Loaded com.facebook.presto.server.remotetask.HttpRemoteTask$$Lambda$1219/540041709 from com.facebook.presto.server.remotetask.HttpRemoteTask]
[Loaded com.facebook.presto.execution.buffer.BufferInfo from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.server.remotetask.HttpRemoteTask$$Lambda$1220/282660077 from com.facebook.presto.server.remotetask.HttpRemoteTask]
[Loaded com.facebook.presto.execution.buffer.PageBufferInfo from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.operator.TaskStats from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.execution.TaskState$$Lambda$1221/2041745537 from com.facebook.presto.execution.TaskState]
[Loaded com.facebook.presto.execution.buffer.OutputBufferInfo from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.execution.buffer.BufferState from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.execution.buffer.BufferState$$Lambda$1222/538637889 from com.facebook.presto.execution.buffer.BufferState]
[Loaded com.facebook.presto.server.remotetask.ContinuousTaskStatusFetcher from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.server.remotetask.HttpRemoteTask$$Lambda$1223/2144378056 from com.facebook.presto.server.remotetask.HttpRemoteTask]
[Loaded com.facebook.presto.server.remotetask.TaskInfoFetcher from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.server.remotetask.HttpRemoteTask$$Lambda$1224/2034474050 from com.facebook.presto.server.remotetask.HttpRemoteTask]
[Loaded com.facebook.presto.server.remotetask.HttpRemoteTask$$Lambda$1225/2031390260 from com.facebook.presto.server.remotetask.HttpRemoteTask]
[Loaded com.facebook.presto.execution.MemoryTrackingRemoteTaskFactory$UpdatePeakMemory from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.execution.SqlStageExecution$$Lambda$1226/286159297 from com.facebook.presto.execution.SqlStageExecution]
[Loaded com.facebook.presto.execution.SqlStageExecution$$Lambda$1227/1278621946 from com.facebook.presto.execution.SqlStageExecution]
[Loaded com.facebook.presto.execution.NodeTaskMap$NodeTasks$$Lambda$1228/1790956576 from com.facebook.presto.execution.NodeTaskMap$NodeTasks]
[Loaded java.util.concurrent.ConcurrentHashMap$KeySpliterator from /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.171-7.b10.37.amzn1.x86_64/jre/lib/rt.jar]
[Loaded com.facebook.presto.execution.SqlStageExecution$StageTaskListener from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.server.remotetask.HttpRemoteTask$$Lambda$1229/316225093 from com.facebook.presto.server.remotetask.HttpRemoteTask]
[Loaded com.facebook.presto.server.remotetask.SimpleHttpResponseHandler from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.server.remotetask.SimpleHttpResponseHandler$ServiceUnavailableException from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.server.remotetask.HttpRemoteTask$$Lambda$1230/1462367267 from com.facebook.presto.server.remotetask.HttpRemoteTask]
[Loaded com.facebook.presto.server.remotetask.TaskInfoFetcher$$Lambda$1231/1976251481 from com.facebook.presto.server.remotetask.TaskInfoFetcher]
[Loaded com.facebook.presto.server.remotetask.HttpRemoteTask$$Lambda$1232/1508774070 from com.facebook.presto.server.remotetask.HttpRemoteTask]
[Loaded com.facebook.presto.execution.scheduler.ScheduleResult$BlockedReason from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.memory.context.MemoryReservationHandler from file:/home/hadoop/presto/presto-server-0.199/lib/presto-memory-context-0.199.jar]
[Loaded com.facebook.presto.TaskSource from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.execution.scheduler.ScheduleResult from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.server.remotetask.HttpRemoteTask$$Lambda$1233/696362582 from com.facebook.presto.server.remotetask.HttpRemoteTask]
[Loaded com.facebook.presto.memory.context.MemoryTrackingContext from file:/home/hadoop/presto/presto-server-0.199/lib/presto-memory-context-0.199.jar]
[Loaded io.airlift.concurrent.ExtendedSettableFuture from file:/home/hadoop/presto/presto-server-0.199/lib/concurrent-0.167.jar]
[Loaded com.facebook.presto.server.remotetask.HttpRemoteTask$$Lambda$1234/2021367602 from com.facebook.presto.server.remotetask.HttpRemoteTask]
[Loaded com.facebook.presto.memory.QueryContext$QueryMemoryReservationHandler from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded io.airlift.concurrent.ExtendedSettableFuture$$Lambda$1235/1955872524 from io.airlift.concurrent.ExtendedSettableFuture]
[Loaded java.util.function.LongFunction from /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.171-7.b10.37.amzn1.x86_64/jre/lib/rt.jar]
[Loaded com.facebook.presto.memory.QueryContext$$Lambda$1236/677606056 from com.facebook.presto.memory.QueryContext]
[Loaded io.airlift.concurrent.ExtendedSettableFuture$$Lambda$1237/1575781025 from io.airlift.concurrent.ExtendedSettableFuture]
[Loaded java.lang.invoke.LambdaForm$DMH/1583664896 from java.lang.invoke.LambdaForm]
[Loaded java.util.function.LongPredicate from /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.171-7.b10.37.amzn1.x86_64/jre/lib/rt.jar]
[Loaded com.facebook.presto.execution.scheduler.SqlQueryScheduler$1 from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.memory.QueryContext$$Lambda$1238/446072813 from com.facebook.presto.memory.QueryContext]
[Loaded com.google.common.collect.LinkedListMultimap$Node from file:/home/hadoop/presto/presto-server-0.199/lib/guava-24.1-jre.jar]
[Loaded com.google.common.collect.LinkedListMultimap$KeyList from file:/home/hadoop/presto/presto-server-0.199/lib/guava-24.1-jre.jar]
2018-05-28T07:57:33.556Z DEBUG query-execution-3 com.facebook.presto.execution.StageStateMachine Stage 20180528_075336_00001_j36cu.1 is SCHEDULING
[Loaded com.facebook.presto.memory.QueryContext$$Lambda$1239/693325861 from com.facebook.presto.memory.QueryContext]
[Loaded com.facebook.presto.execution.scheduler.FixedCountScheduler$$Lambda$1240/72749615 from com.facebook.presto.execution.scheduler.FixedCountScheduler]
[Loaded com.facebook.presto.memory.QueryContext$$Lambda$1241/1362597568 from com.facebook.presto.memory.QueryContext]
[Loaded com.google.common.collect.AbstractMapBasedMultimap$$Lambda$1242/1449500907 from com.google.common.collect.AbstractMapBasedMultimap]
[Loaded com.facebook.presto.memory.QueryContext$$Lambda$1243/582464003 from com.facebook.presto.memory.QueryContext]
[Loaded com.facebook.presto.memory.QueryContext$$Lambda$1244/204529751 from com.facebook.presto.memory.QueryContext]
[Loaded com.google.common.cache.LocalCache$WeakValueReference from file:/home/hadoop/presto/presto-server-0.199/lib/guava-24.1-jre.jar]
[Loaded com.facebook.presto.execution.SqlTaskManager$$Lambda$1245/1292225742 from com.facebook.presto.execution.SqlTaskManager]
[Loaded com.facebook.presto.execution.SqlTask$TaskHolder from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded java.net.URLDecoder from /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.171-7.b10.37.amzn1.x86_64/jre/lib/rt.jar]
[Loaded java.net.URLEncoder from /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.171-7.b10.37.amzn1.x86_64/jre/lib/rt.jar]
[Loaded com.facebook.presto.execution.buffer.LazyOutputBuffer from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded org.eclipse.jetty.util.Fields$Field from file:/home/hadoop/presto/presto-server-0.199/lib/jetty-util-9.4.8.v20171121.jar]
[Loaded com.facebook.presto.execution.SqlTask$$Lambda$1246/1558648737 from com.facebook.presto.execution.SqlTask]
[Loaded com.fasterxml.jackson.databind.RuntimeJsonMappingException from file:/home/hadoop/presto/presto-server-0.199/lib/jackson-databind-2.8.1.jar]
[Loaded com.facebook.presto.execution.TaskStateMachine from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.execution.TaskStateMachine$1 from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.execution.SqlTask$1 from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.execution.SqlTask$$Lambda$1247/1931713430 from com.facebook.presto.execution.SqlTask]
[Loaded sun.util.locale.LocaleExtensions from /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.171-7.b10.37.amzn1.x86_64/jre/lib/rt.jar]
[Loaded java.util.Locale$FilteringMode from /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.171-7.b10.37.amzn1.x86_64/jre/lib/rt.jar]
[Loaded com.facebook.presto.server.TaskResource$$Lambda$1248/2012109027 from com.facebook.presto.server.TaskResource]
[Loaded com.facebook.presto.execution.StageStateMachine$$Lambda$1249/89703972 from com.facebook.presto.execution.StageStateMachine]
[Loaded com.facebook.presto.execution.SqlStageExecution$$Lambda$1250/1694458540 from com.facebook.presto.execution.SqlStageExecution]
2018-05-28T07:57:33.577Z DEBUG query-execution-3 com.facebook.presto.execution.StageStateMachine Stage 20180528_075336_00001_j36cu.1 is SCHEDULED
[Loaded com.facebook.presto.execution.QueryStateMachine$$Lambda$1251/286925901 from com.facebook.presto.execution.QueryStateMachine]
[Loaded com.facebook.presto.execution.scheduler.SqlQueryScheduler$StageLinkage$$Lambda$1252/274045815 from com.facebook.presto.execution.scheduler.SqlQueryScheduler$StageLinkage]
[Loaded java.util.Locale$LanguageRange from /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.171-7.b10.37.amzn1.x86_64/jre/lib/rt.jar]
[Loaded com.fasterxml.jackson.databind.annotation.JsonSerialize$Typing from file:/home/hadoop/presto/presto-server-0.199/lib/jackson-databind-2.8.1.jar]
[Loaded sun.reflect.GeneratedConstructorAccessor82 from __JVM_DefineClass__]
[Loaded io.airlift.http.server.AsyncResponseHandler$$Lambda$1253/1052831620 from io.airlift.http.server.AsyncResponseHandler]
[Loaded io.airlift.http.server.AsyncResponseHandler$$Lambda$1254/1104300591 from io.airlift.http.server.AsyncResponseHandler]
[Loaded org.glassfish.jersey.server.internal.JerseyRequestTimeoutHandler$1 from file:/home/hadoop/presto/presto-server-0.199/lib/jersey-server-2.26.jar]
2018-05-28T07:57:33.592Z DEBUG query-execution-4 com.facebook.presto.execution.StageStateMachine Stage 20180528_075336_00001_j36cu.0 is SCHEDULING
[Loaded com.fasterxml.jackson.annotation.JsonTypeInfo$Id from file:/home/hadoop/presto/presto-server-0.199/lib/jackson-annotations-2.8.1.jar]
[Loaded com.sun.proxy.$Proxy161 from sun.misc.Launcher$AppClassLoader]
[Loaded com.facebook.presto.execution.scheduler.SqlQueryScheduler$$Lambda$1255/702454717 from com.facebook.presto.execution.scheduler.SqlQueryScheduler]
[Loaded com.facebook.presto.execution.scheduler.SqlQueryScheduler$$Lambda$1256/1816105958 from com.facebook.presto.execution.scheduler.SqlQueryScheduler]
[Loaded com.facebook.presto.execution.QueryStateMachine$QueryOutputManager$$Lambda$1257/193814551 from com.facebook.presto.execution.QueryStateMachine$QueryOutputManager]
[Loaded com.fasterxml.jackson.annotation.JsonSubTypes from file:/home/hadoop/presto/presto-server-0.199/lib/jackson-annotations-2.8.1.jar]
2018-05-28T07:57:33.596Z DEBUG query-execution-4 com.facebook.presto.execution.StageStateMachine Stage 20180528_075336_00001_j36cu.0 is SCHEDULED
[Loaded com.fasterxml.jackson.annotation.JsonSubTypes$Type from file:/home/hadoop/presto/presto-server-0.199/lib/jackson-annotations-2.8.1.jar]
[Loaded com.sun.proxy.$Proxy162 from sun.misc.Launcher$AppClassLoader]
[Loaded sun.reflect.GeneratedConstructorAccessor83 from __JVM_DefineClass__]
[Loaded com.sun.proxy.$Proxy163 from sun.misc.Launcher$AppClassLoader]
[Loaded com.fasterxml.jackson.databind.jsontype.impl.StdTypeResolverBuilder from file:/home/hadoop/presto/presto-server-0.199/lib/jackson-databind-2.8.1.jar]
[Loaded com.facebook.presto.operator.BlockedReason from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.fasterxml.jackson.databind.jsontype.impl.AsWrapperTypeDeserializer from file:/home/hadoop/presto/presto-server-0.199/lib/jackson-databind-2.8.1.jar]
[Loaded com.facebook.presto.operator.PipelineStats from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.operator.HttpPageBufferClient from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.fasterxml.jackson.databind.jsontype.impl.AsExternalTypeDeserializer from file:/home/hadoop/presto/presto-server-0.199/lib/jackson-databind-2.8.1.jar]
[Loaded com.facebook.presto.operator.PageTooLargeException from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.fasterxml.jackson.databind.jsontype.impl.AsWrapperTypeSerializer from file:/home/hadoop/presto/presto-server-0.199/lib/jackson-databind-2.8.1.jar]
[Loaded com.facebook.presto.operator.ExchangeClient$ExchangeClientCallback from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.fasterxml.jackson.databind.jsontype.impl.AsExternalTypeSerializer from file:/home/hadoop/presto/presto-server-0.199/lib/jackson-databind-2.8.1.jar]
[Loaded com.facebook.presto.operator.HttpPageBufferClient$$Lambda$1258/446476022 from com.facebook.presto.operator.HttpPageBufferClient]
[Loaded com.fasterxml.jackson.databind.jsontype.impl.AsExistingPropertyTypeSerializer from file:/home/hadoop/presto/presto-server-0.199/lib/jackson-databind-2.8.1.jar]
[Loaded com.google.common.cache.LocalCache$WriteThroughEntry from file:/home/hadoop/presto/presto-server-0.199/lib/guava-24.1-jre.jar]
[Loaded com.facebook.presto.operator.HttpPageBufferClient$PageResponseHandler from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.operator.PageTransportErrorException from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.fasterxml.jackson.databind.annotation.JsonTypeIdResolver from file:/home/hadoop/presto/presto-server-0.199/lib/jackson-databind-2.8.1.jar]
[Loaded com.fasterxml.jackson.annotation.JsonTypeInfo$None from file:/home/hadoop/presto/presto-server-0.199/lib/jackson-annotations-2.8.1.jar]
[Loaded com.fasterxml.jackson.annotation.JsonTypeName from file:/home/hadoop/presto/presto-server-0.199/lib/jackson-annotations-2.8.1.jar]
[Loaded io.airlift.slice.InputStreamSliceInput from file:/home/hadoop/presto/presto-server-0.199/lib/slice-0.33.jar]
[Loaded com.facebook.presto.operator.HttpPageBufferClient$1 from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.operator.PageTransportTimeoutException from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded com.facebook.presto.execution.scheduler.NodeScheduler$$Lambda$1259/297223983 from com.facebook.presto.execution.scheduler.NodeScheduler]
[Loaded java.lang.invoke.LambdaForm$MH/25642588 from java.lang.invoke.LambdaForm]
[Loaded sun.reflect.GeneratedMethodAccessor140 from __JVM_DefineClass__]
[Loaded com.fasterxml.jackson.databind.jsontype.impl.StdTypeResolverBuilder$1 from file:/home/hadoop/presto/presto-server-0.199/lib/jackson-databind-2.8.1.jar]
[Loaded sun.reflect.GeneratedMethodAccessor142 from __JVM_DefineClass__]
[Loaded sun.reflect.GeneratedMethodAccessor143 from __JVM_DefineClass__]
[Loaded com.facebook.presto.execution.buffer.LazyOutputBuffer$PendingRead from file:/home/hadoop/presto/presto-server-0.199/lib/presto-main-0.199.jar]
[Loaded java.lang.invoke.LambdaForm$DMH/1978014140 from java.lang.invoke.LambdaForm]
[Loaded com.facebook.presto.server.TaskResource$$Lambda$1260/701285138 from com.facebook.presto.server.TaskResource]
[Loaded sun.reflect.GeneratedMethodAccessor141 from __JVM_DefineClass__]
[Loaded java.lang.invoke.LambdaForm$DMH/820728928 from java.lang.invoke.LambdaForm]
[Loaded java.lang.invoke.LambdaForm$MH/1755904985 from java.lang.invoke.LambdaForm]
[Loaded com.facebook.presto.server.TaskResource$$Lambda$1261/767964297 from com.facebook.presto.server.TaskResource]
[Loaded com.fasterxml.jackson.databind.jsontype.impl.TypeNameIdResolver from file:/home/hadoop/presto/presto-server-0.199/lib/jackson-databind-2.8.1.jar]
[Loaded java.lang.invoke.LambdaForm$DMH/1658693232 from java.lang.invoke.LambdaForm]
[Loaded com.facebook.presto.server.TaskResource$$Lambda$1262/142587423 from com.facebook.presto.server.TaskResource]
2018-05-28T07:57:33.616Z DEBUG query-execution-6 com.facebook.presto.execution.QueryStateMachine Query 20180528_075336_00001_j36cu is RUNNING
[Loaded com.facebook.presto.server.remotetask.ContinuousTaskStatusFetcher$$Lambda$1263/2102629662 from com.facebook.presto.server.remotetask.ContinuousTaskStat
Highlighting key debug messages from the above log for the query going from PLANNING to RUNNING. Notice the timestamps span is around 4 minutes.
2018-05-28T07:53:36.962Z DEBUG query-execution-0 com.facebook.presto.execution.QueryStateMachine Query 20180528_075336_00001_j36cu is PLANNING
2018-05-28T07:54:36.482Z DEBUG Query-20180528_075336_00001_j36cu-166 com.facebook.presto.cassandra.CassandraPartitionManager <keyspace.table> #partitions: 20201
2018-05-28T07:55:35.541Z DEBUG Query-20180528_075336_00001_j36cu-166 com.facebook.presto.cassandra.CassandraPartitionManager <keyspace.table> #partitions: 20201
2018-05-28T07:56:34.056Z DEBUG Query-20180528_075336_00001_j36cu-166 com.facebook.presto.cassandra.CassandraPartitionManager <keyspace.table> #partitions: 20201
2018-05-28T07:57:32.654Z DEBUG Query-20180528_075336_00001_j36cu-166 com.facebook.presto.cassandra.CassandraPartitionManager <keyspace.table> #partitions: 20201
2018-05-28T07:57:33.224Z DEBUG Query-20180528_075336_00001_j36cu-166 com.facebook.presto.util.CompilerUtils Defining class: com/facebook/presto/$gen/LongStateSerializer_20180528_075733_1
2018-05-28T07:57:33.272Z DEBUG Query-20180528_075336_00001_j36cu-166 com.facebook.presto.util.CompilerUtils Defining class: com/facebook/presto/$gen/SingleLongState_20180528_075733_2
2018-05-28T07:57:33.277Z DEBUG Query-20180528_075336_00001_j36cu-166 com.facebook.presto.util.CompilerUtils Defining class: com/facebook/presto/$gen/GroupedLongState_20180528_075733_3
2018-05-28T07:57:33.280Z DEBUG Query-20180528_075336_00001_j36cu-166 com.facebook.presto.util.CompilerUtils Defining class: com/facebook/presto/$gen/LongStateFactory_20180528_075733_4
2018-05-28T07:57:33.475Z DEBUG query-execution-2 com.facebook.presto.execution.QueryStateMachine Query 20180528_075336_00001_j36cu is STARTING
2018-05-28T07:57:33.488Z DEBUG query-execution-3 com.facebook.presto.execution.StageStateMachine Stage 20180528_075336_00001_j36cu.2 is SCHEDULING
2018-05-28T07:57:33.556Z DEBUG query-execution-3 com.facebook.presto.execution.StageStateMachine Stage 20180528_075336_00001_j36cu.1 is SCHEDULING
2018-05-28T07:57:33.577Z DEBUG query-execution-3 com.facebook.presto.execution.StageStateMachine Stage 20180528_075336_00001_j36cu.1 is SCHEDULED
2018-05-28T07:57:33.592Z DEBUG query-execution-4 com.facebook.presto.execution.StageStateMachine Stage 20180528_075336_00001_j36cu.0 is SCHEDULING
2018-05-28T07:57:33.596Z DEBUG query-execution-4 com.facebook.presto.execution.StageStateMachine Stage 20180528_075336_00001_j36cu.0 is SCHEDULED
2018-05-28T07:57:33.616Z DEBUG query-execution-6 com.facebook.presto.execution.QueryStateMachine Query 20180528_075336_00001_j36cu is RUNNING
I am not sure if this is a problem on cassandra's side or CassandraPartitionManager itself. Any insights here would be great.
aandis
on 28 May 2018
2018-05-28T07:54:36.482Z DEBUG Query-20180528_075336_00001_j36cu-166 com.facebook.presto.cassandra.CassandraPartitionManager <keyspace.table> #partitions: 20201
2018-05-28T07:55:35.541Z DEBUG Query-20180528_075336_00001_j36cu-166 com.facebook.presto.cassandra.CassandraPartitionManager <keyspace.table> #partitions: 20201
2018-05-28T07:56:34.056Z DEBUG Query-20180528_075336_00001_j36cu-166 com.facebook.presto.cassandra.CassandraPartitionManager <keyspace.table> #partitions: 20201
2018-05-28T07:57:32.654Z DEBUG Query-20180528_075336_00001_j36cu-166 com.facebook.presto.cassandra.CassandraPartitionManager <keyspace.table> #partitions: 20201
It shows that Cassandra was queried for partitions 4 times, each time it took a 1 minute to return partitions.
kokosing
on 28 May 2018
@kokosing unless we spend 1 minute doing something else between the calls, no?
@aandis re https://github.com/prestodb/presto/issues/10700#issuecomment-392419391
It looks like most time is spent in CassandraPartitionManager.getPartitions (CassandraMetadata.getTableLayouts). This methods does two things:
- talks to Cassandra to know actual partitions (
getCassandraPartitions>NativeCassandraSession.getPartitions) - filters the partitions (predicate pushdown) (this is a stream expression directly in
CassandraPartitionManager.getPartitions> ... >CassandraPartitionManager.lambda$getPartitions$0) - other stuff (likely negligible)
During planning, this method is called more than once (in some cases, predicate derived from existing partitions may be used by the optimizer to construct narrower predicate on the partitions).
Can you profile specifically CassandraMetadata.getTableLayouts & CassandraPartitionManager.getPartitions method? What's time division between different things this method does?
findepi
on 28 May 2018
unless we spend 1 minute doing something else between the calls, no?
True.
kokosing
on 28 May 2018
@findepi do you have any references on profiling at the method level?
aandis
on 28 May 2018
Alright, it took some doing but I ended up setting presto up from source and adding debug statements for profiling this.
Problem
Turns out, for IN query over multiple columns, cassandra connector executes select distinct for each combination of values for the columns, which takes a long time to finish because the number of such queries is huge. Following log statements should make it clear
2018-05-28T13:32:26.929Z DEBUG query-execution-5 com.facebook.presto.execution.QueryStateMachine Query 20180528_133226_00003_6yerz is PLANNING
2018-05-28T13:32:27.316Z DEBUG Query-20180528_133226_00003_6yerz-146 com.facebook.presto.cassandra.CassandraMetadata Inside getTableLayouts
2018-05-28T13:32:27.316Z DEBUG Query-20180528_133226_00003_6yerz-146 com.facebook.presto.cassandra.CassandraPartitionManager Inside getPartitions
2018-05-28T13:32:27.316Z DEBUG Query-20180528_133226_00003_6yerz-146 com.facebook.presto.cassandra.CassandraPartitionManager Fetching cassandra partitions.
2018-05-28T13:32:27.317Z DEBUG Query-20180528_133226_00003_6yerz-146 com.facebook.presto.cassandra.CassandraPartitionManager Inside getCassandraPartitions
2018-05-28T13:32:27.317Z DEBUG Query-20180528_133226_00003_6yerz-146 com.facebook.presto.cassandra.CassandraPartitionManager Getting partition keyset
2018-05-28T13:32:27.325Z DEBUG Query-20180528_133226_00003_6yerz-146 com.facebook.presto.cassandra.CassandraPartitionManager Recieved partition keyset with size 44408
2018-05-28T13:32:27.325Z DEBUG Query-20180528_133226_00003_6yerz-146 com.facebook.presto.cassandra.CassandraPartitionManager Partition keyset not empty. Looping over set
2018-05-28T13:32:27.325Z DEBUG Query-20180528_133226_00003_6yerz-146 com.facebook.presto.cassandra.NativeCassandraSession Inside Native#getPartitions
2018-05-28T13:32:27.325Z DEBUG Query-20180528_133226_00003_6yerz-146 com.facebook.presto.cassandra.NativeCassandraSession Native#queryPartitionKeys started
2018-05-28T13:32:27.336Z DEBUG Query-20180528_133226_00003_6yerz-146 com.facebook.presto.cassandra.NativeCassandraSession Executing SELECT DISTINCT col2,col3,col4 FROM keyspace.table WHERE col2=17491 AND col3=12 AND col4='val1';
2018-05-28T13:32:27.356Z DEBUG Query-20180528_133226_00003_6yerz-146 com.facebook.presto.cassandra.NativeCassandraSession queryPartitionKeys finished!
2018-05-28T13:32:27.356Z DEBUG Query-20180528_133226_00003_6yerz-146 com.facebook.presto.cassandra.NativeCassandraSession Native#getPartitions finished
2018-05-28T13:32:27.356Z DEBUG Query-20180528_133226_00003_6yerz-146 com.facebook.presto.cassandra.NativeCassandraSession Inside Native#getPartitions
2018-05-28T13:32:27.356Z DEBUG Query-20180528_133226_00003_6yerz-146 com.facebook.presto.cassandra.NativeCassandraSession Native#queryPartitionKeys started
2018-05-28T13:32:27.356Z DEBUG Query-20180528_133226_00003_6yerz-146 com.facebook.presto.cassandra.NativeCassandraSession Executing SELECT DISTINCT col2,col3,col4 FROM keyspace.table WHERE col2=17491 AND col3=12 AND col4='val2';
2018-05-28T13:32:27.359Z DEBUG Query-20180528_133226_00003_6yerz-146 com.facebook.presto.cassandra.NativeCassandraSession queryPartitionKeys finished!
2018-05-28T13:32:27.359Z DEBUG Query-20180528_133226_00003_6yerz-146 com.facebook.presto.cassandra.NativeCassandraSession Native#getPartitions finished
2018-05-28T13:32:27.360Z DEBUG Query-20180528_133226_00003_6yerz-146 com.facebook.presto.cassandra.NativeCassandraSession Inside Native#getPartitions
2018-05-28T13:32:27.360Z DEBUG Query-20180528_133226_00003_6yerz-146 com.facebook.presto.cassandra.NativeCassandraSession Native#queryPartitionKeys started
2018-05-28T13:32:27.360Z DEBUG Query-20180528_133226_00003_6yerz-146 com.facebook.presto.cassandra.NativeCassandraSession Executing SELECT DISTINCT col2,col3,col4 FROM keyspace.table WHERE col2=17491 AND col3=13 AND col4='val1';
2018-05-28T13:32:27.363Z DEBUG Query-20180528_133226_00003_6yerz-146 com.facebook.presto.cassandra.NativeCassandraSession queryPartitionKeys finished!
2018-05-28T13:32:27.364Z DEBUG Query-20180528_133226_00003_6yerz-146 com.facebook.presto.cassandra.NativeCassandraSession Native#getPartitions finished
2018-05-28T13:32:27.364Z DEBUG Query-20180528_133226_00003_6yerz-146 com.facebook.presto.cassandra.NativeCassandraSession Inside Native#getPartitions
2018-05-28T13:32:27.364Z DEBUG Query-20180528_133226_00003_6yerz-146 com.facebook.presto.cassandra.NativeCassandraSession Native#queryPartitionKeys started
2018-05-28T13:32:27.365Z DEBUG Query-20180528_133226_00003_6yerz-146 com.facebook.presto.cassandra.NativeCassandraSession Executing SELECT DISTINCT col2,col3,col4 FROM keyspace.table WHERE col2=17491 AND col3=13 AND col4='val2';
2018-05-28T13:32:27.367Z DEBUG Query-20180528_133226_00003_6yerz-146 com.facebook.presto.cassandra.NativeCassandraSession queryPartitionKeys finished!
2018-05-28T13:32:27.367Z DEBUG Query-20180528_133226_00003_6yerz-146 com.facebook.presto.cassandra.NativeCassandraSession Native#getPartitions finished
2018-05-28T13:32:27.367Z DEBUG Query-20180528_133226_00003_6yerz-146 com.facebook.presto.cassandra.NativeCassandraSession Inside Native#getPartitions
2018-05-28T13:32:27.368Z DEBUG Query-20180528_133226_00003_6yerz-146 com.facebook.presto.cassandra.NativeCassandraSession Native#queryPartitionKeys started
2018-05-28T13:32:27.368Z DEBUG Query-20180528_133226_00003_6yerz-146 com.facebook.presto.cassandra.NativeCassandraSession Executing SELECT DISTINCT col2,col3,col4 FROM keyspace.table WHERE col2=17491 AND col3=14 AND col4='val1';
2018-05-28T13:32:27.372Z DEBUG Query-20180528_133226_00003_6yerz-146 com.facebook.presto.cassandra.NativeCassandraSession queryPartitionKeys finished!
2018-05-28T13:32:27.372Z DEBUG Query-20180528_133226_00003_6yerz-146 com.facebook.presto.cassandra.NativeCassandraSession Native#getPartitions finished
2018-05-28T13:32:27.372Z DEBUG Query-20180528_133226_00003_6yerz-146 com.facebook.presto.cassandra.NativeCassandraSession Inside Native#getPartitions
2018-05-28T13:32:27.373Z DEBUG Query-20180528_133226_00003_6yerz-146 com.facebook.presto.cassandra.NativeCassandraSession Native#queryPartitionKeys started
..
..
<Executes for all possible combination of <col2, col3, col4>
This goes on for close to a minute.
Possible fixes
- Skip
select distinctqueries - I am not sure I understand the reason forSELECT DISTINCT col2,col3,col4 FROM keyspace.table WHERE col2=? AND col3=? AND col4=?queries when the columns being queried form the complete partition key set.<col2, col3, col4>in this case form the complete partition keyset, soSELECT DISTINCTwill always return atmost 1 record. - Execute them concurrently - In case, we _have_ to fire the above queries, we could fire them concurrently using executeAsync. Something similar to execute_concurrent_with_args
I am open to sending a patch to fix this based on what's decided here.
aandis
on 28 May 2018
👍1
\cc: @findepi @kokosing
aandis
on 28 May 2018
Digging more into it, looks like those select queries are fired for creating instances of CassandraPartition which would store TupleDomain.
aandis
on 28 May 2018
@aandis the reason for these SELECT DISTINCT queries might be that your query could contain a predicate like col2 BETWEEN 10 AND 100.
If, for example, there exist only values 20, 50 in this range, it might be useful to know that during planning. E.g. if col2 is later used as a join condition with a partitioned Hive table, we could skip reading some partitions.
This, of course, is just an example trying to explain why those queries are not "obviously redundant". In your case there wasn't any Join, but planner code planning the table scan from Cassandra didn't know that.
Async / parallel might do a trick, but may also spike resource usage if multiple queries are being planned concurrently and planner threads trigger multiple concurrent (async) queries towards Cassandra.
@electrum do you have an opinion?
cc @arhimondr
findepi
on 28 May 2018
oh, btw, @aandis thank you for your in-depth analysis!
findepi
on 28 May 2018
❤1
👍1
@findepi that makes sense. A minor issue I noticed earlier - I tried changing the IN query for one of the int columns to a BETWEEN query like col2 BETWEEN <start_int> AND <end_int> and although the query started instantaneously, presto ended up doing a full table scan. It'd be neat to treat BETWEEN clauses for int columns like IN clauses. I'll file a separate issue for this.
Let me know if concurrent queries sound like a good idea and I'll start working on it.
aandis
on 28 May 2018
Let me know if concurrent queries sound like a good idea and I'll start working on it.
I am do not think running concurrent queries is a good idea, let's keep this option as a last resort.
If you have a tuple domain with multiple columns with values set, then the query which is send to Cassandra should use IN predicate as your original Presto query.
SELECT DISTINCT col2,col3,col4 FROM keyspace.table WHERE col2 IN (17491) AND col3 IN (12, 13, 14) AND col4 IN ('val1', 'val2');
👍1
@kokosing I just tried the query in cassandra. Works great. I'll test it out.
aandis
on 28 May 2018
I tried this today and the planning time was reduced to around 80 seconds using IN queries. The individual IN query takes around 20 seconds to complete and given that it's called 4 times by the planner, planning takes ~80 seconds in total in my case. Much improvement from the original ~5 minutes (~75% faster), but I would still consider it a bit high.
Majority of the time is spent processing the result rows. I'll try to see if we can parallelize it using parallel stream api.
https://github.com/prestodb/presto/blob/a981087e622c878ef4f0a4bae7ba274bd7bbb0e6/presto-cassandra/src/main/java/com/facebook/presto/cassandra/NativeCassandraSession.java#L353
aandis
on 29 May 2018
Please do one thing a time, can you post a PR where you used IN predicate first.
How much rows are there?
kokosing
on 29 May 2018
@kokosing sure. I'll open a pr.
There were 20k rows for the above query for which getPartitions took 20 seconds. I tested another query where there were ~160k partitions, and it took close to 5 minutes. See log
# Query 1
2018-05-29T07:12:16.989Z DEBUG Query-20180529_064836_00005_6c79a-346 com.facebook.presto.cassandra.NativeCassandraSession Executing SELECT DISTINCT col1,col2,col3,col4 FROM keyspace.table1 WHERE col1 IN (<int_list>) AND col2 IN (<string_list>) AND col3 IN (<string_list>) AND col4 IN (<string_list>);
2018-05-29T07:12:36.568Z DEBUG Query-20180529_064836_00005_6c79a-346 com.facebook.presto.cassandra.CassandraPartitionManager keyspace.table #partitions: 20201
# Query 2
2018-05-29T07:05:22.647Z DEBUG Query-20180529_064836_00005_6c79a-346 com.facebook.presto.cassandra.NativeCassandraSession Executing SELECT DISTINCT col1,col2,col3,col4 FROM keyspace.table2 WHERE col1 IN (<int_list>) AND col2 IN (<str_list>) AND col3 IN (<str_list>) AND col4 IN (<int_list>);
2018-05-29T07:11:48.308Z DEBUG Query-20180529_064836_00005_6c79a-346 com.facebook.presto.cassandra.CassandraPartitionManager keyspace.table #partitions: 159137
@kokosing #10720
aandis
on 29 May 2018
https://github.com/prestodb/presto/issues/10700#issuecomment-392672042 didn't help. Processing isn't slow, fetching the rows is. I tried increasing the fetch size and fetching results in the background asynchronously but didn't really help. I'll fix tests on #10720 so it's not blocked for merge.
aandis
on 29 May 2018
@aandis @kokosing
Hi,this optimize merge in to the latest version?
I use presto on cassandra and encountered the same problem,.
My partition key is time(2018-11-20 00:01). When I query all day data must be like this: time in (2018-11-20 00:01,2018-11-20 00:02,2018-11-20 00:03.........).
And plan query will be cost much time,however data processing so fast.
imperio-wxm
on 20 Nov 2018
I think you need to change the partition key to DAY. Your partitions are too fine grained. How much data each your partition contain in MBs, how partitions do you have, how much time Presto needs to process each partition?
kokosing
on 20 Nov 2018
@kokosing
Each partition has less data, may not exceed 1MB, each table has 1440 partitions（one minute per partition）, cassandra data saved for 7 days.
I don no know how much time process each partition, but process all data not cost much time, when job started plan state cost much time, and the more partitions the longer it takes.
imperio-wxm
on 20 Nov 2018
Was this page helpful?
0 / 5 - 0 ratings
Related issues
shigechuanqi
·
3Comments
haozhun
·
4Comments
tesseract2048
·
3Comments
tomz
·
3Comments
xerial
·
3Comments
Most helpful comment
oh, btw, @aandis thank you for your in-depth analysis!