Cockroach: sql: mismatched type panic during apply join

Created on 14 Jul 2019  ·  6Comments  ·  Source: cockroachdb/cockroach

Running the following produces a panic:

SELECT (SELECT ARRAY (SELECT c.relname FROM pg_inherits AS i JOIN pg_class AS c ON c.oid = i.inhparent WHERE i.inhrelid = rel.oid ORDER BY inhseqno)) FROM pg_class AS rel;

panic: invalid datum type given: int[], expected name[] [recovered]
        panic: panic while executing 1 statements: SELECT (SELECT ARRAY (SELECT _._ FROM _ AS _ JOIN _ AS _ ON _._ = _._ WHERE _._ = _._ ORDER BY _)) FROM _ AS _; caused by invalid datum type given: int[], expected name[]

goroutine 503 [running]:
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).closeWrapper(0xc00365ed80, 0x7a25e00, 0xc0036d4d00, 0x6c8d000, 0xc006f48cb0)
        /Users/jordan/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:727 +0x330
github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn.func1(0xc00365ed80, 0x7a25e00, 0xc0036d4d00)
        /Users/jordan/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:440 +0x61
panic(0x6c8d000, 0xc006f48cb0)
        /usr/local/Cellar/go/1.12.4/libexec/src/runtime/panic.go:522 +0x1b5
github.com/cockroachdb/cockroach/pkg/sql/sqlbase.DatumToEncDatum(0xc0067fd180, 0x7a7cec0, 0xc006bcbe60, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /Users/jordan/go/src/github.com/cockroachdb/cockroach/pkg/sql/sqlbase/encoded_datum.go:167 +0x1e2
github.com/cockroachdb/cockroach/pkg/sql.(*planNodeToRowSource).Next(0xc006b78f00, 0xc000039b00, 0x7a28280, 0xc003fdaeb0, 0x0)
        /Users/jordan/go/src/github.com/cockroachdb/cockroach/pkg/sql/plan_node_to_row_source.go:179 +0x517
github.com/cockroachdb/cockroach/pkg/sql/distsqlrun.Run(0x7a25ec0, 0xc006fe1080, 0x7a31600, 0xc006b78f00, 0x7a04380, 0xc00098a700)
        /Users/jordan/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsqlrun/base.go:171 +0x35
github.com/cockroachdb/cockroach/pkg/sql/distsqlrun.(*ProcessorBase).Run(0xc006b78f00, 0x7a25ec0, 0xc006fe1080)
        /Users/jordan/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsqlrun/processors.go:793 +0x92
github.com/cockroachdb/cockroach/pkg/sql/distsqlrun.(*Flow).Run(0xc00396fa40, 0x7a25ec0, 0xc006fe1080, 0x73320a8, 0x0, 0x0)
        /Users/jordan/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsqlrun/flow.go:654 +0x1f2
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).Run(0xc000a62a20, 0xc0082e5aa0, 0xc003a3e6c0, 0xc003f49330, 0xc003808000, 0xc003991800, 0x0)
        /Users/jordan/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:295 +0x35c
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRun(0xc000a62a20, 0x7a25ec0, 0xc006fe0930, 0xc003991800, 0xc0082e5aa0, 0xc003a3e6c0, 0x7a28280, 0xc003fdaeb0, 0xc003808000)
        /Users/jordan/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:859 +0x208
github.com/cockroachdb/cockroach/pkg/sql.(*applyJoinNode).runRightSidePlan(0xc0037f1e00, 0x7a25ec0, 0xc006fe0930, 0xc0032a5400, 0xc000143600, 0xc003791a40, 0x0, 0x0)
        /Users/jordan/go/src/github.com/cockroachdb/cockroach/pkg/sql/apply_join.go:328 +0x5de
github.com/cockroachdb/cockroach/pkg/sql.(*applyJoinNode).Next(0xc0037f1e00, 0x7a25ec0, 0xc006fe0930, 0xc0032a5400, 0xc000143600, 0x0, 0x7a25ec0, 0xc006fe0930)
        /Users/jordan/go/src/github.com/cockroachdb/cockroach/pkg/sql/apply_join.go:273 +0x6bf
github.com/cockroachdb/cockroach/pkg/sql.(*planNodeToRowSource).Next(0xc006b78500, 0xc000143600, 0x7a26f40, 0xc0037f1e00, 0x0)
        /Users/jordan/go/src/github.com/cockroachdb/cockroach/pkg/sql/plan_node_to_row_source.go:171 +0x43c
github.com/cockroachdb/cockroach/pkg/sql/distsqlrun.Run(0x7a25ec0, 0xc006fe0930, 0x7a31600, 0xc006b78500, 0x7a04380, 0xc0035d3c00)
        /Users/jordan/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsqlrun/base.go:171 +0x35
github.com/cockroachdb/cockroach/pkg/sql/distsqlrun.(*ProcessorBase).Run(0xc006b78500, 0x7a25ec0, 0xc006fe0930)
        /Users/jordan/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsqlrun/processors.go:793 +0x92
github.com/cockroachdb/cockroach/pkg/sql/distsqlrun.(*Flow).Run(0xc00396f4a0, 0x7a25ec0, 0xc006fe0930, 0x73320a8, 0x0, 0x0)
        /Users/jordan/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsqlrun/flow.go:654 +0x1f2
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).Run(0xc000a62a20, 0xc0082e51a0, 0xc003a3e6c0, 0xc003f49dd0, 0xc00038b600, 0xc0032a4e00, 0x0)
        /Users/jordan/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:295 +0x35c
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRun(0xc000a62a20, 0x7a25ec0, 0xc006fe3020, 0xc0032a4e00, 0xc0082e51a0, 0xc003a3e6c0, 0x7a26f40, 0xc0037f1e00, 0xc00038b600)
        /Users/jordan/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:859 +0x208
github.com/cockroachdb/cockroach/pkg/sql.(*applyJoinNode).runRightSidePlan(0xc0037f1200, 0x7a25ec0, 0xc006fe3020, 0xc0037f1400, 0xc00365f0d0, 0xc003791800, 0x0, 0x0)
        /Users/jordan/go/src/github.com/cockroachdb/cockroach/pkg/sql/apply_join.go:328 +0x5de
github.com/cockroachdb/cockroach/pkg/sql.(*applyJoinNode).Next(0xc0037f1200, 0x7a25ec0, 0xc006fe3020, 0xc0037f1400, 0xc00365f0d0, 0x0, 0x7a25ec0, 0xc006fe3020)
        /Users/jordan/go/src/github.com/cockroachdb/cockroach/pkg/sql/apply_join.go:273 +0x6bf
github.com/cockroachdb/cockroach/pkg/sql.(*planNodeToRowSource).Next(0xc000657900, 0xc00365f0d0, 0x7a26f40, 0xc0037f1200, 0x0)
        /Users/jordan/go/src/github.com/cockroachdb/cockroach/pkg/sql/plan_node_to_row_source.go:171 +0x43c
github.com/cockroachdb/cockroach/pkg/sql/distsqlrun.Run(0x7a25ec0, 0xc006fe3020, 0x7a31600, 0xc000657900, 0x7a04380, 0xc000b7ce00)
        /Users/jordan/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsqlrun/base.go:171 +0x35
github.com/cockroachdb/cockroach/pkg/sql/distsqlrun.(*ProcessorBase).Run(0xc000657900, 0x7a25ec0, 0xc006fe3020)
        /Users/jordan/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsqlrun/processors.go:793 +0x92
github.com/cockroachdb/cockroach/pkg/sql/distsqlrun.(*Flow).Run(0xc00396ed20, 0x7a25ec0, 0xc006fe3020, 0x73320a8, 0x0, 0x0)
        /Users/jordan/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsqlrun/flow.go:654 +0x1f2
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).Run(0xc000a62a20, 0xc0082e4ae0, 0xc003a3e6c0, 0xc003f4a870, 0xc00038b340, 0xc00365f1a8, 0x0)
        /Users/jordan/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:295 +0x35c
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRun(0xc000a62a20, 0x7a25ec0, 0xc006bcbb00, 0xc00365f1a8, 0xc0082e4ae0, 0xc003a3e6c0, 0x7a27bc0, 0xc0035c07e0, 0xc00038b340)
        /Users/jordan/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:859 +0x208

The EXPLAIN(opt,types) output is:

+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
  project
   ├── columns: array:61(name[])
   ├── stats: [rows=1000]
   ├── cost: 346.042712
   ├── prune: (61)
   ├── inner-join-apply
   │    ├── columns: oid:1(oid) relname:2(name) relnamespace:3(oid) reltype:4(oid) relowner:5(oid) relam:6(oid) relfilenode:7(oid) reltablespace:8(oid) relpages:9(int) reltuples:10(float) relallvisible:11(int) reltoastrelid:12(oid) relhasindex:13(bool) relisshared:14(bool) relpersistence:15(char) relistemp:16(bool) relkind:17(char) relnatts:18(int) relchecks:19(int) relhasoids:20(bool) relhaspkey:21(bool) relhasrules:22(bool) relhastriggers:23(bool) relhassubclass:24(bool) relfrozenxid:25(int) relacl:26(string[]) reloptions:27(string[]) array_agg:58(int[]) array:59(name[])
   │    ├── stats: [rows=1000]
   │    ├── cost: 326.032712
   │    ├── virtual-scan defaultdb.pg_catalog.pg_class
   │    │    ├── columns: oid:1(oid) relname:2(name) relnamespace:3(oid) reltype:4(oid) relowner:5(oid) relam:6(oid) relfilenode:7(oid) reltablespace:8(oid) relpages:9(int) reltuples:10(float) relallvisible:11(int) reltoastrelid:12(oid) relhasindex:13(bool) relisshared:14(bool) relpersistence:15(char) relistemp:16(bool) relkind:17(char) relnatts:18(int) relchecks:19(int) relhasoids:20(bool) relhaspkey:21(bool) relhasrules:22(bool) relhastriggers:23(bool) relhassubclass:24(bool) relfrozenxid:25(int) relacl:26(string[]) reloptions:27(string[])
   │    │    ├── stats: [rows=1000]
   │    │    └── cost: 10.01
   │    ├── inner-join-apply
   │    │    ├── columns: array_agg:58(int[]) array:59(name[])
   │    │    ├── outer: (1)
   │    │    ├── cardinality: [1 - 1]
   │    │    ├── stats: [rows=1]
   │    │    ├── cost: 293.495212
   │    │    ├── key: ()
   │    │    ├── fd: ()-->(58,59)
   │    │    ├── prune: (59)
   │    │    ├── project
   │    │    │    ├── columns: array_agg:58(int[])
   │    │    │    ├── outer: (1)
   │    │    │    ├── cardinality: [1 - 1]
   │    │    │    ├── stats: [rows=1]
   │    │    │    ├── cost: 293.425212
   │    │    │    ├── key: ()
   │    │    │    ├── fd: ()-->(58)
   │    │    │    ├── prune: (58)
   │    │    │    ├── group-by
   │    │    │    │    ├── columns: inhrelid:28(oid) array_agg:60(int[])
   │    │    │    │    ├── internal-ordering: +30 opt(28)
   │    │    │    │    ├── outer: (1)
   │    │    │    │    ├── cardinality: [1 - 1]
   │    │    │    │    ├── stats: [rows=1]
   │    │    │    │    ├── cost: 293.395212
   │    │    │    │    ├── key: ()
   │    │    │    │    ├── fd: ()-->(28,60)
   │    │    │    │    ├── sort
   │    │    │    │    │    ├── columns: inhrelid:28(oid) inhparent:29(oid) inhseqno:30(int) oid:31(oid) relname:32(name) relnamespace:33(oid) reltype:34(oid) relowner:35(oid) relam:36(oid) relfilenode:37(oid) reltablespace:38(oid) relpages:39(int) reltuples:40(float) relallvisible:41(int) reltoastrelid:42(oid) relhasindex:43(bool) relisshared:44(bool) relpersistence:45(char) relistemp:46(bool) relkind:47(char) relnatts:48(int) relchecks:49(int) relhasoids:50(bool) relhaspkey:51(bool) relhasrules:52(bool) relhastriggers:53(bool) relhassubclass:54(bool) relfrozenxid:55(int) relacl:56(string[]) reloptions:57(string[])
   │    │    │    │    │    ├── outer: (1)
   │    │    │    │    │    ├── cardinality: [1 - ]
   │    │    │    │    │    ├── stats: [rows=100, distinct(1)=1, null(1)=0, distinct(28)=1, null(28)=0]
   │    │    │    │    │    ├── cost: 291.375212
   │    │    │    │    │    ├── fd: (29)==(31), (31)==(29)
   │    │    │    │    │    ├── ordering: +30 opt(28) [actual: +30]
   │    │    │    │    │    └── right-join (hash)
   │    │    │    │    │         ├── columns: inhrelid:28(oid) inhparent:29(oid) inhseqno:30(int) oid:31(oid) relname:32(name) relnamespace:33(oid) reltype:34(oid) relowner:35(oid) relam:36(oid) relfilenode:37(oid) reltablespace:38(oid) relpages:39(int) reltuples:40(float) relallvisible:41(int) reltoastrelid:42(oid) relhasindex:43(bool) relisshared:44(bool) relpersistence:45(char) relistemp:46(bool) relkind:47(char) relnatts:48(int) relchecks:49(int) relhasoids:50(bool) relhaspkey:51(bool) relhasrules:52(bool) relhastriggers:53(bool) relhassubclass:54(bool) relfrozenxid:55(int) relacl:56(string[]) reloptions:57(string[])
   │    │    │    │    │         ├── outer: (1)
   │    │    │    │    │         ├── cardinality: [1 - ]
   │    │    │    │    │         ├── stats: [rows=100, distinct(1)=1, null(1)=0, distinct(28)=1, null(28)=0]
   │    │    │    │    │         ├── cost: 276.0775
   │    │    │    │    │         ├── fd: (29)==(31), (31)==(29)
   │    │    │    │    │         ├── inner-join (hash)
   │    │    │    │    │         │    ├── columns: inhrelid:28(oid) inhparent:29(oid!null) inhseqno:30(int) oid:31(oid!null) relname:32(name) relnamespace:33(oid) reltype:34(oid) relowner:35(oid) relam:36(oid) relfilenode:37(oid) reltablespace:38(oid) relpages:39(int) reltuples:40(float) relallvisible:41(int) reltoastrelid:42(oid) relhasindex:43(bool) relisshared:44(bool) relpersistence:45(char) relistemp:46(bool) relkind:47(char) relnatts:48(int) relchecks:49(int) relhasoids:50(bool) relhaspkey:51(bool) relhasrules:52(bool) relhastriggers:53(bool) relhassubclass:54(bool) relfrozenxid:55(int) relacl:56(string[]) reloptions:57(string[])
   │    │    │    │    │         │    ├── stats: [rows=10000, distinct(28)=100, null(28)=100, distinct(29)=100, null(29)=0, distinct(31)=100, null(31)=0]
   │    │    │    │    │         │    ├── cost: 150.03
   │    │    │    │    │         │    ├── fd: (29)==(31), (31)==(29)
   │    │    │    │    │         │    ├── virtual-scan defaultdb.pg_catalog.pg_inherits
   │    │    │    │    │         │    │    ├── columns: inhrelid:28(oid) inhparent:29(oid) inhseqno:30(int)
   │    │    │    │    │         │    │    ├── stats: [rows=1000, distinct(28)=100, null(28)=10, distinct(29)=100, null(29)=10]
   │    │    │    │    │         │    │    └── cost: 10.01
   │    │    │    │    │         │    ├── virtual-scan defaultdb.pg_catalog.pg_class
   │    │    │    │    │         │    │    ├── columns: oid:31(oid) relname:32(name) relnamespace:33(oid) reltype:34(oid) relowner:35(oid) relam:36(oid) relfilenode:37(oid) reltablespace:38(oid) relpages:39(int) reltuples:40(float) relallvisible:41(int) reltoastrelid:42(oid) relhasindex:43(bool) relisshared:44(bool) relpersistence:45(char) relistemp:46(bool) relkind:47(char) relnatts:48(int) relchecks:49(int) relhasoids:50(bool) relhaspkey:51(bool) relhasrules:52(bool) relhastriggers:53(bool) relhassubclass:54(bool) relfrozenxid:55(int) relacl:56(string[]) reloptions:57(string[])
   │    │    │    │    │         │    │    ├── stats: [rows=1000, distinct(31)=100, null(31)=10]
   │    │    │    │    │         │    │    └── cost: 10.01
   │    │    │    │    │         │    └── filters
   │    │    │    │    │         │         └── eq [type=bool, outer=(29,31), constraints=(/29: (/NULL - ]; /31: (/NULL - ]), fd=(29)==(31), (31)==(29)]
   │    │    │    │    │         │              ├── variable: oid [type=oid]
   │    │    │    │    │         │              └── variable: inhparent [type=oid]
   │    │    │    │    │         ├── values
   │    │    │    │    │         │    ├── cardinality: [1 - 1]
   │    │    │    │    │         │    ├── stats: [rows=1]
   │    │    │    │    │         │    ├── cost: 0.02
   │    │    │    │    │         │    ├── key: ()
   │    │    │    │    │         │    └── tuple [type=tuple]
   │    │    │    │    │         └── filters
   │    │    │    │    │              └── eq [type=bool, outer=(1,28), constraints=(/1: (/NULL - ]; /28: (/NULL - ]), fd=(1)==(28), (28)==(1)]
   │    │    │    │    │                   ├── variable: inhrelid [type=oid]
   │    │    │    │    │                   └── variable: oid [type=oid]
   │    │    │    │    └── aggregations
   │    │    │    │         ├── array-agg [type=int[], outer=(30)]
   │    │    │    │         │    └── variable: inhseqno [type=int]
   │    │    │    │         └── any-not-null-agg [type=oid, outer=(28)]
   │    │    │    │              └── variable: inhrelid [type=oid]
   │    │    │    └── projections
   │    │    │         └── case [type=int[], outer=(28,60)]
   │    │    │              ├── true [type=bool]
   │    │    │              ├── when [type=int[]]
   │    │    │              │    ├── is-not [type=bool]
   │    │    │              │    │    ├── variable: inhrelid [type=oid]
   │    │    │              │    │    └── null [type=unknown]
   │    │    │              │    └── variable: array_agg [type=int[]]
   │    │    │              └── null [type=unknown]
   │    │    ├── values
   │    │    │    ├── columns: array:59(name[])
   │    │    │    ├── outer: (58)
   │    │    │    ├── cardinality: [1 - 1]
   │    │    │    ├── stats: [rows=1]
   │    │    │    ├── cost: 0.02
   │    │    │    ├── key: ()
   │    │    │    ├── fd: ()-->(59)
   │    │    │    ├── prune: (59)
   │    │    │    └── tuple [type=tuple{int[]}]
   │    │    │         └── coalesce [type=int[]]
   │    │    │              ├── variable: array_agg [type=int[]]
   │    │    │              └── const: ARRAY[] [type=int[]]
   │    │    └── filters (true)
   │    └── filters (true)
   └── projections
        └── variable: array [type=name[], outer=(59)]
(113 rows)

Time: 3.032ms

It looks like the outer apply join is maybe returning the columns in the wrong order - I'm not really too sure yet, but just filing this for now.

C-bug O-sqlsmith S-2-temp-unavailability
Source
picture jordanlewis

Most helpful comment

Yep I can take a look!

picture rytaft on 8 Aug 2019
👍2

All 6 comments

@RaduBerinde would you or somebody from your team be able to take a quick look at the EXPLAIN(opt,types) to see if there's something wrong with the produced plan? I'm assuming this is an execution issue but just want to double check.

jordanlewis picture jordanlewis on 14 Jul 2019

I don't see a problem in the plan. Looking at the applyJoinNode code, what ensures that rightProps actually contains a presentation? Without a presentation, in principle it's possible that every time we plan the right side, we get columns in a different order at the root. Somewhere we should be enforcing a specific presentation, even if we have to choose an arbitrary one.

RaduBerinde picture RaduBerinde on 15 Jul 2019

Hey @RaduBerinde could you or someone else on opt team help us deal with this problem? I see what you're getting at with the presentation thing but I don't know how to make that happen in the code.

jordanlewis picture jordanlewis on 8 Aug 2019

I spent some time on this. I was wrong, the execbuilder always sets the presentation. I can't find a problem in the apply join code.
What is wrong here is that this values node is busted (it says it returns a column of name[] but it contains int[]s).

   │    │    ├── values
   │    │    │    ├── columns: array:59(name[])
   │    │    │    ├── outer: (58)
   │    │    │    ├── cardinality: [1 - 1]
   │    │    │    ├── stats: [rows=1]
   │    │    │    ├── cost: 0.02
   │    │    │    ├── key: ()
   │    │    │    ├── fd: ()-->(59)
   │    │    │    ├── prune: (59)
   │    │    │    └── tuple [type=tuple{int[]}]
   │    │    │         └── coalesce [type=int[]]
   │    │    │              ├── variable: array_agg [type=int[]]
   │    │    │              └── const: ARRAY[] [type=int[]]

The problem is when we determine the type of ArrayFlatten; we take the first column ID (assuming there is just one) but it's possible that the input has other columns not in the presentation:

// typeArrayFlatten returns the type of the subquery as an array.
func typeArrayFlatten(e opt.ScalarExpr) *types.T {
    input := e.Child(0).(RelExpr)
    colID, _ := input.Relational().OutputCols.Next(0)  // <--------- we should be using ArrayFlattenExpr.RequestedCol
    return types.MakeArray(input.Memo().Metadata().ColumnMeta(colID).Type)
}

Similar issue I think in subqueryHoister.hoistAll and probably in other places where ArrayFlatten is handled. I made the fixes for what I found but this still crashes. More investigation is needed, I'm not sure I'll be able to do it today; I'll continue when I get back after next week.

CC @rytaft if she wants to take this on.

RaduBerinde picture RaduBerinde on 8 Aug 2019

Some notes for when we fix this:

  • add The rightProps always includes a presentation. to the comment for exec.Factory.ConstructApplyJoin
  • add a helper method opt.ColSet.SingleColumn() and have it return the only column, or panic if the set size is not 1; replace uses of Next(0) for subqueries with that
  • consider setting RequestedCol in all cases (in all subquery privates).
RaduBerinde picture RaduBerinde on 8 Aug 2019

Yep I can take a look!

rytaft picture rytaft on 8 Aug 2019
👍2
Was this page helpful?
0 / 5 - 0 ratings

Related issues

xudongzheng picture xudongzheng  ·  3Comments
melskyzy picture melskyzy  ·  3Comments
HeikoOnnebrink picture HeikoOnnebrink  ·  4Comments
nvanbenschoten picture nvanbenschoten  ·  3Comments
petermattis picture petermattis  ·  4Comments