Metatron-discovery: Add validation for geo(wkt) type

[deidera08]

Is your feature request related to a problem? Please describe.
현재 Geo point 타입은 인제스천 할 때 유저가 생성해야만 사용할 수 있습니다.
그러나 기존에 생성해 놓은 Geo point를 활용하는 사용성이 커졌으며, 일일히 Geo point 타입을 생성하기에 번거로움이 있습니다.

Describe the solution you'd like
Geo point type으로 변경할 수 있어야 합니다.
더불어 WKT 포맷만 지원합니다. 이미 WKT 포맷으로 작성된 데이터일 경우에만 변경할 수 있습니다.

Describe alternatives you've considered

1. Datasource > Create Datasrouce
   : 데이터 소스로 인제스천 할 때 변경 지원.
   
   .
   .
2. Datasource > Column details > Configure schema
   
   .
   .
3. Metadata > Column details
   

[brandon-wonjune]

~이 이슈는 #1540 에서 같이 진행하겠습니다~

[deidera08]

@kyungtaak @AnnieHwang
질문이 있습니다.

1. 현재 '상세 > 컬럼 스키마' 화면에서 Geo Point만 수정하는 것으로 기획했었는데, 인제스천 화면에서는 Geo point 뿐 아니라 Geo line, Geo polygon까지 변경할 수 있습니다. 모두 동일하게 point,line, polygon까지 지원해야 할까요?

2. Geo point 로 변경시 WKT포맷인지 아닌지만 판단한다고 가이드 받았었는데요
   현재 인제스천 화면에서는 Geo coordinates를 설정할 수 있습니다.
   Geo coordinates 설정부분이 '상세 > 컬럼 스키마'화면에도 동일하게 적용되어야 하는지 궁금합니다.
   

[deidera08]

@kyungtaak @AnnieHwang @brandon-wonjune @ninezero90hy

1. 현재 '상세 > 컬럼 스키마' 화면에서 Geo Point만 수정하는 것으로 기획했었는데, 인제스천 화면에서는 Geo point 뿐 아니라 Geo line, Geo polygon까지 변경할 수 있습니다. 모두 동일하게 point,line, polygon까지 지원해야 할까요?
2. Geo point 로 변경시 WKT포맷인지 아닌지만 판단한다고 가이드 받았었는데요
   현재 인제스천 화면에서는 Geo coordinates를 설정할 수 있습니다.
   Geo coordinates 설정부분이 '상세 > 컬럼 스키마'화면에도 동일하게 적용되어야 하는지 궁금합니다.

위 상황에 대하여 인제스천, 컬럼스키마 모두 동일한 UI가 적용되어야 합니다.
이에 따라 변경된 화면 공유합니다.

1. Datasource > Ingestion
   (Geo) Point/Line/Polygon타입으로 수정 즉시 WKT포맷 유효성 검사가 이뤄져야 합니다.
   더불어 유효하지 않을경우 경고를 띄우며, 좌표계 Select box를 비활성화 해주세요.
   

.
.

1. Datasource > Column details > Configure the schema
   
   
   (Geo) Point/Line/Polygon 수정을 지원합니다. 마찬가지로 수정 즉시 WKT포맷 유효성 검사가 이뤄지며 유효하지 않을 경우 저장되지 않습니다.
   information 레이어에서 좌표계 설정을 할 수 있습니다.

.
.

1. Metadata > Column schema
   
   2번과 같은 내용을 지원합니다.

ppt화면 입니다.
#1625_Change GeoPoint Type.pptx
참고부탁드립니다.

[ninezero90hy]

@kyungtaak 메타데이터 컬럼 목록 조회시 포맷 필드가 없습니다. 확인 부탁드립니다

• 메타데이터 컬럼조회 - [GET] /api/metadatas/{id}/columns?projection=forListView

[{
    "name": "id",
    "id": "42",
    "type": "STRING",
    "role": "DIMENSION",
    "popularity": 0.0,
    "physicalType": "STRING",
    "physicalName": "id"
}, {
    "name": "created_by",
    "id": "43",
    "type": "STRING",
    "role": "DIMENSION",
    "popularity": 0.0,
    "physicalType": "STRING",
    "physicalName": "created_by"
}, {
    "name": "created_time",
    "id": "44",
    "type": "TIMESTAMP",
    "role": "TIMESTAMP",
    "popularity": 0.0,
    "physicalType": "TIMESTAMP",
    "physicalName": "created_time"
}, {
    "name": "modified_by",
    "id": "45",
    "type": "STRING",
    "role": "DIMENSION",
    "popularity": 0.0,
    "physicalType": "STRING",
    "physicalName": "modified_by"
}, {
    "name": "modified_time",
    "id": "46",
    "type": "TIMESTAMP",
    "role": "DIMENSION",
    "popularity": 0.0,
    "physicalType": "TIMESTAMP",
    "physicalName": "modified_time"
}, {
    "name": "version",
    "id": "47",
    "type": "INTEGER",
    "role": "MEASURE",
    "popularity": 0.0,
    "physicalType": "LONG",
    "physicalName": "version"
}, {
    "name": "ds_conn_type",
    "id": "48",
    "type": "STRING",
    "role": "DIMENSION",
    "popularity": 0.0,
    "physicalType": "STRING",
    "physicalName": "ds_conn_type"
}, {
    "name": "datasource_contexts",
    "id": "49",
    "type": "STRING",
    "role": "DIMENSION",
    "popularity": 0.0,
    "physicalType": "STRING",
    "physicalName": "datasource_contexts"
}, {
    "name": "ds_desc",
    "id": "50",
    "type": "STRING",
    "role": "DIMENSION",
    "popularity": 0.0,
    "physicalType": "STRING",
    "physicalName": "ds_desc"
}, {
    "name": "ds_type",
    "id": "51",
    "type": "STRING",
    "role": "DIMENSION",
    "popularity": 0.0,
    "physicalType": "STRING",
    "physicalName": "ds_type"
}, {
    "name": "ds_engine_name",
    "id": "52",
    "type": "STRING",
    "role": "DIMENSION",
    "popularity": 0.0,
    "physicalType": "STRING",
    "physicalName": "ds_engine_name"
}, {
    "name": "ds_fail_on_engine",
    "id": "53",
    "type": "STRING",
    "role": "DIMENSION",
    "popularity": 0.0,
    "physicalType": "STRING",
    "physicalName": "ds_fail_on_engine"
}, {
    "name": "ds_fields_matched",
    "id": "54",
    "type": "STRING",
    "role": "DIMENSION",
    "popularity": 0.0,
    "physicalType": "STRING",
    "physicalName": "ds_fields_matched"
}, {
    "name": "ds_granularity",
    "id": "55",
    "type": "STRING",
    "role": "DIMENSION",
    "popularity": 0.0,
    "physicalType": "STRING",
    "physicalName": "ds_granularity"
}, {
    "name": "ds_include_geo",
    "id": "56",
    "type": "STRING",
    "role": "DIMENSION",
    "popularity": 0.0,
    "physicalType": "STRING",
    "physicalName": "ds_include_geo"
}, {
    "name": "ingestion_conf",
    "id": "57",
    "type": "STRING",
    "role": "DIMENSION",
    "popularity": 0.0,
    "physicalType": "STRING",
    "physicalName": "ingestion_conf"
}, {
    "name": "ds_linked_workspaces",
    "id": "58",
    "type": "INTEGER",
    "role": "MEASURE",
    "popularity": 0.0,
    "physicalType": "INTEGER",
    "physicalName": "ds_linked_workspaces"
}, {
    "name": "ds_name",
    "id": "59",
    "type": "STRING",
    "role": "DIMENSION",
    "popularity": 0.0,
    "physicalType": "STRING",
    "physicalName": "ds_name"
}, {
    "name": "ds_owner_id",
    "id": "60",
    "type": "STRING",
    "role": "DIMENSION",
    "popularity": 0.0,
    "physicalType": "STRING",
    "physicalName": "ds_owner_id"
}, {
    "name": "ds_partition_keys",
    "id": "61",
    "type": "STRING",
    "role": "DIMENSION",
    "popularity": 0.0,
    "physicalType": "STRING",
    "physicalName": "ds_partition_keys"
}, {
    "name": "ds_partition_separator",
    "id": "62",
    "type": "STRING",
    "role": "DIMENSION",
    "popularity": 0.0,
    "physicalType": "STRING",
    "physicalName": "ds_partition_separator"
}, {
    "name": "ds_published",
    "id": "63",
    "type": "STRING",
    "role": "DIMENSION",
    "popularity": 0.0,
    "physicalType": "STRING",
    "physicalName": "ds_published"
}, {
    "name": "ds_seg_granularity",
    "id": "64",
    "type": "STRING",
    "role": "DIMENSION",
    "popularity": 0.0,
    "physicalType": "STRING",
    "physicalName": "ds_seg_granularity"
}, {
    "name": "ds_src_type",
    "id": "65",
    "type": "STRING",
    "role": "DIMENSION",
    "popularity": 0.0,
    "physicalType": "STRING",
    "physicalName": "ds_src_type"
}, {
    "name": "ds_status",
    "id": "66",
    "type": "STRING",
    "role": "DIMENSION",
    "popularity": 0.0,
    "physicalType": "STRING",
    "physicalName": "ds_status"
}, {
    "name": "dc_id",
    "id": "67",
    "type": "STRING",
    "role": "DIMENSION",
    "popularity": 0.0,
    "physicalType": "STRING",
    "physicalName": "dc_id"
}, {
    "name": "ss_id",
    "id": "68",
    "type": "STRING",
    "role": "DIMENSION",
    "popularity": 0.0,
    "physicalType": "STRING",
    "physicalName": "ss_id"
}, {
    "name": "smy_id",
    "id": "69",
    "type": "INTEGER",
    "role": "MEASURE",
    "popularity": 0.0,
    "physicalType": "LONG",
    "physicalName": "smy_id"
}]

---

• 해당 메타데이터의 데이터소스 정보

{
    ...
    "name": "datasource",
    "fields": [{
        "id": 10047066,
        "name": "id",
        "logicalName": "id",
        "type": "STRING",
        "logicalType": "STRING",
        "role": "DIMENSION",
        "aggrType": "NONE",
        "seq": 0
    }, {
        "id": 10047067,
        "name": "created_by",
        "logicalName": "created_by",
        "type": "STRING",
        "logicalType": "STRING",
        "role": "DIMENSION",
        "aggrType": "NONE",
        "seq": 1
    }, {
        "id": 10047068,
        "name": "created_time",
        "logicalName": "created_time",
        "type": "TIMESTAMP",
        "logicalType": "TIMESTAMP",
        "role": "TIMESTAMP",
        "aggrType": "NONE",
        "seq": 2,
        "format": {
            "type": "time_format",
            "format": "yyyy-MM-dd HH:mm:ss.SSSSSS",
            "timeZone": "Asia/Seoul",
            "locale": "en"
        }
    }, {
        "id": 10047069,
        "name": "modified_by",
        "logicalName": "modified_by",
        "type": "STRING",
        "logicalType": "STRING",
        "role": "DIMENSION",
        "aggrType": "NONE",
        "seq": 3
    }, {
        "id": 10047070,
        "name": "modified_time",
        "logicalName": "modified_time",
        "type": "TIMESTAMP",
        "logicalType": "TIMESTAMP",
        "role": "DIMENSION",
        "aggrType": "NONE",
        "seq": 4,
        "format": {
            "type": "time_format",
            "format": "yyyy-MM-dd HH:mm:ss.SSSSSS",
            "timeZone": "Asia/Seoul",
            "locale": "en"
        }
    }, {
        "id": 10047071,
        "name": "version",
        "logicalName": "version",
        "type": "LONG",
        "logicalType": "INTEGER",
        "role": "MEASURE",
        "aggrType": "NONE",
        "seq": 5
    }, {
        "id": 10047072,
        "name": "ds_conn_type",
        "logicalName": "ds_conn_type",
        "type": "STRING",
        "logicalType": "STRING",
        "role": "DIMENSION",
        "aggrType": "NONE",
        "seq": 6
    }, {
        "id": 10047073,
        "name": "datasource_contexts",
        "logicalName": "datasource_contexts",
        "type": "STRING",
        "logicalType": "STRING",
        "role": "DIMENSION",
        "aggrType": "NONE",
        "seq": 7
    }, {
        "id": 10047074,
        "name": "ds_desc",
        "logicalName": "ds_desc",
        "type": "STRING",
        "logicalType": "STRING",
        "role": "DIMENSION",
        "aggrType": "NONE",
        "seq": 8
    }, {
        "id": 10047075,
        "name": "ds_type",
        "logicalName": "ds_type",
        "type": "STRING",
        "logicalType": "STRING",
        "role": "DIMENSION",
        "aggrType": "NONE",
        "seq": 9
    }, {
        "id": 10047076,
        "name": "ds_engine_name",
        "logicalName": "ds_engine_name",
        "type": "STRING",
        "logicalType": "STRING",
        "role": "DIMENSION",
        "aggrType": "NONE",
        "seq": 10
    }, {
        "id": 10047077,
        "name": "ds_fail_on_engine",
        "logicalName": "ds_fail_on_engine",
        "type": "STRING",
        "logicalType": "STRING",
        "role": "DIMENSION",
        "aggrType": "NONE",
        "seq": 11
    }, {
        "id": 10047078,
        "name": "ds_fields_matched",
        "logicalName": "ds_fields_matched",
        "type": "STRING",
        "logicalType": "STRING",
        "role": "DIMENSION",
        "aggrType": "NONE",
        "seq": 12
    }, {
        "id": 10047079,
        "name": "ds_granularity",
        "logicalName": "ds_granularity",
        "type": "STRING",
        "logicalType": "STRING",
        "role": "DIMENSION",
        "aggrType": "NONE",
        "seq": 13
    }, {
        "id": 10047080,
        "name": "ds_include_geo",
        "logicalName": "ds_include_geo",
        "type": "STRING",
        "logicalType": "STRING",
        "role": "DIMENSION",
        "aggrType": "NONE",
        "seq": 14
    }, {
        "id": 10047081,
        "name": "ingestion_conf",
        "logicalName": "ingestion_conf",
        "type": "STRING",
        "logicalType": "STRING",
        "role": "DIMENSION",
        "aggrType": "NONE",
        "seq": 15
    }, {
        "id": 10047082,
        "name": "ds_linked_workspaces",
        "logicalName": "ds_linked_workspaces",
        "type": "INTEGER",
        "logicalType": "INTEGER",
        "role": "MEASURE",
        "aggrType": "NONE",
        "seq": 16
    }, {
        "id": 10047083,
        "name": "ds_name",
        "logicalName": "ds_name",
        "type": "STRING",
        "logicalType": "STRING",
        "role": "DIMENSION",
        "aggrType": "NONE",
        "seq": 17
    }, {
        "id": 10047084,
        "name": "ds_owner_id",
        "logicalName": "ds_owner_id",
        "type": "STRING",
        "logicalType": "STRING",
        "role": "DIMENSION",
        "aggrType": "NONE",
        "seq": 18
    }, {
        "id": 10047085,
        "name": "ds_partition_keys",
        "logicalName": "ds_partition_keys",
        "type": "STRING",
        "logicalType": "STRING",
        "role": "DIMENSION",
        "aggrType": "NONE",
        "seq": 19
    }, {
        "id": 10047086,
        "name": "ds_partition_separator",
        "logicalName": "ds_partition_separator",
        "type": "STRING",
        "logicalType": "STRING",
        "role": "DIMENSION",
        "aggrType": "NONE",
        "seq": 20
    }, {
        "id": 10047087,
        "name": "ds_published",
        "logicalName": "ds_published",
        "type": "STRING",
        "logicalType": "STRING",
        "role": "DIMENSION",
        "aggrType": "NONE",
        "seq": 21
    }, {
        "id": 10047088,
        "name": "ds_seg_granularity",
        "logicalName": "ds_seg_granularity",
        "type": "STRING",
        "logicalType": "STRING",
        "role": "DIMENSION",
        "aggrType": "NONE",
        "seq": 22
    }, {
        "id": 10047089,
        "name": "ds_src_type",
        "logicalName": "ds_src_type",
        "type": "STRING",
        "logicalType": "STRING",
        "role": "DIMENSION",
        "aggrType": "NONE",
        "seq": 23
    }, {
        "id": 10047090,
        "name": "ds_status",
        "logicalName": "ds_status",
        "type": "STRING",
        "logicalType": "STRING",
        "role": "DIMENSION",
        "aggrType": "NONE",
        "seq": 24
    }, {
        "id": 10047091,
        "name": "dc_id",
        "logicalName": "dc_id",
        "type": "STRING",
        "logicalType": "STRING",
        "role": "DIMENSION",
        "aggrType": "NONE",
        "seq": 25
    }, {
        "id": 10047092,
        "name": "ss_id",
        "logicalName": "ss_id",
        "type": "STRING",
        "logicalType": "STRING",
        "role": "DIMENSION",
        "aggrType": "NONE",
        "seq": 26
    }, {
        "id": 10047093,
        "name": "smy_id",
        "logicalName": "smy_id",
        "type": "LONG",
        "logicalType": "INTEGER",
        "role": "MEASURE",
        "aggrType": "NONE",
        "seq": 27
    }],
    "id": "9f8d9c91-2067-416c-8571-d84e654ae5b6",
    "engineName": "datasource",
    "description": "",
    "status": "ENABLED",
    "ingestion": {
        "type": "single",
        "connection": {
            "implementor": "MYSQL",
            "type": "JDBC",
            "hostname": "localhost",
            "port": 3306,
            "username": "polaris",
            "password": "polaris",
            "authenticationType": "MANUAL",
            "linkedWorkspaces": 0,
            "connectUrl": "jdbc:mysql://localhost:3306/"
        },
        "database": "polaris",
        "dataType": "TABLE",
        "query": "datasource",
        "rollup": true,
        "intervals": ["2019-03-10T09/2019-03-18T17"],
        "fetchSize": 200,
        "maxLimit": 10000000,
        "scope": "ALL"
    },
    "contexts": {},
    "dsType": "MASTER",
    ...
}

[koeun222]

@AnnieHwang 추가된내용 디자인 첨부합니다- 검토 부탁드려요

1. Datasource > Ingestion : 아래내용 적용 부탁드려요-

   (Geo) Point/Line/Polygon타입으로 수정 즉시 WKT포맷 유효성 검사가 이뤄져야 합니다.
   더불어 유효하지 않을경우 경고를 띄우며, 좌표계 Select box를 비활성화 해주세요.

2. Datasource > Column details > Configure the schema
   

3. Metadata > Column schema
   

[brandon-wonjune]

@kyungtaak GEO타입으로 데이터소스 생성중 에러 문의 입니다.
아래의 파일을 이용하여 테스트하였습니다.

sales_geo_test.csv.zip

1. 데이터소스 생성 > 파일로 생성 > 파일 업로드
2. config 화면에서 가장 첫번째 column인 GeoPoint를 GEO_POINT타입으로 변경
3. 데이터소스 생성
4. error

// api
api/datasources

// error
2019-03-21T04:31:30,130 INFO [task-runner-0-priority-0] io.druid.indexing.overlord.TaskRunnerUtils - Task [index_test4_2_2019-03-21T04:31:25.849Z] status changed to [FAILED]. 
2019-03-21T04:31:30,132 INFO [task-runner-0-priority-0] io.druid.indexing.worker.executor.ExecutorLifecycle - Task completed with status: { 
"id" : "index_test4_2_2019-03-21T04:31:25.849Z", 
"status" : "FAILED", 
"duration" : 338, 
"reason" : "Exception: [io.druid.segment.serde.StructMetricSerde$2.extractElement(StructMetricSerde.java:189), io.druid.segment.serde.StructMetricSerde$2.extractValue(StructMetricSerde.java:137), io.druid.segment.ColumnSelectorFactories$FromInputRow$8.get(ColumnSelectorFactories.java:576), io.druid.query.aggregation.Aggregators$3.aggregate(Aggregators.java:245), io.druid.segment.incremental.OnheapIncrementalIndex.aggregate(OnheapIncrementalIndex.java:238), io.druid.segment.incremental.OnheapIncrementalIndex.factorizeAggs(OnheapIncrementalIndex.java:223), io.druid.segment.incremental.OnheapIncrementalIndex.addToFacts(OnheapIncrementalIndex.java:195), io.druid.segment.incremental.IncrementalIndex.addTimeAndDims(IncrementalIndex.java:445), io.druid.segment.incremental.IncrementalIndex.add(IncrementalIndex.java:440), io.druid.segment.realtime.plumber.Sink.add(Sink.java:160), io.druid.indexing.common.index.YeOldePlumberSchool$1.add(YeOldePlumberSchool.java:139), io.druid.indexing.common.task.IndexTask.generateSegment(IndexTask.java:415), ... more]" 
} 

[kyungtaak]

@brandon-wonjune "GeoPoint" 컬럼값이 "25.9597,-80.1403" 인데요.

1. 이미 wkt 유효값을 검증하면 오류가 나야 정상입니다.
2. "POINT (lat lon)" 로 만들고 테스트 해보시면 됩니다.

[brandon-wonjune]

@kyungtaak wkt 유효값 검증시 사용된 파라메터 정보입니다.
wkt유효값 검증시 valid가 TRUE가 되고있습니다.

// api
api/datasources/validation/wkt

// result
message: null
suggestType: null
valid: true

// param
{
 geoType: "GEO_POINT",
 values: [
"POINT (25.9597 -80.1403)",
"POINT (27.5569 -99.4907)",
"POINT (27.5569 -99.4907)",
"POINT (27.7898 -82.7243)",
"POINT (27.7898 -82.7243)",
"POINT (29.8941 -95.6481)",
"POINT (30.6448 -95.5798)",
"POINT (30.6448 -95.5798)",
"POINT (32.5449 -93.7038)",
"POINT (32.5449 -93.7038)",
"POINT (32.5449 -93.7038)",
"POINT (32.5449 -93.7038)",
"POINT (32.5449 -93.7038)",
"POINT (32.8455 -117.2521)",
"POINT (32.8455 -117.2521)",
"POINT (32.8455 -117.2521)",
"POINT (32.8455 -117.2521)",
"POINT (32.8473 -79.8206)",
"POINT (33.0535 -117.2689)"
 ]
}

[kyungtaak]

@brandon-wonjune 주신 샘플은 그게 아니던데요.;;

[brandon-wonjune]

@kyungtaak 죄송합니다.
실 데이터가 POINT (32.8455 -117.2521)형태로 쌓여있어야 가능한거군요.
클라이언트에서 GEO 타입에 맞게 변환하여 valid 체크를 하는 줄 알았습니다.
수정하여 다시 테스트를 하였습니다.
테스트에 사용된 파일은 아래 입니다.
geo_test.zip

테스트 순서는 다음과 같습니다.

1. 데이터소스 생성 > 파일로 생성 > 파일 업로드
2. config 화면에서 가장 첫번째 column인 GeoPoint를 GEO_POINT타입으로 변경
3. validation 체크 > true 확인

// api 
api/datasources/validation/wkt

// param
{geoType:"GEO_POINT",
values:["POINT (25.9597 -80.1403)","POINT (27.5569 -99.4907)","POINT (27.5569 -99.4907)","POINT (27.7898 -82.7243)","POINT (27.7898 -82.7243)","POINT (29.8941 -95.6481)","POINT (30.6448 -95.5798)","POINT (30.6448 -95.5798)","POINT (32.5449 -93.7038)","POINT (32.5449 -93.7038)","POINT (32.5449 -93.7038)","POINT (32.5449 -93.7038)","POINT (32.5449 -93.7038)","POINT (32.8455 -117.2521)","POINT (32.8455 -117.2521)","POINT (32.8455 -117.2521)","POINT (32.8455 -117.2521)","POINT (32.8473 -79.8206)","POINT (33.0535 -117.2689)"]
}

// result
{
message: null
suggestType: null
valid: true
}

1. 데이터소스 생성
2. 생성된 데이터소스 화면으로 이동
3. ingestion fail error

// error
2019-03-22 05:06:44.558 ERROR [-] [ingestion-02a617cb-02da-4e1c-9191-14e22732515d-0] a.m.d.d.d.i.job.IngestionJobRunner       : Fail to ingestion : IngestionHistory{dataSourceId='02a617cb-02da-4e1c-9191-14e22732515d', ingestionId='index__hacye_2019-03-22T05:06:35.402Z', ingestionMethod=null, duration=null, status=FAILED, errorCode=INGESTION_ENGINE_TASK_ERROR, progress=ENGINE_RUNNING_TASK, cause='DataSourceIngestionException: An error occurred while loading the data source : Exception: [io.druid.segment.serde.StructMetricSerde$2.extractElement(StructMetricSerde.java:189), io.druid.segment.serde.StructMetricSerde$2.extractValue(StructMetricSerde.java:137), io.druid.segment.ColumnSelectorFactories$FromInputRow$8.get(ColumnSelectorFactories.java:576), io.druid.query.aggregation.Aggregators$3.aggregate(Aggregators.java:245), io.druid.segment.incremental.OnheapIncrementalIndex.aggregate(OnheapIncrementalIndex.java:238), io.druid.segment.incremental.OnheapIncrementalIndex.factorizeAggs(OnheapIncrementalIndex.java:223), io.druid.segment.incremental.OnheapIncrementalIndex.addToFacts(OnheapIncrementalIndex.java:195), io.druid.segment.incremental.IncrementalIndex.addTimeAndDims(IncrementalIndex.java:445), io.druid.segment.incremental.IncrementalIndex.add(IncrementalIndex.java:440), io.druid.segment.realtime.plumber.Sink.add(Sink.java:160), io.druid.indexing.common.index.YeOldePlumberSchool$1.add(YeOldePlumberSchool.java:139), io.druid.indexing.common.task.IndexTask.generateSegment(IndexTask.java:415), ... more]'} app.metatron.discovery.domain.datasource.ingestion.IngestionHistory@6ebce9c2

app.metatron.discovery.domain.datasource.DataSourceIngestionException: An error occurred while loading the data source : Exception: [io.druid.segment.serde.StructMetricSerde$2.extractElement(StructMetricSerde.java:189), io.druid.segment.serde.StructMetricSerde$2.extractValue(StructMetricSerde.java:137), io.druid.segment.ColumnSelectorFactories$FromInputRow$8.get(ColumnSelectorFactories.java:576), io.druid.query.aggregation.Aggregators$3.aggregate(Aggregators.java:245), io.druid.segment.incremental.OnheapIncrementalIndex.aggregate(OnheapIncrementalIndex.java:238), io.druid.segment.incremental.OnheapIncrementalIndex.factorizeAggs(OnheapIncrementalIndex.java:223), io.druid.segment.incremental.OnheapIncrementalIndex.addToFacts(OnheapIncrementalIndex.java:195), io.druid.segment.incremental.IncrementalIndex.addTimeAndDims(IncrementalIndex.java:445), io.druid.segment.incremental.IncrementalIndex.add(IncrementalIndex.java:440), io.druid.segment.realtime.plumber.Sink.add(Sink.java:160), io.druid.indexing.common.index.YeOldePlumberSchool$1.add(YeOldePlumberSchool.java:139), io.druid.indexing.common.task.IndexTask.generateSegment(IndexTask.java:415), ... more]
    at app.metatron.discovery.domain.datasource.ingestion.job.IngestionJobRunner.ingestion(IngestionJobRunner.java:198)
    at app.metatron.discovery.domain.datasource.DataSourceEventHandler.lambda$checkCreateAuthority$0(DataSourceEventHandler.java:234)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

[kyungtaak]

@brandon-wonjune point 타입의 wkt 형식에 대한 적재 이슈는 별도 이슈로 관리하겠습니다.

[brandon-wonjune]

데이터소스 컬럼 변경 시, 시간타입을 String으로 변경 처리 되나, Data탭에서 에러문구뜸

[kyungtaak]

@brandon-wonjune 적재 이슈는 해결되었습니다. 원래 목표하던 기능이 완료 되었으면, 이번릴리즈에 함께 포함해서 가는게 좋을것 같네요.

[brandon-wonjune]

@kyungtaak 메타데이터 쪽 확인이 되면 PR오픈하도록 하겠습니다

[brandon-wonjune]

@kyungtaak 소스생성후 상세화면에서 적재시 에러가 발생합니다.
테스트한 파일과 순서는 위와 동일하게 하였습니다.
druid는 https://github.com/metatron-app/metatron-discovery 메인화면에 있는것으로 하였습니다.

// error
2019-03-28 01:27:04.429 ERROR [-] [ingestion-796ce20e-0371-4e71-9c81-f12e6eb78c7b-0] a.m.d.d.d.i.job.IngestionJobRunner       : Fail to ingestion : IngestionHistory{dataSourceId='796ce20e-0371-4e71-9c81-f12e6eb78c7b', ingestionId='null', ingestionMethod=null, duration=null, status=FAILED, errorCode=INGESTION_ENGINE_ACCESS_ERROR, progress=ENGINE_INIT_TASK, cause='QueryTimeExcetpion: [{"error":"Could not resolve type id 'latlon.shape' into a subtype of [simple type, class io.druid.segment.lucene.LuceneIndexingStrategy]\n at [Source: HttpInputOverHTTP@327bc1b1; line: 1, column: 1766] (through reference chain: java.util.LinkedHashMap[\"GeoPoint\"]->java.util.ArrayList[0])"}]'} app.metatron.discovery.domain.datasource.ingestion.IngestionHistory@3080a5f4

app.metatron.discovery.domain.datasource.DataSourceIngestionException: Failed to access to the engine
    at app.metatron.discovery.domain.datasource.ingestion.job.AbstractIngestionJob.doIngestion(AbstractIngestionJob.java:184)
    at app.metatron.discovery.domain.datasource.ingestion.job.FileIngestionJob.process(FileIngestionJob.java:127)
    at app.metatron.discovery.domain.datasource.ingestion.job.IngestionJobRunner.ingestion(IngestionJobRunner.java:191)
    at app.metatron.discovery.domain.datasource.DataSourceEventHandler.lambda$checkCreateAuthority$0(DataSourceEventHandler.java:244)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: app.metatron.discovery.domain.engine.EngineException: Fail to process response : [{"error":"Could not resolve type id 'latlon.shape' into a subtype of [simple type, class io.druid.segment.lucene.LuceneIndexingStrategy]\n at [Source: HttpInputOverHTTP@327bc1b1; line: 1, column: 1766] (through reference chain: java.util.LinkedHashMap[\"GeoPoint\"]->java.util.ArrayList[0])"}]
    at app.metatron.discovery.domain.engine.AbstractEngineRepository.call(AbstractEngineRepository.java:126)
    at app.metatron.discovery.domain.engine.AbstractEngineRepository.call(AbstractEngineRepository.java:111)
    at app.metatron.discovery.domain.engine.DruidEngineRepository.ingestion(DruidEngineRepository.java:71)
    at app.metatron.discovery.domain.engine.DruidEngineRepository$$FastClassBySpringCGLIB$$7c06ef0f.invoke(<generated>)
    at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
    at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:85)
    at app.metatron.discovery.domain.engine.QueryServiceAspect.engineQueryProcessing(QueryServiceAspect.java:119)
    at sun.reflect.GeneratedMethodAccessor232.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:629)
    at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:618)
    at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
    at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
    at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673)
    at app.metatron.discovery.domain.engine.DruidEngineRepository$$EnhancerBySpringCGLIB$$90f07aaf.ingestion(<generated>)
    at app.metatron.discovery.domain.datasource.ingestion.job.AbstractIngestionJob.doIngestion(AbstractIngestionJob.java:180)
    ... 8 common frames omitted
Caused by: app.metatron.discovery.domain.datasource.data.QueryTimeExcetpion: [{"error":"Could not resolve type id 'latlon.shape' into a subtype of [simple type, class io.druid.segment.lucene.LuceneIndexingStrategy]\n at [Source: HttpInputOverHTTP@327bc1b1; line: 1, column: 1766] (through reference chain: java.util.LinkedHashMap[\"GeoPoint\"]->java.util.ArrayList[0])"}]
    at app.metatron.discovery.domain.engine.DruidEngineRepository$QueryResponseErrorHandler.handleError(DruidEngineRepository.java:115)
    at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:707)
    at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:660)
    at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:620)
    at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:538)
    at app.metatron.discovery.domain.engine.AbstractEngineRepository.call(AbstractEngineRepository.java:120)
    ... 28 common frames omitted