Dolphinscheduler: [Feature][JsonSplit] Propose for process definition json split

Created on 28 Dec 2020  ·  6Comments  ·  Source: apache/dolphinscheduler

we have discussed the split design proposal in mail list for a long time, i make up a list about this. If there are not any further suggestions, i think we can start coding that.
the mail list:
https://lists.apache.org/thread.html/r5dff090f16cf2aaa76af3cd6aa374f6815de7607734d77e36ed40650%40%3Cdev.dolphinscheduler.apache.org%3E

The following tasks need to be carried out in the order of the main title. If you are interested in any task, please comment or reply below the title.

我们已经在邮件列表中讨论了很长一段时间的json拆分的设计方案，我对此做了一个列表。如果没有任何进一步的建议，我想我们可以开始编码了。
以下任务需要按照大标题的顺序进行，如果你对任何任务感兴趣，请在标题下面评论或者回复

new feature
Source
picture lenboo

Most helpful comment

4. master refactoring

Replace the existing workflow/task structure in master with a new structure definition

4. master重构

将master里现有的工作流/任务结构，换成新的结构定义

picture lenboo on 28 Dec 2020
👍3

All 6 comments

1. New tables and data structures, DB query interface (new structures do not delete/modify existing workflow/task structures at this time):


    1. New Workflow Definition Table: t_ds_Process_Definition

|Column Name|Description|

| -------------------------------| ---------------- |

| id | Self-Increasing id |

| code | encoding (original name field)|

| version | version |

| description | description |

| project_code | project code |

| task_type | job type |

| task_params | Job Custom Parameters |

| run_flag | run flag |

| task_priority | job priority |

| worker_group | worker grouping |

| fail_retry_times | Number of failed retries |

| fail_retry_interval | failed retry interval |

| timeout_flag | timeout flag |

| timeout_notify_strategy | Timeout Notification Policy |

| timeout_duration | timeout length |

| create_time | creation time |

| update_time | modification time |

    1. Workflow Job Relationships Table: t_ds_Process_Task_Relation

| Column Name | Description |
| ----------------------- | ------------------------- ------------- |
| id | self incrementing ID |
| project_code | project code |
| process_definition_code | Workflow coding |
| pre_project_code | pre-quoted project code |
| pre_task_code | pre-reference job code |
| condition_type | condition type 0: None 1: Judgment 2: Delayed |
| condition_params | condition parameters (json) |
| post_project_code | post reference project code |
| post_task_code | post reference job code |
| create_time | creation time |
| update_time | modification time |

    1. Job Definition Table: t_ds_Task_Definition

|Column Name|Description|

| -------------------------------| ---------------- |

| id | Self-Increasing id |

| code | encoding (original name field)|

| version | version |

| description | description |

| project_code | project code |

| task_type | job type |

| task_params | Job Custom Parameters |

| run_flag | run flag |

| task_priority | job priority |

| worker_group | worker grouping |

| fail_retry_times | Number of failed retries |

| fail_retry_interval | failed retry interval |

| timeout_flag | timeout flag |

| timeout_notify_strategy | Timeout Notification Policy |

| timeout_duration | timeout length |

| create_time | creation time |

| update_time | modification time |

    1. Workflow Definition Log Table: t_ds_Process_Definition_Log

Add operation type (add, modify, delete), operator, operation time based on workflow definition table


    1. Workflow job relationship log table: t_ds_Process_Task_Relation_Log

Add workflow version, operation type (add, modify, delete), operator, operation time based on job relationship table


    1. Job Definition Log Table: t_ds_Task_Definition_Log

Add operation type (add, modify, delete), operator, operation time based on job definition table


    1. Workflow Instance Table Modification: t_ds_Process_Instance

Add Workflow Definition Version Number Field

1. 新增表以及数据结构，db查询接口(新增结构暂时不要删除/修改现有的工作流/任务结构):


    1. 新增工作流定义表：t_ds_process_definition

| 列名 | 描述 |
| ---- | ---- |
| id            | 自增ID                         |
| code          | 编码（原name字段）               |
| version       | 版本                           |
| description   | 描述                           |
| project_code  | 项目编码                       |
| release_state | 发布状态                       |
| user_id       | 所属用户ID                     |
| global_params | 全局参数                       |
| flag          | 流程是否可用：0 不可用，1 可用 |
|alert_group_id|告警人分组id            |
| timeout       | 超时时间                       |
| tenant_id     | 租户ID                         |
| locations     | 节点坐标信息                    |
| create_time   | 创建时间                       |
| update_time   | 修改时间                       |

    1. 工作流作业关系表：t_ds_process_task_relation

| 列名                    | 描述                                   |
| ----------------------- | -------------------------------------- |
| id                      | 自增ID                                 |
| project_code            | 项目编码                               |
| process_definition_code | 工作流编码                             |
| pre_project_code        | 前置引用项目编码                        |
| pre_task_code           | 前置引用作业编码                        |
| condition_type          | 条件类型 0：无 1：判断 2：延迟            |
| condition_params        | 条件参数（json）                        |
| post_project_code       | 后置引用项目编码                        |
| post_task_code          | 后置引用作业编码                        |
| create_time             | 创建时间                               |
| update_time             | 修改时间                               |

    1. 作业定义表：t_ds_task_definition

| 列名                    | 描述           |
| ----------------------- | -------------- |
| id                      | 自增id         |
| code                    | 编码（原name字段） |
| version                 | 版本           |
| description             | 描述           |
| project_code            | 项目编码        |
| task_type               | 作业类型       |
| task_params             | 作业自定义参数 |
| run_flag                | 运行标志       |
| task_priority           | 作业优先级     |
| worker_group            | worker分组     |
| fail_retry_times        | 失败重试次数   |
| fail_retry_interval     | 失败重试间隔   |
| timeout_flag            | 超时标志       |
| timeout_notify_strategy | 超时通知策略   |
| timeout_duration        | 超时时长       |
| create_time             | 创建时间       |
| update_time             | 修改时间       |
    1. 工作流定义日志表：t_ds_process_definition_log

    工作流定义表基础上增加操作类型（新增、修改、删除）、操作人、操作时间

    1. 工作流作业关系日志表：t_ds_process_task_relation_log

    作业关系表基础上增加工作流版本、操作类型（新增、修改、删除）、操作人、操作时间

    1. 作业定义日志表：t_ds_task_definition_log

    作业定义表基础上增加操作类型（新增、修改、删除）、操作人、操作时间

    1. 工作流实例表修改：t_ds_process_instance

    新增工作流定义版本号字段

lenboo picture lenboo on 28 Dec 2020

2. Refactoring api: use new structure completely, add new functionality to service without modifying controller


    1. Implement code encoding scheme

    2. suggestion: snowflake + check

When need to add a new code, use snowflake algorithm (IP is worker)_ ID / or a fixed ID) to get a new code, check whether the code exists in the database, if so, regenerate the code, and then check...

    1. Refactoring workflow data related processing interfaces, including version processing (query, new, modify, delete, import, export, offline...)
    1. Refactoring workflow instance data related processing interfaces (query, modification, Gantt chart)
    1. Refactoring job instance query interface
    1. Rebuild workflow instance, job instance related statistics interface (UI system homepage, project homepage statistics, related monitoring data)
    1. Job management related interfaces, including version management (add delete check, go online...)

2. 重构api：全部使用新结构, service中新增功能，不用修改controller


    1. 实现code编码方案

    2. 雪花+检查

      在需要新增code的时候，使用雪花算法（ip为worker_id/或是一个固定id）获取一个新的code，在数据库中检查此code是否存在，如果存在，重新生成code,再进行检查。。。


    1. 重构工作流数据相关处理接口，包含版本处理（查询、新建、修改、删除、导入、导出、上下线...）


    1. 重构工作流实例数据相关处理接口（查询、修改、甘特图）


    1. 重构作业实例查询接口


    1. 重构工作流实例、作业实例相关统计接口（UI系统首页、项目首页统计数据、相关监控数据）


    1. 作业管理相关接口，包含版本管理（增删改查，上下线...）

lenboo picture lenboo on 28 Dec 2020

3. Modify the front part:

    1. Workflow editing, saving workflow information, job relationship information, job information
    1. Create workflow a. Reference project-job, B. Create job
    1. API part: use controller to call new interface, front end to dock new interface, delete old interface
    1. Job management list page, create, update, delete, view details

3. 修改前端部分：


    1. 工作流编辑，保存工作流信息，作业关系信息，作业信息


    1. 创建工作流 a.引用项目-作业，b.创建作业


    1. api部分：使用controller调用新接口，前端对接新接口，将旧的接口删掉


    1. 作业管理列表页，创建，更新，删除，查看详情

lenboo picture lenboo on 28 Dec 2020

4. master refactoring

Replace the existing workflow/task structure in master with a new structure definition

4. master重构

将master里现有的工作流/任务结构，换成新的结构定义

lenboo picture lenboo on 28 Dec 2020
👍3

I will do this part

JinyLeeChina picture JinyLeeChina on 31 Dec 2020

I will join

simon824 picture simon824 on 31 Dec 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

yangyichao-mango picture yangyichao-mango  ·  4Comments
gaoyang-projects picture gaoyang-projects  ·  6Comments
LiemLin picture LiemLin  ·  6Comments
boandai picture boandai  ·  4Comments
13430647361 picture 13430647361  ·  3Comments