shardingsphere scaling out component

[avalon566]

i am developing distributed database scaling out component which support mysql and postgresql,the data sync flow split into two step,history data sync and realtime data sync.
i have a idea,this component can integration with shardingsphrere as its scaling out component,also can support small table sync,two-level index sync etc.

[terrymanu]

It is a great good news for auto scaling out feature here.
Hope ti will merge to dev soon.

[avalon566]

here is project repo sharding-scaling

[dongzl]

Hi @avalon566 , Is this component already in use in the production environment?

[sluk3r]

Hi @avalon566 , Is this component already in use in the production environment?

I saw the commit history. The first commit date is 2019-09-18， so I think this compoment has not been in use in production environment.

And there is no unit test in the source code.

@avalon566 Or is there any misunderstand, please give me a point.

[avalon566]

yes,it is not ready for production ,the first beta version will publish in October!

[sluk3r]

Hi @avalon566 , for now, is the only function of sharding-scaling just sync data? and other functions concerning scalingout, like rehashing data, the newly-added data source registry, would be implemented in future?

[terrymanu]

It is not for production ready, but open source is make it together and grow to more and more maturity.
For every new features, we want discuss and design first, do not only put a result here.
So please don't care about whether it is production ready only or not, please pay more attention for how to improvement it together.

[avalon566]

@sluk3r yes,i am working with ss team to finish ss scaling out feature.

[dongzl]

Hi @terrymanu, what conditions dose this component can merge to shardingsphere? How can we contribute to this work? What should we do?

[KomachiSion]

Hi @dongzl, we welcome everyone to join this work.

what conditions dose this component can merge to shardingsphere?

About the component, I think, the first important work is handling dependencies(now working).
And then we need to develop some based function so that it can work with shardingsphere.
If these works finished, it can be merged to shardingsphere.

How can we contribute to this work? What should we do?

open issue in repo sharding-scaling or repo apache/incubator-shardingsphere tell us what's your want to do.
Then Fork repo sharding-scaling to code and open PR, ss team and @avalon566 will discuss the PR with you.

[dongzl]

Hi @dongzl, we welcome everyone to join this work.

what conditions dose this component can merge to shardingsphere?

About the component, I think, the first important work is handling dependencies(now working).
And then we need to develop some based function so that it can work with shardingsphere.
If these works finished, it can be merged to shardingsphere.

How can we contribute to this work? What should we do?

open issue in repo sharding-scaling or repo apache/incubator-shardingsphere tell us what's your want to do.
Then Fork repo sharding-scaling to code and open PR, ss team and @avalon566 will discuss the PR with you.

hi @KomachiSion , maybe I don't describe my question clearly. I know I can fork repo and submit PR , but what based functions must be develop, should we discuss a list of based functions and list them.

[KomachiSion]

Hi @dongzl , I open a issue in sharding-scaling, and welcome to discuss prepare works in there.

Also in #3115.

[ssxlulu]

Hi, @avalon566 ,It takes me some time to overview the issues about the scaling out component, the component has almost coded complete？Right？
So, Do you have some documents about the component? Just like design document, architecture documentation, etc. I think these documents will help other people to kown and take part in the project easily.

[ssxlulu]

When i first know the component, i want to know its particle size of migration, only can migrate database or can migrate tables separately. And do you use canal to parse the mysql binlog?

[avalon566]

@ssxlulu

1. Yes, the first alpha version is coming soon.
2. It is a good advice! We will provide these docs soon. If you are interesting, welcome to join with us.
3. The componment focus on database scaling, now only support migrate whole database, and the schem must same.
4. Base on some reason we implement our own mysql binlog parse compoment.

[KomachiSion]

Hi, @terrymanu @avalon566 @dongzl

The prepare works have almost done, see #3115 .
I think we can write proposal now and start to vote after 4.0.0-RC3 release.

What do you think about?

[ssxlulu]

@avalon566 It is my privilege to join you! So how do you communicate with each other, email or wechat group?

[KomachiSion]

@ssxlulu, Welcome to join us, you can discuss what's you want to do in #3115, sharding-scaling#2, mail-list of [email protected] or in wechat group.

[ssxlulu]

@KomachiSion Thank you. I will take some time for deeply understanding the component, and then discuss with you.

[dongzl]

Hi, @terrymanu @avalon566 @dongzl

The prepare works have almost done, see #3115 .
I think we can write proposal now and start to vote after 4.0.0-RC3 release.

What do you think about?

I have no problem, I think it's OK.

[KomachiSion]

The proposal is:
https://cwiki.apache.org/confluence/display/SHARDINGSPHERE/ShardingScaling+Donation+Proposal

[KomachiSion]

Sharding-scaling has been merged to branch sharding-scaling