Koalas: Add percentiles to describe

[rxin]

pandas supports percentiles: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.describe.html

I think we can do a similar thing, but use Spark's approximate percentiles and document that.

[patryk-oleniuk]

I would like to start contributing and as it's marked, this may be a good issue to start.

I should add to koalas.frame.describe() the percentiles=[0.25, 0.5, 0.75] optional parameter. Then using sparks df stat capabilities, add the stat rows:

sdf.stat.approxQuantile("column_name", percentiles , relativeError)

Question:
from https://spark.apache.org/docs/2.0.0/api/R/approxQuantile.html

relativeError - The relative target precision to achieve (>= 0). If set to zero, the exact quantiles are computed, which could be very expensive.

Where do I take the relativeError value from? Or do we set it arbitrarily as 0.0 ?

[rxin]

I think we should have a parameter eg approx_quantile_error so users can
set it, but default to a value reasonable for big data, eg 0.01.

On Wed, May 22, 2019 at 9:48 PM Patryk Oleniuk [e-mail]
wrote:

I would like to start contributing and as it's marked, this may be a good
issue to start.

As I understand, I should add to koalas.frame.describe() the percentiles=[0.25,
0.5, 0.75] optional parameter. Then using sparks df stat capabilities,
add the stat rows:

sdf.stat.approxQuantile("column_name", percentiles , relativeError)

Question:
from https://spark.apache.org/docs/2.0.0/api/R/approxQuantile.html

relativeError - The relative target precision to achieve (>= 0). If set to
zero, the exact quantiles are computed, which could be very expensive.

Where do I take the relativeError value from? Or do we set it arbitrarily
as 0.0 ?

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/databricks/koalas/issues/376?email_source=notifications&email_token=AACO6PGTFTMYTG2EOSBBTTDPWW5TBA5CNFSM4HOXOOMKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWAOL7Q#issuecomment-494986750,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AACO6PEXSNZ4LEYICDB74Q3PWW5TBANCNFSM4HOXOOMA
.

[ueshin]

I remember @garawalid is also trying to improve the function (https://github.com/databricks/koalas/pull/352#issuecomment-494737733).
@garawalid @patryk-oleniuk Could you share your progress to avoid conflict work?

[patryk-oleniuk]

@ueshin thanks for keeping an eye. I'm currently tackling the fact that approxQuantile() returns the list of the list of quantiles. That needs to be appended as rows to the previously computed sdf.describe() dataframe. Workin on koalas/frame.py#L2629.

@garawalid let me know if you have something ready and if I can help.

[ueshin]

I guess we can use sdf.summary() with appropriate parameters instead of sdf.describe().

[patryk-oleniuk]

My current work: https://github.com/patryk-oleniuk/koalas/tree/describe-percentiles
Will submit a PR within a few hours.

[garawalid]

Thanks @patryk-oleniuk, @ueshin

@patryk-oleniuk you can go for it. I'll share my work in your PR.
@ueshin we can combine summary(), describe() and approxQuantile().

[patryk-oleniuk]

Just submitted a PR: https://github.com/databricks/koalas/pull/378
@garawalid feel free to add or comment

[ueshin]

This was resolved by #378.