Pandas-profiling: Export Summary Report to tabular format

Created on 23 May 2020  ·  3Comments  ·  Source: pandas-profiling/pandas-profiling

Proposed feature

  1. Have a simple table format for all columns that were profiled
  2. Give the ability to choose which attributes (obviously you disregard attributes like 'value_counts','chi_squared','length','scatter_data','histogram_data')
  3. Make the result savable to csv

Alternatives considered

here is what I have done

  1. I saved the profile report to json
  2. I read this json
  3. parsed through and picked only the keys/values that I need
  4. saved that as a DF
  5. Pivoted DF --> Index= columns in original csv, columns = Attributes from profiling report , Values = the actual result from profiling

Additional context

Field | 25% | 5% | 50% | 75% | 95% | count | cv | distinct_count | distinct_count_with_nan | distinct_count_without_nan | freq | histogram_bins | iqr | is_unique | kurtosis | mad | max | max_length | mean | mean_length | median_length | memory_size | min | min_length | mode | n | n_infinite | n_missing | n_unique | n_zeros | p_infinite | p_missing | p_unique | p_zeros | range | skewness | std | sum | top | type | variance
-- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | --
Action   Required | 44 | 13 | 46 | 522 | 522 | 35981 | 2.095601 | 66 | 66 | 66 |   | 50 | 478 | False | 252.279 | 4 | 8714 |   | 220.0795 |   |   | 287848 | 2 |   | 44 | 35981 | 0 | 0 | 66 | 0 | 0 | 0 | 0.001834 | 0 | 8712 | 13.95437 | 461.1989 | 7918682 |   | Variable.TYPE_NUM | 212704.4
Action   Required Desc |   |   |   |   |   | 35981 |   | 77 | 77 | 77 | 12560 |   |   | False |   |   |   | 40 |   | 10.04833 | 7 | 287848 |   | 3 | Repair | 35981 |   | 0 | 77 |   |   | 0 | 0.00214 |   |   |   |   |   | Repair | Variable.TYPE_CAT
Change Date |   |   |   |   |   | 2360 |   | 101 | 102 | 101 | 425 |   |   | False |   |   |   | 10 |   | 3.390039 | 3 | 287848 |   | 3 | 1/8/2014 | 35981 |   | 33621 | 101 |   |   | 0.93441 | 0.042797 |   |   |   |   |   | 1/8/2014 | Variable.TYPE_CAT
Condition |   |   |   |   |   | 35981 |   | 222 | 222 | 222 | 4778 |   |   | False |   |   |   | 4 |   | 4 | 4 | 287848 |   | 4 | 0022 | 35981 |   | 0 | 222 |   |   | 0 | 0.00617 |   |   |   |   |   | 0022 | Variable.TYPE_CAT
Condition   Description |   |   |   |   |   | 35981 |   | 243 | 243 | 243 | 4767 |   |   | False |   |   |   | 40 |   | 16.96565 | 14 | 287848 |   | 3 | Damaged/Broken | 35981 |   | 0 | 243 |   |   | 0 | 0.006754 |   |   |   |   |   | Damaged/Broken | Variable.TYPE_CAT
Create Date |   |   |   |   |   | 35981 |   | 72 | 72 | 72 | 17896 |   |   | False |   |   |   | 10 |   | 8.418276 | 8 | 287848 |   | 8 | 7/9/2015 | 35981 |   | 0 | 72 |   |   | 0 | 0.002001 |   |   |   |   |   | 7/9/2015 | Variable.TYPE_CAT
Design   Required |   |   |   |   |   | 5014 |   | 1 | 2 | 1 | 5014 |   |   | False |   |   |   | 3 |   | 2.721297 | 3 | 287848 |   | 1 |   | 35981 |   | 30967 | 1 |   |   | 0.860649 | 0.000199 |   |   |   |   |   | X | Variable.TYPE_CAT
Elevation | 14 | 10 | 300 | 300 | 300 | 35981 | 0.75038 | 9 | 9 | 9 |   | 9 | 286 | False | 34.99847 | 0 | 5001 |   | 188.9755 |   |   | 287848 | 8 |   | 300 | 35981 | 0 | 0 | 9 | 0 | 0 | 0 | 0.00025 | 0 | 4993 | 0.646436 | 141.8034 | 6799527 |   | Variable.TYPE_NUM | 20108.19
Elevation   Description |   |   |   |   |   | 35981 |   | 50 | 50 | 50 | 21435 |   |   | False |   |   |   | 22 |   | 11.84809 | 10 | 287848 |   | 3 | Substation | 35981 |   | 0 | 50 |   |   | 0 | 0.00139 |   |   |   |   |   | Substation | Variable.TYPE_CAT
End Date |   |   |   |   |   | 3900 |   | 10 | 11 | 10 | 3250 |   |   | False |   |   |   | 10 |   | 3.568717 | 3 | 287848 |   | 3 | 6/2/2010 | 35981 |   | 32081 | 10 |   |   | 0.891609 | 0.002564 |   |   |   |   |   | 6/2/2010 | Variable.TYPE_CAT
Obj Part   Description |   |   |   |   |   | 35981 |   | 709 | 709 | 709 | 2592 |   |   | False |   |   |   | 40 |   | 12.45621 | 11 | 287848 |   | 3 | Structure/Equip Itself | 35981 |   | 0 | 709 |   |   | 0 | 0.019705 |   |   |   |   |   | Structure/Equip Itself | Variable.TYPE_CAT
ObjType   Description |   |   |   |   |   | 35981 |   | 127 | 127 | 127 | 3271 |   |   | False |   |   |   | 35 |   | 14.63281 | 15 | 287848 |   | 3 | Switch | 35981 |   | 0 | 127 |   |   | 0 | 0.00353 |   |   |   |   |   | Switch | Variable.TYPE_CAT
Object Part |   |   |   |   |   | 35981 |   | 707 | 707 | 707 | 2597 |   |   | False |   |   |   | 4 |   | 4 | 4 | 287848 |   | 4 | 0068 | 35981 |   | 0 | 707 |   |   | 0 | 0.019649 |   |   |   |   |   | 0068 | Variable.TYPE_CAT
Object Type |   |   |   |   |   | 35981 |   | 122 | 122 | 122 | 2625 |   |   | False |   |   |   | 10 |   | 6.73989 | 7 | 287848 |   | 5 | ES_CB | 35981 |   | 0 | 122 |   |   | 0 | 0.003391 |   |   |   |   |   | ES_CB | Variable.TYPE_CAT
Planner Group |   |   |   |   |   | 45 |   | 2 | 3 | 2 | 44 |   |   | False |   |   |   | 3 |   | 3 | 3 | 287848 |   | 3 | 2AP | 35981 |   | 35936 | 2 |   |   | 0.998749 | 0.044444 |   |   |   |   |   | 2AP | Variable.TYPE_CAT
Planner Group   Desc |   |   |   |   |   | 45 |   | 2 | 3 | 2 | 44 |   |   | False |   |   |   | 14 |   | 3.013702 | 3 | 287848 |   | 3 | Apparatus Plng | 35981 |   | 35936 | 2 |   |   | 0.998749 | 0.044444 |   |   |   |   |   | Apparatus Plng | Variable.TYPE_CAT
Problem Code |   |   |   |   |   | 35981 |   | 35981 | 35981 | 35981 | 1 |   |   | True |   |   |   | 4 |   | 3.999972 | 4 | 287848 |   | 3 | 000A | 35981 |   | 0 | 35981 |   |   | 0 | 1 |   |   |   |   |   | 1AA7 | Variable.TYPE_CAT
Problem   Statement |   |   |   |   |   | 35981 |   | 34031 | 34031 | 34031 | 6 |   |   | False |   |   |   | 40 |   | 33.00122 | 33 | 287848 |   | 17 | PLC FLDREQ PUBLIC POLE | 35981 |   | 0 | 34031 |   |   | 0 | 0.945805 |   |   |   |   |   | PLC FLDREQ PUBLIC POLE | Variable.TYPE_CAT
Type of Work |   |   |   |   |   | 35981 |   | 13 | 13 | 13 | 21812 |   |   | False |   |   |   | 34 |   | 15.23938 | 15 | 287848 |   | 5 | Substation Crew | 35981 |   | 0 | 13 |   |   | 0 | 0.000361 |   |   |   |   |   | Substation Crew | Variable.TYPE_CAT
Work Center   Descript |   |   |   |   |   | 2806 |   | 18 | 19 | 18 | 1189 |   |   | False |   |   |   | 40 |   | 4.077458 | 3 | 287848 |   | 3 | Substation Apparatus Crew | 35981 |   | 33175 | 18 |   |   | 0.922014 | 0.006415 |   |   |   |   |   | Substation Apparatus Crew | Variable.TYPE_CAT
Work Type |   |   |   |   |   | 35981 |   | 5 | 5 | 5 | 22007 |   |   | False |   |   |   | 2 |   | 1.046358 | 1 | 287848 |   | 1 | S | 35981 |   | 0 | 5 |   |   | 0 | 0.000139 |   |   |   |   |   | S | Variable.TYPE_CAT
Work center |   |   |   |   |   | 2806 |   | 19 | 20 | 19 | 1189 |   |   | False |   |   |   | 7 |   | 3.178761 | 3 | 287848 |   | 2 | 3APPRTS | 35981 |   | 33175 | 19 |   |   | 0.922014 | 0.006771 |   |   |   |   |   | 3APPRTS | Variable.TYPE_CAT
  |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |  
feature request 💬 getting started ☝ help wanted 🙋 no-issue-activity
Source
picture iolap-mmeyyappan

All 3 comments

Hey, I would like to take this.

ppsha3 picture ppsha3 on 1 Jun 2020
👍1

I want to help this as well.

vionwinnie picture vionwinnie on 27 Jul 2020
👍1

Stale issue

github-actions[bot] picture github-actions[bot] on 26 Sep 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

TheMLman picture TheMLman  ·  5Comments
piyushP7pravin picture piyushP7pravin  ·  4Comments
castorfou picture castorfou  ·  5Comments
shahanesanket picture shahanesanket  ·  6Comments
abegong picture abegong  ·  5Comments