•   绝大多数数据科学技术并不喜欢清理和整理数据；
•   只有20%的时间是在做有用的分析；
•   数据质量问题如果不尽早处理，将会产生级联现象并影响后续工作。

1.   加载一个混乱的数据集；
2.   分析数据质量问题；
3.   进一步挖掘警告信息；
4.   应用策略来减轻这些问题；
5.   检查在半清洗过后的数据的最终质量分析报告。

pip install ydata-quality

from ydata_quality import DataQualityimport pandas as pddf = pd.read_csv('../datasets/transformed/census_10k.csv')

# create the main class that holds all quality modulesdq = DataQuality(df=df)

# run the testsresults = dq.evaluate()

Warnings: TOTAL: 5 warning(s) Priority 1: 1 warning(s) Priority 2: 4 warning(s)

Priority 1 - heavy impact expected: * [DUPLICATES - DUPLICATE COLUMNS] Found 1 columns with exactly the same feature values as other columns.Priority 2 - usage allowed, limited human intelligibility: * [DATA RELATIONS - HIGH COLLINEARITY - NUMERICAL] Found 3 numerical variables with high Variance Inflation Factor (VIF>5.0). The variables listed in results are highly collinear with other variables in the dataset. These will make model explainability harder and potentially give way to issues like overfitting. Depending on your end goal you might want to remove the highest VIF variables. * [ERRONEOUS DATA - PREDEFINED ERRONEOUS DATA] Found 1960 ED values in the dataset. * [DATA RELATIONS - HIGH COLLINEARITY - CATEGORICAL] Found 10 categorical variables with significant collinearity (p-value < 0.05). The variables listed in results are highly collinear with other variables in the dataset and sorted descending according to propensity. These will make model explainability harder and potentially give way to issues like overfitting. Depending on your end goal you might want to remove variables following the provided order. * [DUPLICATES - EXACT DUPLICATES] Found 3 instances with exact duplicate feature values.

• 警告（Warning）：其中包括数据质量分析过程中检测到的问题细节。

• 优先级（Priority）：对每一个检测到的问题，基于该问题预期的影响来分配一个优先级（越低的值表明越高的优先性）。

• 模块（Modules）：每个检测到的问题与某一个模块（例如：数据关系，重复值，等）执行的数据质量检验相关联。

dq.get_warnings(test="DuplicateColumns")

[QualityWarning(category='Duplicates', test='Duplicate Columns', description='Found 1 columns with exactly the same feature values as other columns.', priority=, data={'workclass': ['workclass2']})]

from ydata_quality.bias_fairness import BiasFairness

#create the main class that holds all quality modulesbf = BiasFairness(df=df, sensitive_features=['race', 'sex'], label='income')

# run the testsbf_results = bf.evaluate()

Warnings: TOTAL: 2 warning(s) Priority 2: 2 warning(s)

Priority 2 - usage allowed, limited human intelligibility: * [BIAS&FAIRNESS - PROXY IDENTIFICATION] Found 1 feature pairs of correlation to sensitive attributes with values higher than defined threshold (0.5). * [BIAS&FAIRNESS - SENSITIVE ATTRIBUTE REPRESENTATIVITY] Found 2 values of 'race' sensitive attribute with low representativity in the dataset (below 1.00%).

bf.get_warnings(test='Proxy Identification')

[QualityWarning(category='Bias&Fairness', test='Proxy Identification', description='Found 1 feature pairs of correlation to sensitive attributes with values higher than defined threshold (0.5).', priority=2>, data=features

relationship_sex 0.650656

Name: association, dtype: float64)]

def improve_quality(df: pd.DataFrame):    """Clean the data based on the Data Quality issues found previously."""    # Bias & Fairness df = df.replace({'relationship': {'Husband': 'Married', 'Wife': 'Married'}}) # Substitute gender-based 'Husband'/'Wife' for generic 'Married'

   # Duplicates df = df.drop(columns=['workclass2']) # Remove the duplicated column df = df.drop_duplicates() # Remove exact feature value duplicates

   return df

clean_df = improve_quality(df.copy())

*DataQuality Engine Report:*

Warnings:  TOTAL: 3 warning(s) Priority 2: 3 warning(s)


Priority 2 - usage allowed, limited human intelligibility: * [ERRONEOUS DATA - PREDEFINED ERRONEOUS DATA] Found 1360 ED values in the dataset. * [DATA RELATIONS - HIGH COLLINEARITY - NUMERICAL] Found 3 numerical variables with high Variance Inflation Factor (VIF>5.0). The variables listed in results are highly collinear with other variables in the dataset. These will make model explainability harder and potentially give way to issues like overfitting. Depending on your end goal you might want to remove the highest VIF variables. * [DATA RELATIONS - HIGH COLLINEARITY - CATEGORICAL] Found 9 categorical variables with significant collinearity (p-value < 0.05). The variables listed in results are highly collinear with other variables in the dataset and sorted descending according to propensity. These will make model explainability harder and potentially give way to issues like overfitting. Depending on your end goal you might want to remove variables following the provided order.

*Bias & Fairness Report:*

Warnings:  TOTAL: 1 warning(s) Priority 2: 1 warning(s)


Priority 2 - usage allowed, limited human intelligibility: * [BIAS&FAIRNESS - SENSITIVE ATTRIBUTE REPRESENTATIVITY] Found 2 values of 'race' sensitive attribute with low representativity in the dataset (below 1.00%).