机器学习基础:11 种特征选择策略总结!
机器学习算法与Python实战
共 4616字,需浏览 10分钟
· 2022-06-09
本文与你分享可应用于特征选择的各种技术的有用指南。
删除未使用的列 删除具有缺失值的列 不相关的特征 低方差特征 多重共线性 特征系数 p 值 方差膨胀因子 (VIF) 基于特征重要性的特征选择 使用 sci-kit learn 进行自动特征选择 主成分分析 (PCA)
import pandas as pddata = 'https://raw.githubusercontent.com/pycaret/pycaret/master/datasets/automobile.csv'
df = pd.read_csv(data)
df.sample(5)
df.columns
>> Index(['symboling', 'normalized-losses', 'make', 'fuel-type', 'aspiration', 'num-of-doors', 'body-style', 'drive-wheels', 'engine-location','wheel-base', 'length', 'width', 'height', 'curb-weight', 'engine-type', 'num-of-cylinders', 'engine-size', 'fuel-system', 'bore', 'stroke', 'compression-ratio', 'horsepower', 'peak-rpm', 'city-mpg', 'highway-mpg', 'price'], dtype='object')
现在让我们深入研究特征选择的 11 种策略。
1.删除未使用的列
2.删除具有缺失值的列
# total null values per column
df.isnull().sum()
>>
symboling 0
normalized-losses 35
make 0
fuel-type 0
aspiration 0
num-of-doors 2
body-style 0
drive-wheels 0
engine-location 0
wheel-base 0
length 0
width 0
height 0
curb-weight 0
engine-type 0
num-of-cylinders 0
engine-size 0
fuel-system 0
bore 0
stroke 0
compression-ratio 0
horsepower 0
peak-rpm 0
city-mpg 0
highway-mpg 0
price 0
dtype: int64
3.不相关的特征
# correlation between target and features
(df.corr().loc['price']
.plot(kind='barh', figsize=(4,10)))
# drop uncorrelated numeric features (threshold <0.2)
corr = abs(df.corr().loc['price'])
corr = corr[corr<0.2]
cols_to_drop = corr.index.to_list()
df = df.drop(cols_to_drop, axis=1)
import seaborn as sns
sns.boxplot(y = 'price', x = 'fuel-type', data=df)
4.低方差特征
import numpy as np
# variance of numeric features
(df
.select_dtypes(include=np.number)
.var()
.astype('str'))
df['bore'].describe()
5.多重共线性
import matplotlib.pyplot as plt
sns.set(rc={'figure.figsize':(16,10)})
sns.heatmap(df.corr(),
annot=True,
linewidths=.5,
center=0,
cbar=False,
cmap="PiYG")
plt.show()
# drop correlated features
df = df.drop(['length', 'width', 'curb-weight', 'engine-size', 'city-mpg'], axis=1)
还可以使用称为方差膨胀因子 (VIF) 的方法来确定多重共线性并根据高 VIF 值删除特征。我稍后会展示这个例子。
df_cat = df[['fuel-type', 'body-style']]
df_cat.sample(5)
crosstab = pd.crosstab(df_cat['fuel-type'], df_cat['body-style'])
crosstab
from scipy.stats import chi2_contingency
chi2_contingency(crosstab)
# drop columns with missing values
df = df.dropna()
from sklearn.model_selection import train_test_split
# get dummies for categorical features
df = pd.get_dummies(df, drop_first=True)
# X features
X = df.drop('price', axis=1)
# y target
y = df['price']
# split data into training and testing set
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)
from sklearn.linear_model import LinearRegression
# scaling
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
X_train = scaler.fit_transform(X_train)
X_test = scaler.fit_transform(X_test)
# convert back to dataframe
X_train = pd.DataFrame(X_train, columns = X.columns.to_list())
X_test = pd.DataFrame(X_test, columns = X.columns.to_list())
# instantiate model
model = LinearRegression()# fit
model.fit(X_train, y_train)
6.特征系数
# feature coefficients
coeffs = model.coef_
# visualizing coefficients
index = X_train.columns.tolist()
(pd.DataFrame(coeffs, index = index, columns = ['coeff']).sort_values(by = 'coeff')
.plot(kind='barh', figsize=(4,10)))
# filter variables near zero coefficient value
temp = pd.DataFrame(coeffs, index = index, columns = ['coeff']).sort_values(by = 'coeff')
temp = temp[(temp['coeff']>1) | (temp['coeff']< -1)]
# drop those features
cols_coeff = temp.index.to_list()
X_train = X_train[cols_coeff]
X_test = X_test[cols_coeff]
7.p 值
import statsmodels.api as sm
ols = sm.OLS(y, X).fit()
print(ols.summary())
8.方差膨胀因子 (VIF)
VIF = 1 表示无相关性 VIF = 1-5 中等相关性 VIF >5 高相关
from statsmodels.stats.outliers_influence import variance_inflation_factor
# calculate VIF
vif = pd.Series([variance_inflation_factor(X.values, i) for i in range(X.shape[1])], index=X.columns)
# display VIFs in a table
index = X_train.columns.tolist()
vif_df = pd.DataFrame(vif, index = index, columns = ['vif']).sort_values(by = 'vif', ascending=False)
vif_df[vif_df['vif']<10]
9.基于特征重要性选择
from sklearn.ensemble import RandomForestClassifier
# instantiate model
model = RandomForestClassifier(n_estimators=200, random_state=0)
# fit model
model.fit(X,y)
importances = model.feature_importances_
cols = X.columns
(pd.DataFrame(importances, cols, columns = ['importance'])
.sort_values(by='importance', ascending=True)
.plot(kind='barh', figsize=(4,10)))
# calculate standard deviation of feature importances
std = np.std([i.feature_importances_ for i in model.estimators_], axis=0)
# visualization
feat_with_importance = pd.Series(importances, X.columns)
fig, ax = plt.subplots(figsize=(12,5))
feat_with_importance.plot.bar(yerr=std, ax=ax)
ax.set_title("Feature importances")
ax.set_ylabel("Mean decrease in impurity")
10.使用 Scikit Learn 自动选择特征
# import modules
from sklearn.feature_selection import (SelectKBest, chi2, SelectPercentile, SelectFromModel, SequentialFeatureSelector, SequentialFeatureSelector)
基于卡方的技术
# select K best features
X_best = SelectKBest(chi2, k=10).fit_transform(X,y)
# number of best features
X_best.shape[1]
>> 10
# keep 75% top features
X_top = SelectPercentile(chi2, percentile = 75).fit_transform(X,y)
# number of best features
X_top.shape[1]
>> 36
# implement algorithm
from sklearn.svm import LinearSVC
model = LinearSVC(penalty= 'l1', C = 0.002, dual=False)
model.fit(X,y)
# select features using the meta transformer
selector = SelectFromModel(estimator = model, prefit=True)
X_new = selector.transform(X)
X_new.shape[1]
>> 2
# names of selected features
feature_names = np.array(X.columns)
feature_names[selector.get_support()]
>> array(['wheel-base', 'horsepower'], dtype=object)
# instantiate model
model = RandomForestClassifier(n_estimators=100, random_state=0)
# select features
selector = SequentialFeatureSelector(estimator=model, n_features_to_select=10, direction='backward', cv=2)
selector.fit_transform(X,y)
# check names of features selected
feature_names = np.array(X.columns)
feature_names[selector.get_support()]
>> array(['bore', 'make_mitsubishi', 'make_nissan', 'make_saab',
'aspiration_turbo', 'num-of-doors_two', 'body style_hatchback', 'engine-type_ohc', 'num-of-cylinders_twelve', 'fuel-system_spdi'], dtype=object)
11.主成分分析 (PCA)
# import PCA module
from sklearn.decomposition import PCA
# scaling data
X_scaled = scaler.fit_transform(X)
# fit PCA to data
pca = PCA()
pca.fit(X_scaled)
evr = pca.explained_variance_ratio_
# visualizing the variance explained by each principal components
plt.figure(figsize=(12, 5))
plt.plot(range(0, len(evr)), evr.cumsum(), marker="o", linestyle="--")
plt.xlabel("Number of components")
plt.ylabel("Cumulative explained variance")
总结
评论
学习开放日:开放复杂科学、AI+X 海量学习资源!
Datawhale干货 学习开放日:4月27-28日1. 什么是学习开放日?以AI为代表的技术突飞猛进,人类知识森林快速扩张,仅凭一人之力不仅难以覆盖,更是难以串联知识线索。唯有像蚂蚁探索最优路径一样,我们才能在信息爆炸的知识森林中探索出更好的方向!因此,今年集智斑图联合国内最
Datawhale
1
微服务与领域驱动设计,架构实践总结
来源:知了一笑👉 欢迎加入小哈的星球 ,你将获得: 专属的项目实战 / Java 学习路线 / 一对一提问 / 学习打卡 / 赠书福利全栈前后端分离博客项目 2.0 版本完结啦, 演示链接:http://116.62.199.48/ ,新
小哈学Java
0
文末送书 | 大模型时代下如何学习云原生
《containerd 原理剖析与实战》新书内购中,点击阅读原文,限时 69.9 元购买。文末免费赠书大模型与云原生近年来,大语言模型的热度可谓是愈发高涨,尤其是今年年初 Sora 的出现,更是让全球再次看到了AIGC 的巨大威力。Sora 生成实例视频---几头巨大的长毛猛犸踏着积雪的草地而来在当
云原生实验室
10
轻松学习C#:百度行驶证C++离线SDK接入详解
效果 先看最终效果SDK 拿到完整包如图,687M解压后看看内容发现有个readme.txt,那就先看看内容1:用vs2015打开sln工程,最好用vs2015 comunity版本,可微软官网下载。2:sdk的doc目录有pdf接口文档。3:工程总入口main.cpp、请参考示例实现您的功能。4:
DotNet NB
9
日语五十音图学习难吗?
学习日语的初学者们都知道,我们学习日语最初就必须要接触日语五十音图,虽然相对其他语言来说,五十音图挺不是很难学,但是很多初学者可能会记不住,别担心,老师为大家带来了学习经验分享,一起来看看吧!五十音图就是日语最基础的基础,也就是地基。由于日语的元音比较单调,只有a/i/u/e/o这五种,再和各种辅音
python教程
0
实现订单30分钟自动取消的策略
原文:juejin.cn/post/7285167401821798400简介在电商和其他涉及到在线支付的应用中,通常需要实现一个功能:如果用户在生成订单后的一定时间内未完成支付,系统将自动取消该订单。本文将详细介绍基于Spring Boot框架实现订单30分钟内未支付自动取消的几种方案,并提供实例
JAVA乐园
0
这两个BI可视化软件如何选择?亲测推荐
如果你是用过BI,一定在Power BI和Tableau中纠结过,到底该用哪一个?这两个都是很不错的BI可视化软件,在Gartner BI排名中名列前二,Power BI更出圈,Tableau则在中大型公司中用的多。具体用哪个,标准很简单,主要看你工作中的需求,公司用哪个BI就学哪个。如果你暂时用不
Python大数据分析
8
【总结】2021- 作为前端,工作中处理过什么复杂的需求?
先说背景,我目前在腾讯IMWeb团队,负责在线教育腾讯课堂的前端研发。都说疫情期间在线教育是风口,我想说,打的赢扛得住也许是机遇,打不赢完全是炮灰。先说流量从春节假期到现在,我们遭遇了前所未有的流量峰值,虽然具体数字不方便透露,但是可以预想得到,那么多所学校在期间强制网络上课,学生加老师的数量是多么
前端自习课
10