ML之FE：基于单个csv文件数据集(自动切分为两个dataframe表)利用featuretools工具实现自动特征生成/特征衍生
目录
基于单个csv文件数据集(自动切分为两个dataframe表)利用featuretools工具实现自动特征生成/特征衍生
推荐文章
Py之featuretools：featuretools库的简介、安装、使用方法之详细攻略
ML之FE：基于单个csv文件数据集(自动切分为两个dataframe表)利用featuretools工具实现自动特征生成/特征衍生
ML之FE：基于单个csv文件数据集(自动切分为两个dataframe表)利用featuretools工具实现自动特征生成/特征衍生实现
基于单个csv文件数据集(自动切分为两个dataframe表)利用featuretools工具实现自动特征生成/特征衍生
设计思路
1、定义数据集
contents={"name": ['Bob', 'LiSa', 'Mary', 'Alan'],
"ID": [1, 2, 3, 4], # 输出 NaN
"age": [np.nan, 28, 38 , '' ], # 输出
"born": [pd.NaT, pd.Timestamp("1990-01-01"), pd.Timestamp("1980-01-01"), ''], # 输出 NaT
"sex": ['男', '女', '女', '男',], # 输出 None
"hobbey":['打篮球', '打羽毛球', '打乒乓球', '',], # 输出
"money":[200.0, 240.0, 290.0, 300.0], # 输出
"weight":[140.5, 120.8, 169.4, 155.6], # 输出
}
2、DFS设计
- (1)、指定一个包含数据集中所有实体的字典
- (2)、指定实体间如何关联：当两个实体有一对多关系时，我们称之为“one”实体，即“parent entity”。
- (3)、运行深度特征合成：DFS的最小输入是一组实体、一组关系和计算特性的“target_entity”。DFS的输出是一个特征矩阵和相应的特征定义列表。
让我们首先为数据中的每个客户创建一个特性矩阵，那么现在有几十个新特性来描述客户的行为。 - (4)、改变目标的实体：DFS如此强大的原因之一是它可以为我们的数据中的任何实体创建一个特征矩阵。例如，如果我们想为会话构建特性
- (5)、理解特征输出：一般来说，Featuretools通过特性名称引用生成的特性。
为了让特性更容易理解，Featuretools提供了两个额外的工具，Featuretools .graph_feature()和Featuretools .describe_feature()，
来帮助解释什么是特性以及Featuretools生成特性的步骤。 - (6)、特征谱系图
特征谱系图可视地遍历功能生成过程。从基本数据开始，它们一步一步地展示应用的原语和生成的中间特征，以创建最终特征。 - (7)、特征描述：功能工具还可以自动生成功能的英文句子描述。特性描述有助于解释什么是特性，并且可以通过包含手动定义的自定义来进一步改进。
有关如何自定义自动生成的特性描述的详细信息，请参见生成特性描述。
输出结果
-
name ID age born sex hobbey money weight
-
0 Bob
1 NaN NaT 男 打篮球
200.0
140.5
-
1 LiSa
2
28
1990
-01
-01 女 打羽毛球
240.0
120.8
-
2 Mary
3
38
1980
-01
-01 女 打乒乓球
290.0
169.4
-
3 Alan
4 NaT 男
300.0
155.6
-
-------------------------------------------
-
nums_df：----------------------------------
-
name ID age money weight
-
0 Bob
1 NaN
200.0
140.5
-
1 LiSa
2
28.0
240.0
120.8
-
2 Mary
3
38.0
290.0
169.4
-
3 Alan
4 NaN
300.0
155.6
-
cats_df：----------------------------------
-
ID hobbey sex born
-
0
4 NaN 男 NaN
-
1
1 打篮球 男 NaN
-
2
2 打羽毛球 女
1990
-01
-01
-
---------------------------------DFS设计：-----------------------------------
-
feature_matrix_nums
-
ID age money weight cats.hobbey cats.sex cats.COUNT(nums) \
-
name
-
Bob
1 NaN
200.0
140.5 打篮球 男
1.0
-
LiSa
2
28.0
240.0
120.8 打羽毛球 女
1.0
-
Mary
3
38.0
290.0
169.4 NaN NaN NaN
-
-
cats.MAX(nums.age) cats.MAX(nums.money) cats.MAX(nums.weight) \
-
name
-
Bob NaN
200.0
140.5
-
LiSa
28.0
240.0
120.8
-
Mary NaN NaN NaN
-
-
cats.MEAN(nums.age) cats.MEAN(nums.money) cats.MEAN(nums.weight) \
-
name
-
Bob NaN
200.0
140.5
-
LiSa
28.0
240.0
120.8
-
Mary NaN NaN NaN
-
-
cats.MIN(nums.age) cats.MIN(nums.money) cats.MIN(nums.weight) \
-
name
-
Bob NaN
200.0
140.5
-
LiSa
28.0
240.0
120.8
-
Mary NaN NaN NaN
-
-
cats.SKEW(nums.age) cats.SKEW(nums.money) cats.SKEW(nums.weight) \
-
name
-
Bob NaN NaN NaN
-
LiSa NaN NaN NaN
-
Mary NaN NaN NaN
-
-
cats.STD(nums.age) cats.STD(nums.money) cats.STD(nums.weight) \
-
name
-
Bob NaN NaN NaN
-
LiSa NaN NaN NaN
-
Mary NaN NaN NaN
-
-
cats.SUM(nums.age) cats.SUM(nums.money) cats.SUM(nums.weight) \
-
name
-
Bob
0.0
200.0
140.5
-
LiSa
28.0
240.0
120.8
-
Mary NaN NaN NaN
-
-
cats.DAY(born) cats.MONTH(born) cats.WEEKDAY(born) cats.YEAR(born)
-
name
-
Bob NaN NaN NaN NaN
-
LiSa
1.0
1.0
0.0
1990.0
-
Mary NaN NaN NaN NaN
-
features_defs_nums：
29 [<Feature: ID>, <Feature: age>, <Feature: money>, <Feature: weight>, <Feature: cats.hobbey>, <Feature: cats.sex>, <Feature: cats.COUNT(nums)>, <Feature: cats.MAX(nums.age)>, <Feature: cats.MAX(nums.money)>, <Feature: cats.MAX(nums.weight)>, <Feature: cats.MEAN(nums.age)>, <Feature: cats.MEAN(nums.money)>, <Feature: cats.MEAN(nums.weight)>, <Feature: cats.MIN(nums.age)>, <Feature: cats.MIN(nums.money)>, <Feature: cats.MIN(nums.weight)>, <Feature: cats.SKEW(nums.age)>, <Feature: cats.SKEW(nums.money)>, <Feature: cats.SKEW(nums.weight)>, <Feature: cats.STD(nums.age)>, <Feature: cats.STD(nums.money)>, <Feature: cats.STD(nums.weight)>, <Feature: cats.SUM(nums.age)>, <Feature: cats.SUM(nums.money)>, <Feature: cats.SUM(nums.weight)>, <Feature: cats.DAY(born)>, <Feature: cats.MONTH(born)>, <Feature: cats.WEEKDAY(born)>, <Feature: cats.YEAR(born)>]
-
feature_matrix_cats_df
-
hobbey sex COUNT(nums) MAX(nums.age) MAX(nums.money) MAX(nums.weight) \
-
ID
-
4 NaN 男
1 NaN
300.0
155.6
-
1 打篮球 男
1 NaN
200.0
140.5
-
2 打羽毛球 女
1
28.0
240.0
120.8
-
-
MEAN(nums.age) MEAN(nums.money) MEAN(nums.weight) MIN(nums.age) \
-
ID
-
4 NaN
300.0
155.6 NaN
-
1 NaN
200.0
140.5 NaN
-
2
28.0
240.0
120.8
28.0
-
-
MIN(nums.money) MIN(nums.weight) SKEW(nums.age) SKEW(nums.money) \
-
ID
-
4
300.0
155.6 NaN NaN
-
1
200.0
140.5 NaN NaN
-
2
240.0
120.8 NaN NaN
-
-
SKEW(nums.weight) STD(nums.age) STD(nums.money) STD(nums.weight) \
-
ID
-
4 NaN NaN NaN NaN
-
1 NaN NaN NaN NaN
-
2 NaN NaN NaN NaN
-
-
SUM(nums.age) SUM(nums.money) SUM(nums.weight) DAY(born) MONTH(born) \
-
ID
-
4
0.0
300.0
155.6 NaN NaN
-
1
0.0
200.0
140.5 NaN NaN
-
2
28.0
240.0
120.8
1.0
1.0
-
-
WEEKDAY(born) YEAR(born)
-
ID
-
4 NaN NaN
-
1 NaN NaN
-
2
0.0
1990.0
-
features_defs_cats_df：
25 [<Feature: hobbey>, <Feature: sex>, <Feature: COUNT(nums)>, <Feature: MAX(nums.age)>, <Feature: MAX(nums.money)>, <Feature: MAX(nums.weight)>, <Feature: MEAN(nums.age)>, <Feature: MEAN(nums.money)>, <Feature: MEAN(nums.weight)>, <Feature: MIN(nums.age)>, <Feature: MIN(nums.money)>, <Feature: MIN(nums.weight)>, <Feature: SKEW(nums.age)>, <Feature: SKEW(nums.money)>, <Feature: SKEW(nums.weight)>, <Feature: STD(nums.age)>, <Feature: STD(nums.money)>, <Feature: STD(nums.weight)>, <Feature: SUM(nums.age)>, <Feature: SUM(nums.money)>, <Feature: SUM(nums.weight)>, <Feature: DAY(born)>, <Feature: MONTH(born)>, <Feature: WEEKDAY(born)>, <Feature: YEAR(born)>]
-
<Feature: SUM(nums.age)>
-
The sum of the
"age" of all instances of
"nums"
for each
"ID"
in
"cats".
feature_matrix_cats_df.csv
features_defs_cats_df： 25
[<Feature: hobbey>, <Feature: sex>, <Feature: COUNT(nums)>, <Feature: MAX(nums.age)>, <Feature: MAX(nums.money)>, <Feature: MAX(nums.weight)>, <Feature: MEAN(nums.age)>, <Feature: MEAN(nums.money)>, <Feature: MEAN(nums.weight)>, <Feature: MIN(nums.age)>, <Feature: MIN(nums.money)>, <Feature: MIN(nums.weight)>, <Feature: SKEW(nums.age)>, <Feature: SKEW(nums.money)>, <Feature: SKEW(nums.weight)>, <Feature: STD(nums.age)>, <Feature: STD(nums.money)>, <Feature: STD(nums.weight)>, <Feature: SUM(nums.age)>, <Feature: SUM(nums.money)>, <Feature: SUM(nums.weight)>, <Feature: DAY(born)>, <Feature: MONTH(born)>, <Feature: WEEKDAY(born)>, <Feature: YEAR(born)>]
| ID | hobbey | sex | COUNT(nums) | MAX(nums.age) | MAX(nums.money) | MAX(nums.weight) | MEAN(nums.age) | MEAN(nums.money) | MEAN(nums.weight) | MIN(nums.age) | MIN(nums.money) | MIN(nums.weight) | SKEW(nums.age) | SKEW(nums.money) | SKEW(nums.weight) | STD(nums.age) | STD(nums.money) | STD(nums.weight) | SUM(nums.age) | SUM(nums.money) | SUM(nums.weight) | DAY(born) | MONTH(born) | WEEKDAY(born) | YEAR(born) |
| 4 | 男 | 1 | 300 | 155.6 | 300 | 155.6 | 300 | 155.6 | 0 | 300 | 155.6 | ||||||||||||||
| 1 | 打篮球 | 男 | 1 | 200 | 140.5 | 200 | 140.5 | 200 | 140.5 | 0 | 200 | 140.5 | |||||||||||||
| 2 | 打羽毛球 | 女 | 1 | 28 | 240 | 120.8 | 28 | 240 | 120.8 | 28 | 240 | 120.8 | 28 | 240 | 120.8 | 1 | 1 | 0 | 1990 |
| ID | hobbey | sex | COUNT(nums) | ||||||
| 4 | 男 | 1 | |||||||
| 1 | 打篮球 | 男 | 1 | ||||||
| 2 | 打羽毛球 | 女 | 1 | ||||||
| MAX(nums.age) | MAX(nums.money) | MAX(nums.weight) | MEAN(nums.age) | MEAN(nums.money) | MEAN(nums.weight) | MIN(nums.age) | MIN(nums.money) | MIN(nums.weight) | |
| 300 | 155.6 | 300 | 155.6 | 300 | 155.6 | ||||
| 200 | 140.5 | 200 | 140.5 | 200 | 140.5 | ||||
| 28 | 240 | 120.8 | 28 | 240 | 120.8 | 28 | 240 | 120.8 | |
| SKEW(nums.age) | SKEW(nums.money) | SKEW(nums.weight) | STD(nums.age) | STD(nums.money) | STD(nums.weight) | SUM(nums.age) | SUM(nums.money) | SUM(nums.weight) | |
| 0 | 300 | 155.6 | |||||||
| 0 | 200 | 140.5 | |||||||
| 28 | 240 | 120.8 | |||||||
| DAY(born) | MONTH(born) | WEEKDAY(born) | YEAR(born) | ||||||
| 1 | 1 | 0 | 1990 |
字段解释：
- <Feature: hobbey> ： The "hobbey".
- <Feature: sex> ： The "sex".
- <Feature: COUNT(nums)> ： The number of all instances of "nums" for each "ID" in "cats".
- <Feature: MAX(nums.age)> ： The maximum of the "age" of all instances of "nums" for each "ID" in "cats".
- <Feature: MAX(nums.money)> ： The maximum of the "money" of all instances of "nums" for each "ID" in "cats".
- <Feature: MAX(nums.weight)> ： The maximum of the "weight" of all instances of "nums" for each "ID" in "cats".
- <Feature: MEAN(nums.age)> ： The average of the "age" of all instances of "nums" for each "ID" in "cats".
- <Feature: MEAN(nums.money)> ： The average of the "money" of all instances of "nums" for each "ID" in "cats".
- <Feature: MEAN(nums.weight)> ： The average of the "weight" of all instances of "nums" for each "ID" in "cats".
- <Feature: MIN(nums.age)> ： The minimum of the "age" of all instances of "nums" for each "ID" in "cats".
- <Feature: MIN(nums.money)> ： The minimum of the "money" of all instances of "nums" for each "ID" in "cats".
- <Feature: MIN(nums.weight)> ： The minimum of the "weight" of all instances of "nums" for each "ID" in "cats".
- <Feature: SKEW(nums.age)> ： The skewness of the "age" of all instances of "nums" for each "ID" in "cats".
- <Feature: SKEW(nums.money)> ： The skewness of the "money" of all instances of "nums" for each "ID" in "cats".
- <Feature: SKEW(nums.weight)> ： The skewness of the "weight" of all instances of "nums" for each "ID" in "cats".
- <Feature: STD(nums.age)> ： The standard deviation of the "age" of all instances of "nums" for each "ID" in "cats".
- <Feature: STD(nums.money)> ： The standard deviation of the "money" of all instances of "nums" for each "ID" in "cats".
- <Feature: STD(nums.weight)> ： The standard deviation of the "weight" of all instances of "nums" for each "ID" in "cats".
- <Feature: SUM(nums.age)> ： The sum of the "age" of all instances of "nums" for each "ID" in "cats".
- <Feature: SUM(nums.money)> ： The sum of the "money" of all instances of "nums" for each "ID" in "cats".
- <Feature: SUM(nums.weight)> ： The sum of the "weight" of all instances of "nums" for each "ID" in "cats".
- <Feature: DAY(born)> ： The day of the month of the "born".
- <Feature: MONTH(born)> ： The month of the "born".
- <Feature: WEEKDAY(born)> ： The day of the week of the "born".
- <Feature: YEAR(born)> ： The year of the "born".
feature_matrix_nums.csv
features_defs_nums： 29
[<Feature: ID>, <Feature: age>, <Feature: money>, <Feature: weight>, <Feature: cats.hobbey>, <Feature: cats.sex>, <Feature: cats.COUNT(nums)>, <Feature: cats.MAX(nums.age)>, <Feature: cats.MAX(nums.money)>, <Feature: cats.MAX(nums.weight)>, <Feature: cats.MEAN(nums.age)>, <Feature: cats.MEAN(nums.money)>, <Feature: cats.MEAN(nums.weight)>, <Feature: cats.MIN(nums.age)>, <Feature: cats.MIN(nums.money)>, <Feature: cats.MIN(nums.weight)>, <Feature: cats.SKEW(nums.age)>, <Feature: cats.SKEW(nums.money)>, <Feature: cats.SKEW(nums.weight)>, <Feature: cats.STD(nums.age)>, <Feature: cats.STD(nums.money)>, <Feature: cats.STD(nums.weight)>, <Feature: cats.SUM(nums.age)>, <Feature: cats.SUM(nums.money)>, <Feature: cats.SUM(nums.weight)>, <Feature: cats.DAY(born)>, <Feature: cats.MONTH(born)>, <Feature: cats.WEEKDAY(born)>, <Feature: cats.YEAR(born)>]
| name | ID | age | money | weight | cats.hobbey | cats.sex | cats.COUNT(nums) | cats.MAX(nums.age) | cats.MAX(nums.money) | cats.MAX(nums.weight) | cats.MEAN(nums.age) | cats.MEAN(nums.money) | cats.MEAN(nums.weight) | cats.MIN(nums.age) | cats.MIN(nums.money) | cats.MIN(nums.weight) | cats.SKEW(nums.age) | cats.SKEW(nums.money) | cats.SKEW(nums.weight) | cats.STD(nums.age) | cats.STD(nums.money) | cats.STD(nums.weight) | cats.SUM(nums.age) | cats.SUM(nums.money) | cats.SUM(nums.weight) | cats.DAY(born) | cats.MONTH(born) | cats.WEEKDAY(born) | cats.YEAR(born) |
| Bob | 1 | 200 | 140.5 | 打篮球 | 男 | 1 | 200 | 140.5 | 200 | 140.5 | 200 | 140.5 | 0 | 200 | 140.5 | ||||||||||||||
| LiSa | 2 | 28 | 240 | 120.8 | 打羽毛球 | 女 | 1 | 28 | 240 | 120.8 | 28 | 240 | 120.8 | 28 | 240 | 120.8 | 28 | 240 | 120.8 | 1 | 1 | 0 | 1990 | ||||||
| Mary | 3 | 38 | 290 | 169.4 | |||||||||||||||||||||||||
| Alan | 4 | 300 | 155.6 | 男 | 1 | 300 | 155.6 | 300 | 155.6 | 300 | 155.6 | 0 | 300 | 155.6 |
| name | ID | age | money | weight | |||||
| Bob | 1 | 200 | 140.5 | ||||||
| LiSa | 2 | 28 | 240 | 120.8 | |||||
| Mary | 3 | 38 | 290 | 169.4 | |||||
| Alan | 4 | 300 | 155.6 | ||||||
| cats.hobbey | cats.sex | cats.COUNT(nums) | |||||||
| 打篮球 | 男 | 1 | |||||||
| 打羽毛球 | 女 | 1 | |||||||
| 男 | 1 | ||||||||
| cats.MAX(nums.age) | cats.MAX(nums.money) | cats.MAX(nums.weight) | cats.MEAN(nums.age) | cats.MEAN(nums.money) | cats.MEAN(nums.weight) | cats.MIN(nums.age) | cats.MIN(nums.money) | cats.MIN(nums.weight) | |
| 200 | 140.5 | 200 | 140.5 | 200 | 140.5 | ||||
| 28 | 240 | 120.8 | 28 | 240 | 120.8 | 28 | 240 | 120.8 | |
| 300 | 155.6 | 300 | 155.6 | 300 | 155.6 | ||||
| cats.SKEW(nums.age) | cats.SKEW(nums.money) | cats.SKEW(nums.weight) | cats.STD(nums.age) | cats.STD(nums.money) | cats.STD(nums.weight) | cats.SUM(nums.age) | cats.SUM(nums.money) | cats.SUM(nums.weight) | |
| 0 | 200 | 140.5 | |||||||
| 28 | 240 | 120.8 | |||||||
| 0 | 300 | 155.6 | |||||||
| cats.DAY(born) | cats.MONTH(born) | cats.WEEKDAY(born) | cats.YEAR(born) | ||||||
| 1 | 1 | 0 | 1990 | ||||||
字段解释：
- <Feature: ID> ： The "ID".
- <Feature: age> ： The "age".
- <Feature: money> ： The "money".
- <Feature: weight> ： The "weight".
- <Feature: cats.sex> ： The "sex" for the instance of "cats" associated with this instance of "nums".
- <Feature: cats.hobbey> ： The "hobbey" for the instance of "cats" associated with this instance of "nums".
- <Feature: cats.COUNT(nums)> ： The number of all instances of "nums" for each "ID" in "cats" for the instance of "cats" associated with this instance of "nums".
- <Feature: cats.MAX(nums.age)> ： The maximum of the "age" of all instances of "nums" for each "ID" in "cats" for the instance of "cats" associated with this instance of "nums".
- <Feature: cats.MAX(nums.money)> ： The maximum of the "money" of all instances of "nums" for each "ID" in "cats" for the instance of "cats" associated with this instance of "nums".
- <Feature: cats.MAX(nums.weight)> ： The maximum of the "weight" of all instances of "nums" for each "ID" in "cats" for the instance of "cats" associated with this instance of "nums".
- <Feature: cats.MEAN(nums.age)> ： The average of the "age" of all instances of "nums" for each "ID" in "cats" for the instance of "cats" associated with this instance of "nums".
- <Feature: cats.MEAN(nums.money)> ： The average of the "money" of all instances of "nums" for each "ID" in "cats" for the instance of "cats" associated with this instance of "nums".
- <Feature: cats.MEAN(nums.weight)> ： The average of the "weight" of all instances of "nums" for each "ID" in "cats" for the instance of "cats" associated with this instance of "nums".
- <Feature: cats.MIN(nums.age)> ： The minimum of the "age" of all instances of "nums" for each "ID" in "cats" for the instance of "cats" associated with this instance of "nums".
- <Feature: cats.MIN(nums.money)> ： The minimum of the "money" of all instances of "nums" for each "ID" in "cats" for the instance of "cats" associated with this instance of "nums".
- <Feature: cats.MIN(nums.weight)> ： The minimum of the "weight" of all instances of "nums" for each "ID" in "cats" for the instance of "cats" associated with this instance of "nums".
- <Feature: cats.SKEW(nums.age)> ： The skewness of the "age" of all instances of "nums" for each "ID" in "cats" for the instance of "cats" associated with this instance of "nums".
- <Feature: cats.SKEW(nums.money)> ： The skewness of the "money" of all instances of "nums" for each "ID" in "cats" for the instance of "cats" associated with this instance of "nums".
- <Feature: cats.SKEW(nums.weight)> ： The skewness of the "weight" of all instances of "nums" for each "ID" in "cats" for the instance of "cats" associated with this instance of "nums".
- <Feature: cats.STD(nums.age)> ： The standard deviation of the "age" of all instances of "nums" for each "ID" in "cats" for the instance of "cats" associated with this instance of "nums".
- <Feature: cats.STD(nums.money)> ： The standard deviation of the "money" of all instances of "nums" for each "ID" in "cats" for the instance of "cats" associated with this instance of "nums".
- <Feature: cats.STD(nums.weight)> ： The standard deviation of the "weight" of all instances of "nums" for each "ID" in "cats" for the instance of "cats" associated with this instance of "nums".
- <Feature: cats.SUM(nums.age)> ： The sum of the "age" of all instances of "nums" for each "ID" in "cats" for the instance of "cats" associated with this instance of "nums".
- <Feature: cats.SUM(nums.money)> ： The sum of the "money" of all instances of "nums" for each "ID" in "cats" for the instance of "cats" associated with this instance of "nums".
- <Feature: cats.SUM(nums.weight)> ： The sum of the "weight" of all instances of "nums" for each "ID" in "cats" for the instance of "cats" associated with this instance of "nums".
- <Feature: cats.DAY(born)> ： The day of the month of the "born" for the instance of "cats" associated with this instance of "nums".
- <Feature: cats.MONTH(born)> ： The month of the "born" for the instance of "cats" associated with this instance of "nums".
- <Feature: cats.WEEKDAY(born)> ： The day of the week of the "born" for the instance of "cats" associated with this instance of "nums".
- <Feature: cats.YEAR(born)> ： The year of the "born" for the instance of "cats" associated with this instance of "nums".
转载：https://blog.csdn.net/qq_41185868/article/details/115448504
查看评论