这篇文章原文出自kaggle,我大致翻译翻一下,文中给出了reduce_mem_usage方法可以用来自动缩减dataframe占用空间

这篇notebook展示了通过使用更合理的数据类型来减少dataframe的内存使用量

方法如下:

  1. 迭代每一个column
  2. 检查column是否为数字型
  3. 检查column是否可以用integer表示
  4. 找出column下的最大值和最小值
  5. 选择适用于数据范围的最合适的数据类型

通过以上步骤处理后将一份测试数据从1.3GB减少到466MB

源码如下:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
import numpy as np # linear algebra
import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)

def reduce_mem_usage(props):
start_mem_usg = props.memory_usage().sum() / 1024**2
print("Memory usage of properties dataframe is :",start_mem_usg," MB")
NAlist = [] # Keeps track of columns that have missing values filled in.
for col in props.columns:
if props[col].dtype != object: # Exclude strings

# Print current column type
print("******************************")
print("Column: ",col)
print("dtype before: ",props[col].dtype)

# make variables for Int, max and min
IsInt = False
mx = props[col].max()
mn = props[col].min()

# Integer does not support NA, therefore, NA needs to be filled
if not np.isfinite(props[col]).all():
NAlist.append(col)
props[col].fillna(mn-1,inplace=True)

# test if column can be converted to an integer
asint = props[col].fillna(0).astype(np.int64)
result = (props[col] - asint)
result = result.sum()
if result > -0.01 and result < 0.01:
IsInt = True


# Make Integer/unsigned Integer datatypes
if IsInt:
if mn >= 0:
if mx < 255:
props[col] = props[col].astype(np.uint8)
elif mx < 65535:
props[col] = props[col].astype(np.uint16)
elif mx < 4294967295:
props[col] = props[col].astype(np.uint32)
else:
props[col] = props[col].astype(np.uint64)
else:
if mn > np.iinfo(np.int8).min and mx < np.iinfo(np.int8).max:
props[col] = props[col].astype(np.int8)
elif mn > np.iinfo(np.int16).min and mx < np.iinfo(np.int16).max:
props[col] = props[col].astype(np.int16)
elif mn > np.iinfo(np.int32).min and mx < np.iinfo(np.int32).max:
props[col] = props[col].astype(np.int32)
elif mn > np.iinfo(np.int64).min and mx < np.iinfo(np.int64).max:
props[col] = props[col].astype(np.int64)

# Make float datatypes 32 bit
else:
props[col] = props[col].astype(np.float32)

# Print new column type
print("dtype after: ",props[col].dtype)
print("******************************")

# Print final result
print("___MEMORY USAGE AFTER COMPLETION:___")
mem_usg = props.memory_usage().sum() / 1024**2
print("Memory usage is: ",mem_usg," MB")
print("This is ",100*mem_usg/start_mem_usg,"% of the initial size")
return props, NAlist

原文链接:

https://www.kaggle.com/arjanso/reducing-dataframe-memory-size-by-65


我的微信公众号:pyquant