From raju

Importing Pandas is slow

Why this

Organizing my thoughts and trying to systematically debug the issue to either find a fix or report the issue upstream.

There is already a similar issue reported upstream - - "PERF: pandas import is too slow" . It has a lot of comments which make it a bit difficult to follow. As of 2017-12-31, the issue is closed without any fix.


Importing pandas takes ~7s on one machine and ~1s on another. The idea is to figure out why it is slow on one machine.

Computer 1:

$time python -c "import pandas"

real    0m6.852s
user    0m0.000s
sys     0m0.016s

Computer 2:

$ time python3 -c "import pandas"

real    0m1.401s
user    0m1.288s
sys     0m0.096s

Is this a configuration issue?


Computer 1 is ideapad FLEX 5-1570. It has Intel Core i7-7500U CPU @ 2.70 GHz 2.90 GHz, 16 GB ram, Windows 10 Home, Version 1709, OS Build 16299.125, 512 GB SSD drive.

Computer 2 is from a VPS provider. It has Intel Xeon CPU E5-2680 v3 @ 2.50GHz, 1 core, 1GB ram, Debian Stretch, 10 GB SSD Block Storage.

So computer 1 is definitely more powerful than computer 2 but pandas is still slower on 1.

Is this an issue with numpy as well?

Yes, numpy is also loading slower. It explains ~1.5s out of those ~7 s. But what about the rest?

Computer 1

$time python -c "import numpy"

real    0m1.444s
user    0m0.000s
sys     0m0.015s

Computer 2

$ time python3 -c "import numpy"

real    0m0.350s
user    0m0.336s
sys     0m0.008s

What package versions are installed?

Computer1 Computer2
$python -c "import pandas as pd; pd.show_versions()"

commit: None
python-bits: 64
OS: Windows
OS-release: 10
machine: AMD64
processor: Intel64 Family 6 Model 142 Stepping 9, GenuineIntel
byteorder: little
LC_ALL: None
LOCALE: None.None

pandas: 0.20.3
pytest: 3.2.1
pip: 9.0.1
setuptools: 36.5.0.post20170921
Cython: 0.26.1
numpy: 1.13.3
scipy: 0.19.1
xarray: None
IPython: 6.1.0
sphinx: 1.6.3
patsy: 0.4.1
dateutil: 2.6.1
pytz: 2017.2
blosc: None
bottleneck: 1.2.1
tables: 3.4.2
numexpr: 2.6.2
feather: None
matplotlib: 2.1.0
openpyxl: 2.4.8
xlrd: 1.1.0
xlwt: 1.3.0
xlsxwriter: 1.0.2
lxml: 4.1.0
bs4: 4.6.0
html5lib: 0.999999999
sqlalchemy: 1.1.13
pymysql: None
psycopg2: None
jinja2: 2.9.6
s3fs: None
pandas_gbq: None
pandas_datareader: 0.5.0
debian@mercury:~$ python3 -c "import pandas as pd; pd.show_versions()"

commit: None
python-bits: 64
OS: Linux
OS-release: 4.9.0-3-amd64
machine: x86_64
byteorder: little
LC_ALL: None

pandas: 0.19.2
nose: None
pip: None
setuptools: 33.1.1
Cython: None
numpy: 1.12.1
scipy: 0.18.1
statsmodels: None
xarray: None
IPython: None
sphinx: None
patsy: None
dateutil: 2.5.3
pytz: 2016.7
blosc: None
bottleneck: None
tables: 3.3.0
numexpr: 2.6.1
matplotlib: 2.0.0
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: 4.5.3
html5lib: 0.999999999
httplib2: None
apiclient: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: 2.8
boto: None
pandas_datareader: None

dataframe initialization

Is this a bug?

Issue:- Assigning a scalar value to an empty dataframe does not work.


$ python3
Python 3.5.3 (default, Jan 19 2017, 14:11:04) 
[GCC 6.3.0 20170118] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas as pd
>>> a = pd.DataFrame(columns=['foo'])
>>> a
Empty DataFrame
Columns: [foo]
Index: []

Assigning a scalar value to column foo does not work. It does not return an error either.

>>> a['foo'] = 5
>>> a
Empty DataFrame
Columns: [foo]
Index: []

Assigning a list works as expected.

>>> a['foo'] = [5]
>>> a
0    5

But once column foo is created, assigning a scalar value to a new column works.

>>> a['bar'] = 3
>>> a
   foo  bar
0    5    3
>>> exit()