Issues
From raju
Importing Pandas is slow
Why this
Organizing my thoughts and trying to systematically debug the issue to either find a fix or report the issue upstream.
There is already a similar issue reported upstream - https://github.com/pandas-dev/pandas/issues/7282 - "PERF: pandas import is too slow" . It has a lot of comments which make it a bit difficult to follow. As of 2017-12-31, the issue is closed without any fix.
Situation
Importing pandas takes ~7s on one machine and ~1s on another. The idea is to figure out why it is slow on one machine.
Computer 1:
$time python -c "import pandas" real 0m6.852s user 0m0.000s sys 0m0.016s
Computer 2:
$ time python3 -c "import pandas" real 0m1.401s user 0m1.288s sys 0m0.096s
Is this a configuration issue?
No.
Computer 1 is ideapad FLEX 5-1570. It has Intel Core i7-7500U CPU @ 2.70 GHz 2.90 GHz, 16 GB ram, Windows 10 Home, Version 1709, OS Build 16299.125, 512 GB SSD drive.
Computer 2 is from a VPS provider. It has Intel Xeon CPU E5-2680 v3 @ 2.50GHz, 1 core, 1GB ram, Debian Stretch, 10 GB SSD Block Storage.
So computer 1 is definitely more powerful than computer 2 but pandas is still slower on 1.
Is this an issue with numpy as well?
Yes, numpy is also loading slower. It explains ~1.5s out of those ~7 s. But what about the rest?
Computer 1
$time python -c "import numpy" real 0m1.444s user 0m0.000s sys 0m0.015s
Computer 2
$ time python3 -c "import numpy" real 0m0.350s user 0m0.336s sys 0m0.008s
What package versions are installed?
Computer1 | Computer2 |
---|---|
$python -c "import pandas as pd; pd.show_versions()" INSTALLED VERSIONS ------------------ commit: None python: 3.6.3.final.0 python-bits: 64 OS: Windows OS-release: 10 machine: AMD64 processor: Intel64 Family 6 Model 142 Stepping 9, GenuineIntel byteorder: little LC_ALL: None LANG: en_US.UTF-8 LOCALE: None.None pandas: 0.20.3 pytest: 3.2.1 pip: 9.0.1 setuptools: 36.5.0.post20170921 Cython: 0.26.1 numpy: 1.13.3 scipy: 0.19.1 xarray: None IPython: 6.1.0 sphinx: 1.6.3 patsy: 0.4.1 dateutil: 2.6.1 pytz: 2017.2 blosc: None bottleneck: 1.2.1 tables: 3.4.2 numexpr: 2.6.2 feather: None matplotlib: 2.1.0 openpyxl: 2.4.8 xlrd: 1.1.0 xlwt: 1.3.0 xlsxwriter: 1.0.2 lxml: 4.1.0 bs4: 4.6.0 html5lib: 0.999999999 sqlalchemy: 1.1.13 pymysql: None psycopg2: None jinja2: 2.9.6 s3fs: None pandas_gbq: None pandas_datareader: 0.5.0 |
debian@mercury:~$ python3 -c "import pandas as pd; pd.show_versions()" INSTALLED VERSIONS ------------------ commit: None python: 3.5.3.final.0 python-bits: 64 OS: Linux OS-release: 4.9.0-3-amd64 machine: x86_64 processor: byteorder: little LC_ALL: None LANG: en_US.UTF-8 LOCALE: en_US.UTF-8 pandas: 0.19.2 nose: None pip: None setuptools: 33.1.1 Cython: None numpy: 1.12.1 scipy: 0.18.1 statsmodels: None xarray: None IPython: None sphinx: None patsy: None dateutil: 2.5.3 pytz: 2016.7 blosc: None bottleneck: None tables: 3.3.0 numexpr: 2.6.1 matplotlib: 2.0.0 openpyxl: None xlrd: None xlwt: None xlsxwriter: None lxml: None bs4: 4.5.3 html5lib: 0.999999999 httplib2: None apiclient: None sqlalchemy: None pymysql: None psycopg2: None jinja2: 2.8 boto: None pandas_datareader: None |
dataframe initialization
Is this a bug?
Issue:- Assigning a scalar value to an empty dataframe does not work.
Consider
$ python3 Python 3.5.3 (default, Jan 19 2017, 14:11:04) [GCC 6.3.0 20170118] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import pandas as pd >>> a = pd.DataFrame(columns=['foo']) >>> a Empty DataFrame Columns: [foo] Index: []
Assigning a scalar value to column foo does not work. It does not return an error either.
>>> a['foo'] = 5 >>> a Empty DataFrame Columns: [foo] Index: []
Assigning a list works as expected.
>>> a['foo'] = [5] >>> a foo 0 5
But once column foo is created, assigning a scalar value to a new column works.
>>> a['bar'] = 3 >>> a foo bar 0 5 3 >>> exit()