From raju

Importing Pandas is slow

Why this

Organizing my thoughts and trying to systematically debug the issue to either find a fix or report the issue upstream.

There is already a similar issue reported upstream - - "PERF: pandas import is too slow" . It has a lot of comments which make it a bit difficult to follow. As of 2017-12-31, the issue is closed without any fix.


Importing pandas takes ~7s on one machine and ~1s on another. The idea is to figure out why it is slow on one machine.

Computer 1:

    $time python -c "import pandas"
    real    0m6.852s
    user    0m0.000s
    sys     0m0.016s

Computer 2:

    $ time python3 -c "import pandas"
    real    0m1.401s
    user    0m1.288s
    sys     0m0.096s

Is this a configuration issue?


Computer 1 is ideapad FLEX 5-1570. It has Intel Core i7-7500U CPU @ 2.70 GHz 2.90 GHz, 16 GB ram, Windows 10 Home, Version 1709, OS Build 16299.125, 512 GB SSD drive.

Computer 2 is from a VPS provider. It has Intel Xeon CPU E5-2680 v3 @ 2.50GHz, 1 core, 1GB ram, Debian Stretch, 10 GB SSD Block Storage.

So computer 1 is definitely more powerful than computer 2 but pandas is still slower on 1.

Is this an issue with numpy as well?

Yes, numpy is also loading slower. It explains ~1.5s out of those ~7 s. But what about the rest?

Computer 1

    $time python -c "import numpy"
    real    0m1.444s
    user    0m0.000s
    sys     0m0.015s

Computer 2

    $ time python3 -c "import numpy"
    real    0m0.350s
    user    0m0.336s
    sys     0m0.008s

What package versions are installed?

Computer1 Computer2
    $python -c "import pandas as pd; pd.show_versions()"
    commit: None
    python-bits: 64
    OS: Windows
    OS-release: 10
    machine: AMD64
    processor: Intel64 Family 6 Model 142 Stepping 9, GenuineIntel
    byteorder: little
    LC_ALL: None
    LANG: en_US.UTF-8
    LOCALE: None.None
    pandas: 0.20.3
    pytest: 3.2.1
    pip: 9.0.1
    setuptools: 36.5.0.post20170921
    Cython: 0.26.1
    numpy: 1.13.3
    scipy: 0.19.1
    xarray: None
    IPython: 6.1.0
    sphinx: 1.6.3
    patsy: 0.4.1
    dateutil: 2.6.1
    pytz: 2017.2
    blosc: None
    bottleneck: 1.2.1
    tables: 3.4.2
    numexpr: 2.6.2
    feather: None
    matplotlib: 2.1.0
    openpyxl: 2.4.8
    xlrd: 1.1.0
    xlwt: 1.3.0
    xlsxwriter: 1.0.2
    lxml: 4.1.0
    bs4: 4.6.0
    html5lib: 0.999999999
    sqlalchemy: 1.1.13
    pymysql: None
    psycopg2: None
    jinja2: 2.9.6
    s3fs: None
    pandas_gbq: None
    pandas_datareader: 0.5.0
    debian@mercury:~$ python3 -c "import pandas as pd; pd.show_versions()"
    commit: None
    python-bits: 64
    OS: Linux
    OS-release: 4.9.0-3-amd64
    machine: x86_64
    byteorder: little
    LC_ALL: None
    LANG: en_US.UTF-8
    LOCALE: en_US.UTF-8
    pandas: 0.19.2
    nose: None
    pip: None
    setuptools: 33.1.1
    Cython: None
    numpy: 1.12.1
    scipy: 0.18.1
    statsmodels: None
    xarray: None
    IPython: None
    sphinx: None
    patsy: None
    dateutil: 2.5.3
    pytz: 2016.7
    blosc: None
    bottleneck: None
    tables: 3.3.0
    numexpr: 2.6.1
    matplotlib: 2.0.0
    openpyxl: None
    xlrd: None
    xlwt: None
    xlsxwriter: None
    lxml: None
    bs4: 4.5.3
    html5lib: 0.999999999
    httplib2: None
    apiclient: None
    sqlalchemy: None
    pymysql: None
    psycopg2: None
    jinja2: 2.8
    boto: None
    pandas_datareader: None

dataframe initialization

Is this a bug?

Issue:- Assigning a scalar value to an empty dataframe does not work.


    $ python3
    Python 3.5.3 (default, Jan 19 2017, 14:11:04) 
    [GCC 6.3.0 20170118] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import pandas as pd
    >>> a = pd.DataFrame(columns=['foo'])
    >>> a
    Empty DataFrame
    Columns: [foo]
    Index: []

Assigning a scalar value to column foo does not work. It does not return an error either.

    >>> a['foo'] = 5
    >>> a
    Empty DataFrame
    Columns: [foo]
    Index: []

Assigning a list works as expected.

    >>> a['foo'] = [5]
    >>> a
    0    5

But once column foo is created, assigning a scalar value to a new column works.

    >>> a['bar'] = 3
    >>> a
       foo  bar
    0    5    3
    >>> exit()