Python notes

From raju

Contents

Utility functions

print command before executing it

command line arguments

Code Templates

Template code for regression testing a Python3 application

Manipulating dictionaries

Manipulating lists

Manipulating dataframes

Manipulating strings

using printf

    indent = "  "
    printf("%s average = %.4g" %(indent, df['foo'].mean()) )
    printf("%s N = %d" %(indent, df.shape[0]) )
    

See also:-


string interpolation using locals() and format()

def buildString(user, name = 'john', age=22):
    userId = user.getUserId()
    return "Name: {name}, age: {age}, userid:{userId}".format(**locals())

Tested it using Python 3.4.3

Advantage:

  • This is better than
    return "Name: {name}, age: {age}, userid:{userId}".format(name=name, age=age, userId=userId)

which is ugly.

Disadvantage:

  • Do not use it when the format string is user supplied since then it would open up access to every local variable making the application susceptible to injection attacks.

Ref:-

expand %s in a string with variable values

    >>> a = '%s:micro soft, %s:google'
    >>> b = 'MSFT'; c = 'GOOG';
    >>> a
    '%s:micro soft, %s:google'
    >>> b
    'MSFT'
    >>> c
    'GOOG'
    >>> d = a % (b, c)
    >>> d
    'MSFT:micro soft, GOOG:google'
    


string construction

Use a tuple while constructing a string with multiple placeholders

    >>> 'select * from table_name where column in (%s,%s)' % ('val1','val2')
    'select * from table_name where column in (val1,val2)'
    

Using an array will give an error

    >>> 'select * from table_name where column in (%s,%s)' % ['val1','val2']
    Traceback (most recent call last):
    
      File "<ipython-input-66-xxxxxxxxxxxx>", line 1, in <module>
        'select * from table_name where column in (%s,%s)' % ['val1','val2']
    
    TypeError: not enough arguments for format string
    

Manipulating dates

iterate over a range of dates

To iterate on all days including weekends

    for dt in pd.date_range('20160226', '20160303'):
      print(dt.strftime('%Y%m%d'))
    
    20160226
    20160227
    20160228
    20160229
    20160301
    20160302
    20160303
    

To iterate only on weekdays

    for dt in pd.date_range('20160226', '20160303'):
      if (dt.weekday() < 5):
        print(dt.strftime('%Y%m%d'))
    
    20160226
    20160229
    20160301
    20160302
    20160303
    

To iterate in reverse chronological order

    for dt in pd.date_range('20160226', '20160303')[::-1]:
      if (dt.weekday() < 5):
        print(dt.strftime('%Y%m%d'))
    
    20160303
    20160302
    20160301
    20160229
    20160226
    

To experiment with just one date element

    >>> a = pd.date_range('20160226', '20160303')
    >>> a[0].strftime('%Y%m%d')
    '20160226'
    

To create a pandas.tslib.Timestamp variable

    >>> from datetime import datetime
    >>> for dt in pd.date_range('20160929', '20160930')[::-1]:
    ...     dt == pd.Timestamp(datetime(2016, 9, 29))
    ...
    False
    True
    
    >>> type(pd.Timestamp(datetime(2016, 9, 29)))
    <class 'pandas.tslib.Timestamp'>
    
    >>> for dt in pd.date_range('20160929', '20160930')[::-1]:
    ...     type(dt)
    ...
    <class 'pandas.tslib.Timestamp'>
    <class 'pandas.tslib.Timestamp'>
    

tags | convert output from pandas data_range function to YYYYMMDD, date_range reverse order

Ref:-

Convert_YYYYMMDD_to_YYYY-MM-DD

get the beginning and end dates of a month

To get the first date

    >>> from datetime import datetime
    >>> a = datetime.strptime('20160905', '%Y%m%d')
    >>> a
    datetime.datetime(2016, 9, 5, 0, 0)
    >>> b = a.replace(day=1)
    >>> b
    datetime.datetime(2016, 9, 1, 0, 0)
    >>> a
    datetime.datetime(2016, 9, 5, 0, 0)
    

To get the last date

    >>> from datetime import datetime
    >>> import calendar
    >>> a = datetime.strptime('20160905', '%Y%m%d')
    >>> a
    datetime.datetime(2016, 9, 5, 0, 0)
    >>> b = a.replace(day = calendar.monthrange(a.year, a.month)[1])
    >>> b
    datetime.datetime(2016, 9, 30, 0, 0)
    >>> a
    datetime.datetime(2016, 9, 5, 0, 0)
    

Ref:- search for datetime.replace in https://docs.python.org/2/library/datetime.html

Tips

configure vim

" To add the proper PEP8 indentation
au BufRead,BufNewFile *.py,*.pyw set tabstop=4
au BufRead,BufNewFile *.py,*.pyw set softtabstop=4
au BufRead,BufNewFile *.py,*.pyw set shiftwidth=4
au BufRead,BufNewFile *.py,*.pyw set textwidth=79
au BufRead,BufNewFile *.py,*.pyw set expandtab
au BufRead,BufNewFile *.py,*.pyw set autoindent
au BufRead,BufNewFile *.py,*.pyw set fileformat=unix

tested using vim 7.2

Ref:- https://realpython.com/blog/python/vim-and-python-a-match-made-in-heaven/

Dummy

Tutorials

The advantage of the pdf version is that all the content is in a single file. The advantage of html is that python code is shown with syntax highlighting.

Profiling Python

Install python on windows

documentation links

useful articles

  • http://nedbatchelder.com/text/names1.html - article on how names, values, assignment and mutability works in Python. The concepts are explained very well. Highly recommended for C/C++ programmers looking to learn Python.

write either to file or to stdout

Approach 1:

    import os
    odir = "foo"
    ofile = os.path.join(odir, "bar.txt")
    
    with open(ofile, "w") as fh:
      print('writing', ofile)
      data_dumper(data, fh)
    
    ...
    
    def data_dumper(data, fh)
       fh.write(data)
       ...
    

To print data onto stdout

    import sys
    data_dumper(data, sys.stdout)
    

See also: http://stackoverflow.com/questions/17602878/how-to-handle-both-with-open-and-sys-stdout-nicely

Approach 2:

    def dump_foo(arg1, arg2, ..., param={})
        import sys
        # do something
      
        # dump to a file if some parameters are specified otherwise use stdout
        if (('out_dir' in param) & ('out_file' in param)):
            out_dir = param['out_dir']
            if not os.path.exists(out_dir):
                os.makedirs(out_dir, exist_ok=True)
            out_file = os.path.join(out_dir, param['out_file'])
            fh = open(out_file, "w")
            print('writing', out_file)
        else:
             fh = sys.stdout;
    
        fh.write("I am in dump_foo()\n")
    
        if fh is not sys.stdout:
            fh.close()
    

frequently used functions

Use case function Additional notes
get current directory cwd = os.getcwd()
import modules from a different directory
  • sys.path.append('/path/to/dir') - directory searched last
  • sys.path.insert(0, '/path/to/dir') - directory searched first
Add parent directory to the module search path sys.path.append('..') Module Search Path

frequently used pdb commands

    until NNN - continue until line NNN
    

script to plot a column of data from command line

See https://gitlab.com/d3k2mk7/rutils/blob/master/python3/plot_utils/plot_by_index.py

programming notes

check if a file exists

    import os
    if not os.path.isfile(ofile):
        print("File", ofile, "does not exist")
    

Ref:- http://stackoverflow.com/questions/82831/how-do-i-check-whether-a-file-exists-using-python

Check if a list is empty

    >>> a = []
    >>> if (not a):
    ...     print("List is empty")
    ... 
    List is empty
    

check if a key is exists in a dictionary

    if 'key1' in dict:
      print "blah"
    else:
      print "boo"
    

Ref: http://stackoverflow.com/questions/1602934/check-if-a-given-key-already-exists-in-a-dictionary

check if something is not None

In:

    a=None
    
    if (a is not None):
        print("kama")
    else:
        print("raju")
        
    

Out:

    raju
    

In:

    a=2
    
    if (a is not None):
        print("kama")
    else:
        print("raju")
        
    

Out:

    kama
    

check if a list is sorted

     % cat is_list_sorted.py
    def is_list_sorted(l):
        return all( [l[i] < l[i+1] for i in range(len(l)-1)] )
    
    a = [10,1,8,2,5,3,7,9,6,4]
    print(is_list_sorted(a))
    
    b = sorted(a)
    print(is_list_sorted(b))
    
     % python3 -u is_list_sorted.py
    False
    True
    

Ref:- https://stackoverflow.com/questions/3755136/pythonic-way-to-check-if-a-list-is-sorted-or-not

reload a module

    import importlib
    importlib.reload(mymodule)
    

split string and trim white space

    s = "blah, lots  ,  of ,  spaces, here "
    [x.strip() for x in s.split(',')]
    
    Out[18]: ['blah', 'lots', 'of', 'spaces', 'here']
    

Ref:- http://stackoverflow.com/questions/4071396/split-by-comma-and-strip-whitespace-in-python


convert two lists into a dictionary

Use dict with zip to construct a dictionary from two lists. The number of elements in dictionary will be the minimum of the number of elements in each list. The extra elements will be discarded.

     % python3          
    Python 3.5.2+ (default, Sep 22 2016, 12:18:14) 
    [GCC 6.2.0 20160914] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    
    >>> a = [1, 2, 3]; b = ['a', 'b', 'c']; c = dict(zip(a, b)); print(a); print(b); print(c)
    [1, 2, 3]
    ['a', 'b', 'c']
    {1: 'a', 2: 'b', 3: 'c'}
    
    >>> a = [1, 2, 3, 4]; b = ['a', 'b', 'c']; c = dict(zip(a, b)); print(a); print(b); print(c)
    [1, 2, 3, 4]
    ['a', 'b', 'c']
    {1: 'a', 2: 'b', 3: 'c'}
    
    >>> a = [1, 2, 3]; b = ['a', 'b', 'c', 'd']; c = dict(zip(a, b)); print(a); print(b); print(c)
    [1, 2, 3]
    ['a', 'b', 'c', 'd']
    {1: 'a', 2: 'b', 3: 'c'}
    


debug from command line

To pass command line arguments and invoke the debugger directly

    python -u -m pdb mymodule.py --arg1_key arg1_val --arg2_key arg2_val
    

To do it inside python interpreter

    import pdb
    import mymodule
    pdb.run('mymodule.myfunc()')
    

Ref:-

tags | Debugging python programs from the command line

step through a python script

tags | debug Add the following lines at the beginning of the script

    import pdb
    pdb.set_trace()
    

Ref:- https://pythonconquerstheuniverse.wordpress.com/2009/09/10/debugging-in-python/


script execution

To execute a python script in the interpreter

    exec(open('myscript.py').read(), globals())
    

returning values

One way to return multiple values in a function is to use a dictionary.

    return {'key1':value1,'key2':value2,'key3':value3}
    

or

    result = {'key1':value1}
    result['key2'] = value2
    result['key3'] = value3
    return(result)
    

if else if syntax

    if x < 0:
        x = 0
        print('Negative changed to zero')
    elif x == 0:
        print('Zero')
    elif x == 1:
        print('Single')
    else:
        print('More')
    

check for missing data

    if (x is not None and x > 0):
        print('x is not none and positive')
    

logical operators

    x or y, x and y, not x
    

Example

        if (a < 1 or a > 100):
            print('a is not in [1,100]')
    

Ref:- https://docs.python.org/3/library/stdtypes.html

timing a function

    start_time = time.time()
    call_function_of_interest()
    end_time = time.time()
    print('time elapsed = ', end_time - start_time)
    

Remove duplicates

To remove duplicates from a column of a dataframe

    unique_id = df['id'].drop_duplicates()
    

Ref:- http://pandas.pydata.org/pandas-docs/version/0.17.0/generated/pandas.Series.drop_duplicates.html

tags | unique

Replace newline characters with space

    % python3
    Python 3.4.3 (default, Mar  3 2015, 15:56:43)
    [GCC 4.4.7 20120313 (Red Hat 4.4.7-4)] on linux
    
    >>> a = '''kama
    ... raju'''
    >>> a
    'kama\nraju'
    >>> a.replace("\n", " ")
    'kama raju'
    

overlay one column with another

Say you have two columns 'a' and 'b' in a dataframe. To copy the values from column 'a' to column 'b' whenever the values in column 'b' are missing.

    df.loc[ pd.isnull(df['b']), 'b'] = \
      df.loc[ pd.isnull(df['b']), 'a']
    

tags | pandas update data based on a condition, indexing and selecting data, find and replace values in a python dataframe, replace values if null, assign values by index, search and replace data based on another column using python pandas, example on using loc, python pandas boolean array filter

show all variables in python interpreter

use dir()

    >>> dir()
    ['__builtins__', '__doc__', '__loader__', '__name__', '__package__', '__spec__']
    >>> import pandas as pd
    >>> dir()
    ['__builtins__', '__doc__', '__loader__', '__name__', '__package__', '__spec__', 'pd']
    >>> df = pd.DataFrame([[1, 2], [3, 4]], columns=list('AB'))
    >>> dir()
    ['__builtins__', '__doc__', '__loader__', '__name__', '__package__', '__spec__', 'df', 'pd']
    >>> df
       A  B
    0  1  2
    1  3  4
    

writing an if else condition in list comprehension

Syntax

    [ a if C else b for i in items]
    

is equivalent to

    for i in items:
      if (C):
        a
      else:
        b
    

For example, to figure out if a package is installed one can do

    import pandas as pd
    import apt
    cache = apt.Cache()
    
    a = pd.DataFrame({'package': ['python-apt', 'foo', 'dpkg', '0xffff']})
    status = [
        True if ((pkg in cache) and cache[pkg].is_installed) else False
        for pkg in a['package']
    ]
    
    print(status)
    

which will show

    [True, False, True, False]
    

where python-apt, dpkg are installed on the system, '0xffff' is a valid package but not installed and 'foo' is not a valid package name.

Ref: http://stackoverflow.com/questions/4406389/if-else-in-a-list-comprehension

Stackoverflow links

get the username

    >>> import getpass
    >>> getpass.getuser()
    'rajulocal'
    

exit with error

One way is to throw an exception. See http://stackoverflow.com/questions/2052390/manually-raising-throwing-an-exception-in-python for more details.

args and kwargs

args = arguments kwargs = keyword arguments

    def f(*args, **kwargs):
      print('args: ', args, ' kwargs: ', kwargs)
    
    >>> f('a')
    args:  ('a',)  kwargs:  {}
    >>> f(ar='a')
    args:  ()  kwargs:  {'ar': 'a'}
    >>> f(1,2,param=3)
    args:  (1, 2)  kwargs:  {'param': 3}
    
    >>> f(1,[2,3],param=3)
    args:  (1, [2, 3])  kwargs:  {'param': 3}
    
    >>> f(1, [2,3], [4,5,6], param=3, first='kama', last='raju')
    args:  (1, [2, 3], [4, 5, 6])  kwargs:  {'param': 3, 'last': 'raju', 'first': 'kama'}
    

See also: http://stackoverflow.com/questions/682504/what-is-a-clean-pythonic-way-to-have-multiple-constructors-in-python?noredirect=1&lq=1

multiline statements

    df = pd.read_csv(fname)\
    .pipe(func_foo)\
    .pipe(func_bar)
    

numpy

Frequently used numpy functions

use case solution
average np.average
sum np.sum

Challenges

Rotate matrix anti clockwise

Given a 2-D array

    [[1,2,3,4],
     [5,6,7,8],
     [9,10,11,12]]
    

convert it to

    [[4,8,12],
     [3,7,11],
     [2,6,10],
     [1,5,9]]
    
    >>> a = [[1,2,3,4], [5,6,7,8], [9,10,11,12]]
    >>> a
    [[1, 2, 3, 4], [5, 6, 7, 8], [9, 10, 11, 12]]
    >>> b = list(zip(*a))[::-1]
    >>> b
    [(4, 8, 12), (3, 7, 11), (2, 6, 10), (1, 5, 9)]
    >>> c = [list(elem) for elem in b]
    >>> c
    [[4, 8, 12], [3, 7, 11], [2, 6, 10], [1, 5, 9]]
    

Rotate matrix clockwise

Given a 2-D array

    [[1,2,3,4],
     [5,6,7,8],
     [9,10,11,12]]
    

convert it to

    [[9,5,1],
     [10,6,2],
     [11,7,3],
     [12,8,4]]
    
    >>> a = [[1,2,3,4], [5,6,7,8], [9,10,11,12]]
    >>> a
    [[1, 2, 3, 4], [5, 6, 7, 8], [9, 10, 11, 12]]
    >>> b = list(zip(*a[::-1]))
    >>> b
    [(9, 5, 1), (10, 6, 2), (11, 7, 3), (12, 8, 4)]
    >>> c = [list(elem) for elem in b]
    >>> c
    [[9, 5, 1], [10, 6, 2], [11, 7, 3], [12, 8, 4]]
    

index of an item in a sorted list

Use bisect module to find out where an item can be inserted in a sorted list. The return value of bisect is suitable for use as the first parameter to list.insert() assuming that original list is already sorted.

    >>> a = [10,1,8,2,5,3,7,9,6,4]
    >>> a
    [10, 1, 8, 2, 5, 3, 7, 9, 6, 4]
    >>> a.sort()
    >>> a
    [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
    >>> x = 5.5
    >>> from bisect import bisect
    >>> bisect(a,x)
    5
    >>> a.insert( bisect(a,x), x)
    >>> a
    [1, 2, 3, 4, 5, 5.5, 6, 7, 8, 9, 10]
    

Instead of doing bisect and insert, you can also insort which does both.

    >>> a = [10,1,8,2,5,3,7,9,6,4]
    >>> a
    [10, 1, 8, 2, 5, 3, 7, 9, 6, 4]
    >>> a.sort()
    >>> a
    [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
    >>> x = 5.5
    >>> from bisect import insort
    >>> insort(a,x)
    >>> a
    [1, 2, 3, 4, 5, 5.5, 6, 7, 8, 9, 10]
    

Ref:- https://docs.python.org/3/library/bisect.html