Python notes

From raju

Contents

Dummy

Python related pages on this wiki

Utility functions

print command before executing it

command line arguments

Code Templates

Template code for regression testing a Python3 application

Check pep8 conformance via unittest

Functionality testing of python scripts

using printf

indent = "  "
printf("%s average = %.4g" %(indent, df['foo'].mean()) )
printf("%s N = %d" %(indent, df.shape[0]) )

See also:-

where is a module imported from

import imp
imp.find_module("foo")

string interpolation using format()

>>> a = 'kama'; b = 'raju'; c = "foo_{alpha}_bar_{beta}.txt".format(alpha=a, beta=b)
>>> a
'kama'
>>> b
'raju'
>>> c
'foo_kama_bar_raju.txt'

string interpolation using locals() and format()

def buildString(user, name = 'john', age=22):
    userId = user.getUserId()
    return "Name: {name}, age: {age}, userid:{userId}".format(**locals())

Tested it using Python 3.4.3

Advantage:

  • This is better than
    return "Name: {name}, age: {age}, userid:{userId}".format(name=name, age=age, userId=userId)

which is ugly.

Disadvantage:

  • Do not use it when the format string is user supplied since then it would open up access to every local variable making the application susceptible to injection attacks.

Ref:-

expand %s in a string with variable values

>>> a = '%s:micro soft, %s:google'
>>> b = 'MSFT'; c = 'GOOG';
>>> a
'%s:micro soft, %s:google'
>>> b
'MSFT'
>>> c
'GOOG'
>>> d = a % (b, c)
>>> d
'MSFT:micro soft, GOOG:google'


string construction

Use a tuple while constructing a string with multiple placeholders

>>> 'select * from table_name where column in (%s,%s)' % ('val1','val2')
'select * from table_name where column in (val1,val2)'

Using an array will give an error

>>> 'select * from table_name where column in (%s,%s)' % ['val1','val2']
Traceback (most recent call last):

  File "<ipython-input-66-xxxxxxxxxxxx>", line 1, in <module>
    'select * from table_name where column in (%s,%s)' % ['val1','val2']

TypeError: not enough arguments for format string

pep8 write long strings

    user_agent = 'Mozilla/5.0 (Windows NT 6.1; Win64; x64) ' \
        'AppleWebKit/537.36 (KHTML, like Gecko) ' \
        'Chrome/66.0.3359.139 Safari/537.36'

Ref:- https://stackoverflow.com/questions/1874592/how-to-write-very-long-string-that-conforms-with-pep8-and-prevent-e501

Manipulating dates

get current time as YYYYMMDD_HHMMSS

from datetime import datetime
datetime.now().strftime("%Y%m%d_%H%M%S")

iterate over a range of dates

To iterate on all days including weekends

for dt in pd.date_range('20160226', '20160303'):
  print(dt.strftime('%Y%m%d'))

20160226
20160227
20160228
20160229
20160301
20160302
20160303

To iterate only on weekdays

for dt in pd.date_range('20160226', '20160303'):
  if (dt.weekday() < 5):
    print(dt.strftime('%Y%m%d'))

20160226
20160229
20160301
20160302
20160303

To iterate in reverse chronological order

for dt in pd.date_range('20160226', '20160303')[::-1]:
  if (dt.weekday() < 5):
    print(dt.strftime('%Y%m%d'))

20160303
20160302
20160301
20160229
20160226

To experiment with just one date element

>>> a = pd.date_range('20160226', '20160303')
>>> a[0].strftime('%Y%m%d')
'20160226'

To create a pandas.tslib.Timestamp variable

>>> from datetime import datetime
>>> for dt in pd.date_range('20160929', '20160930')[::-1]:
...     dt == pd.Timestamp(datetime(2016, 9, 29))
...
False
True

>>> type(pd.Timestamp(datetime(2016, 9, 29)))
<class 'pandas.tslib.Timestamp'>

>>> for dt in pd.date_range('20160929', '20160930')[::-1]:
...     type(dt)
...
<class 'pandas.tslib.Timestamp'>
<class 'pandas.tslib.Timestamp'>

tags | convert output from pandas data_range function to YYYYMMDD, date_range reverse order

Ref:-

Convert_YYYYMMDD_to_YYYY-MM-DD

get the beginning and end dates of a month

To get the first date

>>> from datetime import datetime
>>> a = datetime.strptime('20160905', '%Y%m%d')
>>> a
datetime.datetime(2016, 9, 5, 0, 0)
>>> b = a.replace(day=1)
>>> b
datetime.datetime(2016, 9, 1, 0, 0)
>>> a
datetime.datetime(2016, 9, 5, 0, 0)

To get the last date

>>> from datetime import datetime
>>> import calendar
>>> a = datetime.strptime('20160905', '%Y%m%d')
>>> a
datetime.datetime(2016, 9, 5, 0, 0)
>>> b = a.replace(day = calendar.monthrange(a.year, a.month)[1])
>>> b
datetime.datetime(2016, 9, 30, 0, 0)
>>> a
datetime.datetime(2016, 9, 5, 0, 0)

Ref:- search for datetime.replace in https://docs.python.org/2/library/datetime.html

add days to date

>>> from datetime import datetime, timedelta
>>> print(datetime.now())
2018-03-15 13:02:36.416505
>>> print(datetime.now() + timedelta(days=10))
2018-03-25 13:02:43.200587

diffdays

>>> d1 = '20180418'
>>> d2 = '20180602'
>>> from datetime import datetime
>>> fmt = '%Y%m%d'
>>> dt1 = datetime.strptime(d1, fmt)
>>> dt2 = datetime.strptime(d2, fmt)
>>> delta = dt2 - dt1
>>> print delta
45 days, 0:00:00
>>> print delta.days
45

Tips

configure vim

" To add the proper PEP8 indentation
au BufRead,BufNewFile *.py,*.pyw set tabstop=4
au BufRead,BufNewFile *.py,*.pyw set softtabstop=4
au BufRead,BufNewFile *.py,*.pyw set shiftwidth=4
au BufRead,BufNewFile *.py,*.pyw set textwidth=79
au BufRead,BufNewFile *.py,*.pyw set expandtab
au BufRead,BufNewFile *.py,*.pyw set autoindent
au BufRead,BufNewFile *.py,*.pyw set fileformat=unix

tested using vim 7.2

Ref:- https://realpython.com/blog/python/vim-and-python-a-match-made-in-heaven/

Dummy

Tutorials

The advantage of the pdf version is that all the content is in a single file. The advantage of html is that python code is shown with syntax highlighting.

Profiling Python

Install python on windows

documentation links

useful articles

  • http://nedbatchelder.com/text/names1.html - article on how names, values, assignment and mutability works in Python. The concepts are explained very well. Highly recommended for C/C++ programmers looking to learn Python.

set PYTHONPATH in windows

Use ';' as the separator. For example

PYTHONPATH="DIR1;DIR2;$PYTHONPATH" python foo.py --arg1 bar

So it may look

PYTHONPATH="C:\Temp\utils;$PYTHONPATH" python foo.py --file 'C:\Temp\input\bar.csv'

Run python from a windows script

call it foo.cmd

@echo off

:: Anaconda path
set ANACONDA_PATH=C:\Path\to\Continuum\Anaconda2

:: Report env variables.
:: set

set PATH=%ANACONDA_PATH%;%ANACONDA_PATH%\Scripts;%PATH%
:: Report the path
set PATH

call activate myenv
c:/Windows/System32/where.exe python
python --version
python -c "import pandas as pd; print(pd.__version__)"
call deactivate

try-except-else

Pseudo-code to understand the try-except-else-finally structure.

try:
    try_this(whatever) 
except SomeException as the_exception:
    handle_SomeException(the_exception)
    # Handle a instance of SomeException or a subclass of it.
except Exception as the_exception:
    generic_handle(the_exception)
    # Handle any other exception that inherits from Exception
    # - doesn't include GeneratorExit, KeyboardInterrupt, SystemExit
    # Avoid bare `except:`
else: # there was no exception whatsoever
    return something() 
    # if no exception, the "something()" gets evaluated, 
    # but the return will not be executed due to the return in the 
    # finally block below.
finally:
    # this block will execute no matter what, even if no exception,
    # after "something" is eval'd but before that value is returned
    # but even if there is an exception.
    # a return here will hijack the return functionality. e.g.:
    return True # hijacks the return in the else clause above

Ref:-


What is the difference between os.environ['foo'] vs os.getenv('foo')

If the variable 'foo' is not defined, os.environ['foo'] gives an exception but os.getenv('foo') returns an empty string.

If the variable is defined, both commands give its value.

$winpty python
Python 3.6.1 |Anaconda 4.4.0 (64-bit)| (default, May 11 2017, 13:25:24) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> var = 'TEMP'
>>> var in os.environ
True
>>> os.getenv(var)
'C:\\Windows\\TEMP'
>>> os.environ[var]
'C:\\Windows\\TEMP'
>>> os.getenv(var) is os.environ[var]
True
>>> var = 'blah'
>>> var in os.environ
False
>>> os.getenv(var)
>>> os.getenv(var) is None
True
>>> os.environ[var]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\ProgramData\Continuum\Anaconda\envs\py36\lib\os.py", line 669, in __getitem__
    raise KeyError(key) from None
KeyError: 'blah'

show an environment variable

import os
try:
    user_paths = os.environ['FOO'].split(os.pathsep)
except KeyError:
    user_paths = []

For example, to show the PYTHONPATH in the python interpreter

import os
os.environ['PYTHONPATH'].split(os.pathsep)

Note: The separator changes depending on the OS. For example, Linux uses ':' which Windows uses ';'.

check the python version

>>> import sys
>>> print(sys.version)
3.6.6 |Anaconda custom (64-bit)| (default, Jun 28 2018, 11:27:44) [MSC v.1900 64 bit (AMD64)]

Add a directory to the module search path

import sys
foo_dir = "C:\\path\\to\dir"
if foo_dir not in sys.path:
    sys.path.append(foo_dir)

demonstrates | import python module into a jupyter notebook

write either to file or to stdout

Approach 1:

import os
odir = "foo"
ofile = os.path.join(odir, "bar.txt")

with open(ofile, "w") as fh:
  print('writing', ofile)
  data_dumper(data, fh)

...

def data_dumper(data, fh)
   fh.write(data)
   ...

To print data onto stdout

import sys
data_dumper(data, sys.stdout)

See also: http://stackoverflow.com/questions/17602878/how-to-handle-both-with-open-and-sys-stdout-nicely

Approach 2:

def dump_foo(arg1, arg2, ..., param={})
    import sys
    # do something
  
    # dump to a file if some parameters are specified otherwise use stdout
    if (('out_dir' in param) & ('out_file' in param)):
        out_dir = param['out_dir']
        if not os.path.exists(out_dir):
            os.makedirs(out_dir, exist_ok=True)
        out_file = os.path.join(out_dir, param['out_file'])
        fh = open(out_file, "w")
        print('writing', out_file)
    else:
         fh = sys.stdout;

    fh.write("I am in dump_foo()\n")

    if fh is not sys.stdout:
        fh.close()

append to a file

This is useful when writing logs. For example

#! /usr/bin/env python3
import os
from datetime import datetime
a = datetime.now().strftime("%Y%m%d_%H%M%S")
out_dir = '/path/to/logdir'
if not os.path.exists(out_dir):
    os.makedirs(out_dir, exist_ok=True)
out_file_name = "log.txt"
out_file_path = os.path.join(out_dir, out_file_name)
with open(out_file_path, 'a') as fh:
    fh.write(a + "\n")

frequently used functions

Use case function Additional notes
get current directory cwd = os.getcwd()
change working directory os.chdir(path) For example os.chdir('C:\Temp\foo')
import modules from a different directory
  • sys.path.append('/path/to/dir') - directory searched last
  • sys.path.insert(0, '/path/to/dir') - directory searched first
Add parent directory to the module search path sys.path.append('..') Module Search Path
Get the PATH
  • os.getenv('PATH')
  • os.getenv('PATH').split(os.pathsep)

frequently used pdb commands

until NNN - continue until line NNN

script to plot a column of data from command line

See https://gitlab.com/d3k2mk7/rutils/blob/master/python3/plot_utils/plot_by_index.py

raw strings and normpath

a = ['test_cases\test1.csv', r'test_cases\test1.csv', 'test_cases/test1.csv', r'test_cases/test1.csv']
a
Out[86]: 
['test_cases\test1.csv',
 'test_cases\\test1.csv',
 'test_cases/test1.csv',
 'test_cases/test1.csv']

b = [os.path.normpath(p) for p in a]
b
Out[88]: 
['test_cases\test1.csv',
 'test_cases\\test1.csv',
 'test_cases\\test1.csv',
 'test_cases\\test1.csv']

c = [eval("r'%s'" %p) for p in a]
c
Out[94]: 
['test_cases\test1.csv',
 'test_cases\\test1.csv',
 'test_cases/test1.csv',
 'test_cases/test1.csv']

d = [p.encode('string_escape') for p in a]
d
Out[96]: 
['test_cases\\test1.csv',
 'test_cases\\\\test1.csv',
 'test_cases/test1.csv',
 'test_cases/test1.csv']

copying files

To copy a single file, you can use shutil.copyfile, shutil.copy(), shtuil.copy2()

syntax

shutil.copyfile(src, dst, *, follow_symlinks=True)
shutil.copy(src, dst, *, follow_symlinks=True)
shutil.copy2(src, dst*, follow_symlinks=True)

shtuil.copyfile()

destination will be overwritten if it already exists. dst has to be a file and cannot be a directory.

For example

shutil.copyfile('C:/Users/kkusuman/x/x1/junk1.txt', 'C:/Users/kkusuman/x/x1/junk2.txt')
shutil.copyfile('C:\\Users\\kkusuman\\x\\x1\\junk1.txt', 'C:\\Users\kkusuman\\x\\x1\\junk2.txt')

shtuil.copy()

dst can either be a file or a directory. It copies the data and file's permission mode. Other metadata such as file's creation and modification times are not preserved.

shtuil.copy2()

same as shutil.copy() except that it preserves all metadata.

Ref:- https://docs.python.org/3/library/shutil.html#shutil.copy

random links

useful links

Virtual Environments

The PyPI version of virtualenv works in most environments. As of Python 3.3, the venv virtual environment module is included as part of the standard library. However, some problems with venv have been reported on Ubuntu. Since virtualenv works with Python 3.6 (and as far back as Python 2.6) and on Ubuntu, use virtualenv.

Setup virtual environment

Using pip in Linux

pip install -U virtualenv
virtualenv -p /path/to/a/python.exe /path/to/env_name
source /path/to/env_name/bin/activate

# do your work
dectivate

Uisng Python in Linux

python3.6 -m pip install -U virtualenv
python3.6 -m virtualenv env_name
source env_name/bin/activate

# do your work
dectivate

Using pip in Windows

pip install -U virtualenv
virtualenv -p /path/to/a/python.exe /path/to/env_name
/path/to/env_name/Scripts/activate.bat

# do your work
dectivate

Using Python in Windows

python3.6 -m pip install -U virtualenv
python3.6 -m virtualenv env_name
env_name/Scripts/activate.bat

# do your work
dectivate

programming notes

script execution

To execute a python script in the interpreter

exec(open('myscript.py').read(), globals())

But this does not allow to pass any arguments to the script. For that use

import sys
import subprocess
subprocess.call([sys.executable, 'foo.py', arg1])

For windows, the filepath can be specified as 'C:/path/to/foo.py'

check if a file exists

import os
if not os.path.isfile(ofile):
    print("File", ofile, "does not exist")

Ref:- http://stackoverflow.com/questions/82831/how-do-i-check-whether-a-file-exists-using-python

Check if a list is empty

>>> a = []
>>> if (not a):
...     print("List is empty")
... 
List is empty

check if a key is exists in a dictionary

if 'key1' in dict:
  print "blah"
else:
  print "boo"

Ref: http://stackoverflow.com/questions/1602934/check-if-a-given-key-already-exists-in-a-dictionary

check if two values are not none

if all(v is not None for v in [A, B, C, D, E]):

check if something is not None

In:

a=None

if (a is not None):
    print("kama")
else:
    print("raju")
    

Out:

raju

In:

a=2

if (a is not None):
    print("kama")
else:
    print("raju")
    

Out:

kama

check if a list is sorted

 % cat is_list_sorted.py
def is_list_sorted(l):
    return all( [l[i] < l[i+1] for i in range(len(l)-1)] )

a = [10,1,8,2,5,3,7,9,6,4]
print(is_list_sorted(a))

b = sorted(a)
print(is_list_sorted(b))
 % python3 -u is_list_sorted.py
False
True

Ref:- https://stackoverflow.com/questions/3755136/pythonic-way-to-check-if-a-list-is-sorted-or-not

reload a module

For python 3

import importlib
importlib.reload(modulename)

For python 2

reload(foo)

Note:- if you are importing functions from a module using

from foo import *

then reload(foo) will throw a NameError.

NameError: name 'foo' is not defined

This happens because there is no object bound to foo. As a work around, I do

from foo import *
import foo
reload(foo)

See https://stackoverflow.com/questions/13455836/nameerror-when-using-reload for more details.

which python executable am I running?

import sys
print(sys.executable)

For example

$python
Python 3.6.1 |Anaconda 4.4.0 (64-bit)| (default, May 11 2017, 13:25:24) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> print(sys.executable)
C:\ProgramData\Continuum\Anaconda\envs\py36\python.exe

split string and trim white space

s = "blah, lots  ,  of ,  spaces, here "
[x.strip() for x in s.split(',')]

Out[18]: ['blah', 'lots', 'of', 'spaces', 'here']

Ref:- http://stackoverflow.com/questions/4071396/split-by-comma-and-strip-whitespace-in-python


convert two lists into a dictionary

Use dict with zip to construct a dictionary from two lists. The number of elements in dictionary will be the minimum of the number of elements in each list. The extra elements will be discarded.

 % python3          
Python 3.5.2+ (default, Sep 22 2016, 12:18:14) 
[GCC 6.2.0 20160914] on linux
Type "help", "copyright", "credits" or "license" for more information.

>>> a = [1, 2, 3]; b = ['a', 'b', 'c']; c = dict(zip(a, b)); print(a); print(b); print(c)
[1, 2, 3]
['a', 'b', 'c']
{1: 'a', 2: 'b', 3: 'c'}

>>> a = [1, 2, 3, 4]; b = ['a', 'b', 'c']; c = dict(zip(a, b)); print(a); print(b); print(c)
[1, 2, 3, 4]
['a', 'b', 'c']
{1: 'a', 2: 'b', 3: 'c'}

>>> a = [1, 2, 3]; b = ['a', 'b', 'c', 'd']; c = dict(zip(a, b)); print(a); print(b); print(c)
[1, 2, 3]
['a', 'b', 'c', 'd']
{1: 'a', 2: 'b', 3: 'c'}


debug from command line

To pass command line arguments and invoke the debugger directly

python -u -m pdb mymodule.py --arg1_key arg1_val --arg2_key arg2_val

To do it inside python interpreter

import pdb
import mymodule
pdb.run('mymodule.myfunc()')

Ref:-

tags | Debugging python programs from the command line

step through a python script

tags | debug Add the following lines at the beginning of the script

import pdb
pdb.set_trace()

Ref:- https://pythonconquerstheuniverse.wordpress.com/2009/09/10/debugging-in-python/

returning values

One way to return multiple values in a function is to use a dictionary.

return {'key1':value1,'key2':value2,'key3':value3}

or

result = {'key1':value1}
result['key2'] = value2
result['key3'] = value3
return(result)

Ref:-

if else if syntax

if x < 0:
    x = 0
    print('Negative changed to zero')
elif x == 0:
    print('Zero')
elif x == 1:
    print('Single')
else:
    print('More')

check for missing data

if (x is not None and x > 0):
    print('x is not none and positive')

logical operators

x or y, x and y, not x

Example

    if (a < 1 or a > 100):
        print('a is not in [1,100]')

Ref:- https://docs.python.org/3/library/stdtypes.html

timing a function

start_time = time.time()
call_function_of_interest()
end_time = time.time()
print('time elapsed = ', end_time - start_time)

For python2

print 'time elapsed = ', end_time - start_time
print 'took %.f sec to finish' %(end_time - start_time)

tags | time a function, tic toc

Remove duplicates

To remove duplicates from a column of a dataframe

unique_id = df['id'].drop_duplicates()

Ref:- http://pandas.pydata.org/pandas-docs/version/0.17.0/generated/pandas.Series.drop_duplicates.html

tags | unique

Replace newline characters with space

% python3
Python 3.4.3 (default, Mar  3 2015, 15:56:43)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-4)] on linux

>>> a = '''kama
... raju'''
>>> a
'kama\nraju'
>>> a.replace("\n", " ")
'kama raju'

overlay one column with another

Say you have two columns 'a' and 'b' in a dataframe. To copy the values from column 'a' to column 'b' whenever the values in column 'b' are missing.

df.loc[ pd.isnull(df['b']), 'b'] = \
  df.loc[ pd.isnull(df['b']), 'a']

tags | pandas update data based on a condition, indexing and selecting data, find and replace values in a python dataframe, replace values if null, assign values by index, search and replace data based on another column using python pandas, example on using loc, python pandas boolean array filter

specify the path of a file

To specify the path of a file relative to the location of the module

fname = os.path.join(os.path.dirname(__file__), 'foo.txt')

__file__ gives the path of current module.

show all variables in python interpreter

use dir()

>>> dir()
['__builtins__', '__doc__', '__loader__', '__name__', '__package__', '__spec__']
>>> import pandas as pd
>>> dir()
['__builtins__', '__doc__', '__loader__', '__name__', '__package__', '__spec__', 'pd']
>>> df = pd.DataFrame([[1, 2], [3, 4]], columns=list('AB'))
>>> dir()
['__builtins__', '__doc__', '__loader__', '__name__', '__package__', '__spec__', 'df', 'pd']
>>> df
   A  B
0  1  2
1  3  4

sql query string

sql_query = '''
select *
  from alpha..beta
 where foo in ('{foo}')
   and bar in ('{boo}')
'''
res = db_query(sql_query.format(foo = "', '".join(foo),
                                bar = "', '".join(bar))
              )

where db_query is a custom function to query the database.

connecting to databases

Using pyodbc to connect to MSSQL

    import pyodbc
    import pandas as pd
    # declare user, password, server, database
    conn_str = 'DRIVER={SQL Server};SERVER=' + server + ';DATABASE=' + database
    if user is not None and password is not None:
        conn_str += ';UID=' + user + ';PWD=' + password
    else:
        conn_str += ';Trusted_Connection=yes'
    connection = pyodbc.connect(conn_str)
    data = pd.io.sql.read_sql(<SQL goes here>, connection)

Using SQLAlchemy

from urllib import quote_plus as urlquote

import sqlalchemy as sqla
import pandas as pd

def create_engine(conn_str):
    url = sqla.engine.url.make_url(conn_str)
    if url.drivername == 'oracle+cx_oracle':
        return sqla.create_engine(url, arraysize=100000)
    else:
        return sqla.create_engine(url)

Then for MSSQL

# declare user, password, server, database
driver = 'ODBC Driver 11 for SQL Server'

if user is not None and password is not None:
    conn_str = "mssql+pyodbc://{}:{}@{}/{}?driver={}".format(
        user, urlquote(password), server, database, urlquote(driver)
    )
else:
    # use windows authentication
    conn_str = "mssql+pyodbc://{}/{}?driver={};trusted_connection=yes".format(
        server, database, urlquote(driver)
    )
engine = create_engine(conn_string)
data = pd.read_sql_query('<SQL goes here>', engine)

and for Oracle

# declare user, password, server
conn_str = "oracle+cx_oracle://{}:{}@{}".format(
    user, urlquote(password), server
)
engine = create_engine(conn_string)
data = pd.read_sql_query('<SQL goes here>', engine)

writing an if else condition in list comprehension

Syntax

[ a if C else b for i in items]

is equivalent to

for i in items:
  if (C):
    a
  else:
    b

For example, to figure out if a package is installed one can do

import pandas as pd
import apt
cache = apt.Cache()

a = pd.DataFrame({'package': ['python-apt', 'foo', 'dpkg', '0xffff']})
status = [
    True if ((pkg in cache) and cache[pkg].is_installed) else False
    for pkg in a['package']
]

print(status)

which will show

[True, False, True, False]

where python-apt, dpkg are installed on the system, '0xffff' is a valid package but not installed and 'foo' is not a valid package name.

Ref: http://stackoverflow.com/questions/4406389/if-else-in-a-list-comprehension

Stackoverflow links

get the username

>>> import getpass
>>> getpass.getuser()
'rajulocal'

exit with error

One way is to throw an exception. See http://stackoverflow.com/questions/2052390/manually-raising-throwing-an-exception-in-python for more details.

args and kwargs

args = arguments kwargs = keyword arguments

def f(*args, **kwargs):
  print('args: ', args, ' kwargs: ', kwargs)

>>> f('a')
args:  ('a',)  kwargs:  {}
>>> f(ar='a')
args:  ()  kwargs:  {'ar': 'a'}
>>> f(1,2,param=3)
args:  (1, 2)  kwargs:  {'param': 3}

>>> f(1,[2,3],param=3)
args:  (1, [2, 3])  kwargs:  {'param': 3}

>>> f(1, [2,3], [4,5,6], param=3, first='kama', last='raju')
args:  (1, [2, 3], [4, 5, 6])  kwargs:  {'param': 3, 'last': 'raju', 'first': 'kama'}

See also: http://stackoverflow.com/questions/682504/what-is-a-clean-pythonic-way-to-have-multiple-constructors-in-python?noredirect=1&lq=1


using kargs

Sample usage

def foo(alpha, beta, **kwargs):
    debug = kwargs.get('debug', False)

multiline statements

df = pd.read_csv(fname)\
.pipe(func_foo)\
.pipe(func_bar)

conditional operator

tags | conditional expressions, ternary

a if condition else b

calling python script from bash

print stack trace

To print stack trace from within the program

import traceback
print 'printing stack'
traceback.print_stack()

Ref:- https://docs.python.org/2/library/traceback.html

print exception and continue

from traceback import format_exc

try:
    # my code
except Exception as exc:
    print 'caught exception while doing foo'
    print format_exc()
    print exc

parse config files

Example 1:

 % cat config.ini
[COUNTRIES]
DENY = AG AL
ALLOW = IN US
 % cat config.py
import configparser
import os

Config = configparser.ConfigParser()
Config.read(os.path.join(os.path.dirname(__file__), 'config.ini'))

To use it

 % python3
Python 3.5.3 (default, Jan 19 2017, 14:11:04)
[GCC 6.3.0 20170118] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from config import Config
>>> Config['COUNTRIES']['DENY']
'AG AL'
>>> Config['COUNTRIES']['ALLOW']
'IN US'

What is the difference between <> and !=

In python 3, there is no <> operator.

In python 2, they both mean the same. != is preferred; <> is obsolescent.

Ref:-

Porting windows code to Linux

Things to look out for when porting python code from windows to Linux

  • use os.path.sep instead of "\\" for parsing directories in a filepath
  • use os.path.join when constructing file name. For example instead of dir_name + "\\" + file_name, use os.path.join(dir_name, file_name)
  • exception during reset or similar

Download a webpage

version 1:-

import requests


def get_html(url):
    # Get html from url
    response = requests.get(url)
    return response.text

Version 2:- Same as version 1, but has user_agent hardcoded

import requests


def get_html(url):
    # Get html from url
    user_agent = 'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.139 Safari/537.36'
    header = {'User-Agent': user_agent}
    response = requests.get(url, headers=header)
    return response.text

what is **{}

$python
Python 3.6.6 |Anaconda, Inc.| (default, Jun 28 2018, 11:27:44) [MSC v.1900 64 bit (AMD64)] on win32

>>> def foo(**bar):
...   print(bar)
...

>>> foo(**{'a':2})
{'a': 2}

>>> foo(**{'a':2, 'b':3})
{'a': 2, 'b': 3}

>>> foo()
{}

See:-


pep8 recommendations

  • use 4-space indentation, and no tabs
  • 79 characters per line
  • use CamelCase for classes and lower_case_with_underscores for functions and methods

See:-

small experiments

infinity

% python3
Python 3.5.3 (default, Jan 19 2017, 14:11:04) 
[GCC 6.3.0 20170118] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> float('inf')
inf
>>> type(float('inf'))
<class 'float'>
>>> float('inf') > 5
True
>>> float('inf') > -float('inf')
True
>>> float('inf') is float('inf')
False
>>> float('inf') == float('inf')
True

Troubleshooting

no module named error when importing from subdirectories

  • Make sure that each directory contains __init__.py

numpy

Frequently used numpy functions

use case solution
average np.average
sum np.sum

Challenges

Rotate matrix anti clockwise

Given a 2-D array

[[1,2,3,4],
 [5,6,7,8],
 [9,10,11,12]]

convert it to

[[4,8,12],
 [3,7,11],
 [2,6,10],
 [1,5,9]]
>>> a = [[1,2,3,4], [5,6,7,8], [9,10,11,12]]
>>> a
[[1, 2, 3, 4], [5, 6, 7, 8], [9, 10, 11, 12]]
>>> b = list(zip(*a))[::-1]
>>> b
[(4, 8, 12), (3, 7, 11), (2, 6, 10), (1, 5, 9)]
>>> c = [list(elem) for elem in b]
>>> c
[[4, 8, 12], [3, 7, 11], [2, 6, 10], [1, 5, 9]]

Rotate matrix clockwise

Given a 2-D array

[[1,2,3,4],
 [5,6,7,8],
 [9,10,11,12]]

convert it to

[[9,5,1],
 [10,6,2],
 [11,7,3],
 [12,8,4]]
>>> a = [[1,2,3,4], [5,6,7,8], [9,10,11,12]]
>>> a
[[1, 2, 3, 4], [5, 6, 7, 8], [9, 10, 11, 12]]
>>> b = list(zip(*a[::-1]))
>>> b
[(9, 5, 1), (10, 6, 2), (11, 7, 3), (12, 8, 4)]
>>> c = [list(elem) for elem in b]
>>> c
[[9, 5, 1], [10, 6, 2], [11, 7, 3], [12, 8, 4]]

index of an item in a sorted list

Use bisect module to find out where an item can be inserted in a sorted list. The return value of bisect is suitable for use as the first parameter to list.insert() assuming that original list is already sorted.

>>> a = [10,1,8,2,5,3,7,9,6,4]
>>> a
[10, 1, 8, 2, 5, 3, 7, 9, 6, 4]
>>> a.sort()
>>> a
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
>>> x = 5.5
>>> from bisect import bisect
>>> bisect(a,x)
5
>>> a.insert( bisect(a,x), x)
>>> a
[1, 2, 3, 4, 5, 5.5, 6, 7, 8, 9, 10]

Instead of doing bisect and insert, you can also insort which does both.

>>> a = [10,1,8,2,5,3,7,9,6,4]
>>> a
[10, 1, 8, 2, 5, 3, 7, 9, 6, 4]
>>> a.sort()
>>> a
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
>>> x = 5.5
>>> from bisect import insort
>>> insort(a,x)
>>> a
[1, 2, 3, 4, 5, 5.5, 6, 7, 8, 9, 10]

Ref:- https://docs.python.org/3/library/bisect.html