Python notes

From raju

Note:- This page will be migrated to http://www.kamaraju.xyz/dk/python_notes. Once the migration is done, it will be deleted.

Troubleshooting

no module named error when importing from subdirectories

  • Make sure that each directory contains __init__.py

ImportError: DLL load failed while importing win32api: The specified module could not be found.

Problem

    >>> import win32api
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    ImportError: DLL load failed while importing win32api: The specified module could not be found.
    

Solution: Install pywin32 package

Ref:- https://stackoverflow.com/questions/58612306/how-to-fix-importerror-dll-load-failed-while-importing-win32api

Utility functions

print command before executing it

command line arguments

Utility Scripts

Code Templates

Template code for regression testing a Python3 application

Check pep8 conformance via unittest

Functionality testing of python scripts

using printf

    indent = "  "
    printf("%s average = %.4g" %(indent, df['foo'].mean()) )
    printf("%s N = %d" %(indent, df.shape[0]) )
    

See also:-

expand %s in a string with variable values

    >>> a = '%s:micro soft, %s:google'
    >>> b = 'MSFT'; c = 'GOOG';
    >>> a
    '%s:micro soft, %s:google'
    >>> b
    'MSFT'
    >>> c
    'GOOG'
    >>> d = a % (b, c)
    >>> d
    'MSFT:micro soft, GOOG:google'
    


string construction

Use a tuple while constructing a string with multiple placeholders

    >>> 'select * from table_name where column in (%s,%s)' % ('val1','val2')
    'select * from table_name where column in (val1,val2)'
    

Using an array will give an error

    >>> 'select * from table_name where column in (%s,%s)' % ['val1','val2']
    Traceback (most recent call last):
    
      File "<ipython-input-66-xxxxxxxxxxxx>", line 1, in <module>
        'select * from table_name where column in (%s,%s)' % ['val1','val2']
    
    TypeError: not enough arguments for format string
    

pep8 write long strings

        user_agent = 'Mozilla/5.0 (Windows NT 6.1; Win64; x64) ' \
            'AppleWebKit/537.36 (KHTML, like Gecko) ' \
            'Chrome/66.0.3359.139 Safari/537.36'
    

Ref:- https://stackoverflow.com/questions/1874592/how-to-write-very-long-string-that-conforms-with-pep8-and-prevent-e501

break up long lines

Do

    if cond1 and \
       cond2:
    

Instead of

    if cond1 and cond2:
    

For formulas

    # Yes: easy to match operators with operands
    income = (gross_wages
              + taxable_interest
              + (dividends - qualified_dividends)
              - ira_deduction
              - student_loan_interest)
    

Ref:- https://www.python.org/dev/peps/pep-0008/#indentation

print objects

    class SimpleRepr(object):
        """
        A mixin implementing a simple __repr__.
        It gives the class name, the (shortened) id, and all of the attributes.
        """
    
        def __repr__(self):
            return "<{klass} @{id:x} {attrs}>".format(
                klass=self.__class__.__name__,
                id=id(self) & 0xFFFFFF,
                attrs=" ".join("{}={!r}".format(k, v) for k, v in self.__dict__.items()),
            )
    

Use it as follows

    class DataWrapper(SimpleRepr):
        def __init__(self, args):
            args_dict = vars(args)
            for (k,v) in args_dict.items():
                setattr(self, k, v)
    
    if __name__ == '__main__':
        # parse arguments using argparser
        data_wrapper = DataWrapper(parsed_args)
    

Ref:-

catching exception

    try:
        # Do something
    except Exception as e:
        self.logger.warnings("doing %s returned exception %s", str(foo), str(e))
    

raising exception

The raise statement causes an exception. To cause a LookupError, which is a built-in exception used to indicate that a lookup operation failed.

    def reverse_lookup(d, v):
        for k in d:
            if d[k] == v:
                return k
        raise LookupError()
    

The effect when you raise an exception is the same as when Python raises one: it prints a traceback and an error message.

When you raise an exception, you can provide a detailed error message as an optional argument. For example:

    raise LookupError('value does not appear in the dictionary')
    

logging

simple example

    import logging
    import sys
    
    logger = logging.getLogger(__name__)
    
    if __name__ == '__main__':
        logging.basicConfig(
            level=logging.INFO,
            format='%(asctime)s %(name)s %(levelname)s %(message)s')
        logger.info('python executable is %s', sys.executable)
    

logging variables

To log a string

    >>> s = 'foo'
    ... logger.info('s is %s' % s)
    
    2020-07-18 11:28:45,226 __main__ INFO s is foo
    

or

    >>> s = 'foo'
    ... logger.info('s is ' + s)
    
    2020-07-18 11:29:05,198 __main__ INFO s is foo
    

To log a boolean variable

    >>> a = True
    ... logger.info('a is %r' % a)
    
    2020-07-18 11:25:31,937 __main__ INFO a is True
    

To log multiple boolean variables

    >>> a = True
    ... b = False
    ... logger.info('a is %r' % a)
    ... logger.info('a and b are %r and %r respectively' % (a, b))
    
    2020-07-18 11:26:45,771 __main__ INFO a is True
    2020-07-18 11:26:45,771 __main__ INFO a and b are True and False respectively
    

list files

Recursively list all python files under a directory

list all directories except one

    $ ls
    bar/  bar.txt  baz/  baz.txt  foo/  foo.txt  script.py
    
    $ cat script.py
    import os
    root_dir = '.'
    dirs = [d for d in os.listdir(root_dir)
            if os.path.isdir(os.path.join(root_dir, d))
            and d != 'bar']
    print(dirs)
    
    $ python ./script.py
    ['baz', 'foo']
    

list directories

  • os.listdir(foo) - list all files and directories under foo. Not recursive. Just shows the top first level stuff.
  • List only directories
    all_dirs = [d for d in os.listdir(root_dir) if os.path.isdir(os.path.join(root_dir, d))]
    

list files in descending order of size

    def large_files_first(dirPath):
        allFiles = (os.path.join(dirPath, file) for file in os.listdir(dirPath))
        sortedFiles = sorted(allFiles, key=os.path.getsize, reverse=True)
        return sortedFiles
    

Sample run:

  • Create the files
    base64 /dev/urandom | head -c 1k > one.txt
    base64 /dev/urandom | head -c 2k > two.txt
    base64 /dev/urandom | head -c 3k > three.txt
    

Check their sizes

    $ du -b *.txt
    1024    one.txt
    3072    three.txt
    2048    two.txt
    

The default listing

    $ ipython
    
    In [1]: import os
       ...: dirPath = os.getcwd()
       ...: allFiles = [os.path.join(dirPath, file) for file in os.listdir(dirPath)]
       ...: allFiles
    Out[1]:
    ['C:\\Users\\kkusuman\\x\\x17\\one.txt',
     'C:\\Users\\kkusuman\\x\\x17\\three.txt',
     'C:\\Users\\kkusuman\\x\\x17\\two.txt']
    

Listing the big ones first

    In [1]: import os
       ...: def large_files_first(dirPath):
       ...:     allFiles = (os.path.join(dirPath, file) for file in os.listdir(dirPath))
       ...:     sortedFiles = sorted(allFiles, key=os.path.getsize, reverse=True)
       ...:     return sortedFiles
       ...:
       ...: dirPath = os.getcwd()
       ...: sortedFiles = large_files_first(dirPath)
       ...: sortedFiles
    Out[1]:
    ['C:\\Users\\kkusuman\\x\\x17\\three.txt',
     'C:\\Users\\kkusuman\\x\\x17\\two.txt',
     'C:\\Users\\kkusuman\\x\\x17\\one.txt']
    


String interpolation

print current function name

print('I am in {} function'.format(__name__))

Ref:- https://stackoverflow.com/questions/251464/how-to-get-a-function-name-as-a-string

listing all the variables in format()

tags | string interpolation using format()

    In [1]: a = 'kama'
       ...: b = 'raju'
       ...: c = "foo_{alpha}_bar_{beta}.txt".format(alpha=a, beta=b)
       ...: print([a, b, c])
    ['kama', 'raju', 'foo_kama_bar_raju.txt']
    

In the format function, you have to specify the variable names and its values. Otherwise it gives a KeyError.

    In [2]: c = "foo_{alpha}_bar_{beta}.txt".format(a, b)
    ---------------------------------------------------------------------------
    KeyError                                  Traceback (most recent call last)
    <ipython-input-2-34f487dabb30> in <module>()
    ----> 1 c = "foo_{alpha}_bar_{beta}.txt".format(a, b)
    
    KeyError: 'alpha'
    

using dictionary in format()

    In [1]: d = {'alpha': 'kama', 'beta': 'raju'}
       ...: d
    Out[1]: {'alpha': 'kama', 'beta': 'raju'}
    
    In [2]: c = "foo_{alpha}_bar_{beta}.txt".format(**d)
       ...: c
    Out[2]: 'foo_kama_bar_raju.txt'
    


using locals() in format()

def buildString(user, name = 'john', age=22):
    userId = user.getUserId()
    return "Name: {name}, age: {age}, userid:{userId}".format(**locals())

Tested it using Python 3.4.3

Advantage:

  • This is better than
    return "Name: {name}, age: {age}, userid:{userId}".format(name=name, age=age, userId=userId)

which is ugly.

Disadvantage:

  • Do not use it when the format string is user supplied since then it would open up access to every local variable making the application susceptible to injection attacks.

Ref:-

% vs. .format()

.format() is recommended over %. .format() was introduced in Python 2.6.

Ref:- https://stackoverflow.com/questions/5082452/string-formatting-vs-format

formatting related

Tips

configure vim

" To add the proper PEP8 indentation
au BufRead,BufNewFile *.py,*.pyw set tabstop=4
au BufRead,BufNewFile *.py,*.pyw set softtabstop=4
au BufRead,BufNewFile *.py,*.pyw set shiftwidth=4
au BufRead,BufNewFile *.py,*.pyw set textwidth=79
au BufRead,BufNewFile *.py,*.pyw set expandtab
au BufRead,BufNewFile *.py,*.pyw set autoindent
au BufRead,BufNewFile *.py,*.pyw set fileformat=unix

tested using vim 7.2

Ref:- https://realpython.com/blog/python/vim-and-python-a-match-made-in-heaven/

Copying files

recursively copy files and overwrite if necessary

tags | copying files, copytree overwrite

tags | preserve timestamp when copying files

Use distutils.dir_util.copytree

Sample code snippet

    from distutils.dir_util import copy_tree
    if os.path.isdir(src):
        copy_tree(src, dst)
    

This will recursively copy whatever is in src to dst (ex:- src/foo to dst/foo, src/foo/bar/baz.txt to dst/foo/bar/baz.txt etc.,)

If same file exists in both, the one in src overwrites the one in dst.

If a file exists only in dst, but not in src, it is not touched.

Notes:

  • src has to be a directory
  • dst will be created if it does not already exist
  • all child directories under dst will be created as needed
  • dst can be an existing directory but not an existing file
  • timestamps are preserved by default

Ref:-

Why not shutil.copytree?

  • shutil.copytree throws an error if dst directory already exists.
    WindowsError: [Error 183] Cannot create a file when that file already exists: '/path/to/dst'
    

distutils.dir_util.copy_tree does not care if that is the case. It will simply overwrite the files inside it.

copying files

To copy a single file, you can use shutil.copyfile, shutil.copy(), shtuil.copy2()

syntax

    shutil.copyfile(src, dst, *, follow_symlinks=True)
    shutil.copy(src, dst, *, follow_symlinks=True)
    shutil.copy2(src, dst, *, follow_symlinks=True)
    

shtuil.copyfile()

destination will be overwritten if it already exists. dst has to be a file and cannot be a directory.

For example

    shutil.copyfile('C:/Users/kkusuman/x/x1/junk1.txt', 'C:/Users/kkusuman/x/x1/junk2.txt')
    shutil.copyfile('C:\\Users\\kkusuman\\x\\x1\\junk1.txt', 'C:\\Users\kkusuman\\x\\x1\\junk2.txt')
    

shtuil.copy()

dst can either be a file or a directory. It copies the data and file's permission mode. Other metadata such as file's creation and modification times are not preserved.

shtuil.copy2()

same as shutil.copy() except that it preserves all metadata.

Ref:- https://docs.python.org/3/library/shutil.html#shutil.copy

Delete files

deleting files

    def remove_file(fname):
        if os.path.isfile(fname):
            try:
                os.remove(fname)
            except Exception as exc:
                print exc
                pass
    

delete directory recursively

    import os
    from distutils.dir_util import remove_tree
    
    if os.path.isdir(fname):
        remove_tree('/path/to/foo')
    


disttuils.dir_util.remove_tree works only on a directory and the directory should exist. Otherwise, it will throw an exception.

If you try to delete a file using remove_tree, it will throw an exception

    NotADirectoryError: [WinError 267] The directory name is invalid: 'file.txt'
    

writing files

write a list of strings to a file

Get the input

    input_file = 'foo.txt'
    with open(input_file) as fh:
        # read first three lines into a list
        header = [fh.readline() for line in range(3)]
    

write it

    output_file = 'bar.txt'
    with open(output_file) as fh:
        for line in header:
            fh.write('%s' % line)
    

tags | array of strings

Dummy

String for SQL where clause

write either to file or to stdout

Approach 1: cat_files() shows how to write binary data, cat_files_with_same_header() shows how write normal data. In both cases, output is written to dest if it is a file path. If it is None or sys.stdout, output is written to stdout.

Tested on | Python 3.8.2

    def cat_files(dest, sources):
        """Cat multiple files into dest."""
        if dest is None or dest is sys.stdout:
            # https://bugs.python.org/issue4571 says to use sys.stdout.buffer
            # when writing binary data to stdout.
            fdst = sys.stdout.buffer
        else:
            fdst = open(dest, 'wb')
    
        for src in sources:
            with open(src, 'rb') as fsrc:
                shutil.copyfileobj(fsrc, fdst)
    
        if fdst is not sys.stdout.buffer:
            fdst.close()
    
    
    def cat_files_with_same_header(dest, sources):
        """Cat multiple files into dest. Ignore headers in all the files except the first."""
        if dest is None or dest is sys.stdout:
            fdst = sys.stdout
        else:
            fdst = open(dest, 'w')
    
        first_src = sources.pop(0)
        with open(first_src, 'r') as fsrc:
            header = fsrc.readline()
            fdst.write(header)
            shutil.copyfileobj(fsrc, fdst)
    
        for src in sources:
            with open(src, 'r') as fsrc:
                # read but ignore the header
                header = fsrc.readline()
                shutil.copyfileobj(fsrc, fdst)
    
        if fdst is not sys.stdout:
            fdst.close()
    

Sample calls

    cat_files(dest_file, source_files)
    cat_files(None, source_files)
    cat_files(sys.stdout, source_files)
    cat_files_with_same_header(dest_file, source_file)
    cat_files_with_same_header(None, source_file)
    cat_files_with_same_header(sys.stdout, source_file)
    

Ref:- https://github.com/KamarajuKusumanchi/rutils/blob/master/python3/cat_files.py - latest version of the functions.


Approach 2:

/* 2020-06-13: Delete this approach if you are happy with the first approach. */

    import os
    odir = "foo"
    ofile = os.path.join(odir, "bar.txt")
    
    with open(ofile, "w") as fh:
      print('writing', ofile)
      data_dumper(data, fh)
    
    ...
    
    def data_dumper(data, fh)
       fh.write(data)
       ...
    

To print data onto stdout

    import sys
    data_dumper(data, sys.stdout)
    

See also: http://stackoverflow.com/questions/17602878/how-to-handle-both-with-open-and-sys-stdout-nicely

Approach 3:

/* 2020-06-13: Delete this approach if you are happy with the first approach. */

    def dump_foo(arg1, arg2, ..., param={})
        import sys
        # do something
      
        # dump to a file if some parameters are specified otherwise use stdout
        if (('out_dir' in param) & ('out_file' in param)):
            out_dir = param['out_dir']
            if not os.path.exists(out_dir):
                os.makedirs(out_dir, exist_ok=True)
            out_file = os.path.join(out_dir, param['out_file'])
            fh = open(out_file, "w")
            print('writing', out_file)
        else:
             fh = sys.stdout;
    
        fh.write("I am in dump_foo()\n")
    
        if fh is not sys.stdout:
            fh.close()
    

list of alphabets

    from string import ascii_letters
    print(ascii_letters)
    
    abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ
    

Ref:- https://docs.python.org/3/library/string.html

Tutorials

The advantage of the pdf version is that all the content is in a single file. The advantage of html is that python code is shown with syntax highlighting.

Profiling Python

Install python on windows

documentation links

range like functions

random integer in [a, b]
randint(a, b)
randrange(a, b+1)
To generate random numbers between 0 and N (including both)
randint(0, N)

range(N) => 0 through N-1; count = N

set PYTHONPATH in windows

Use ';' as the separator. For example

    PYTHONPATH="DIR1;DIR2;$PYTHONPATH" python foo.py --arg1 bar
    

So it may look

    PYTHONPATH="C:\Temp\utils;$PYTHONPATH" python foo.py --file 'C:\Temp\input\bar.csv'
    

Run python from a windows script

call it foo.cmd

    @echo off
    
    :: Anaconda path
    set ANACONDA_PATH=C:\Path\to\Continuum\Anaconda2
    
    :: Report env variables.
    :: set
    
    set PATH=%ANACONDA_PATH%;%ANACONDA_PATH%\Scripts;%PATH%
    :: Report the path
    set PATH
    
    call activate myenv
    c:/Windows/System32/where.exe python
    python --version
    python -c "import pandas as pd; print(pd.__version__)"
    call deactivate
    

try-except-else

Pseudo-code to understand the try-except-else-finally structure.

    try:
        try_this(whatever) 
    except SomeException as the_exception:
        handle_SomeException(the_exception)
        # Handle a instance of SomeException or a subclass of it.
    except Exception as the_exception:
        generic_handle(the_exception)
        # Handle any other exception that inherits from Exception
        # - doesn't include GeneratorExit, KeyboardInterrupt, SystemExit
        # Avoid bare `except:`
    else: # there was no exception whatsoever
        return something() 
        # if no exception, the "something()" gets evaluated, 
        # but the return will not be executed due to the return in the 
        # finally block below.
    finally:
        # this block will execute no matter what, even if no exception,
        # after "something" is eval'd but before that value is returned
        # but even if there is an exception.
        # a return here will hijack the return functionality. e.g.:
        return True # hijacks the return in the else clause above
    

Ref:-

os.path.isfile vs os.path.exists

  • os.path.isfile - is it an existing regular file?
  • os.path.exist - does it exist?

os.path.isfile also checks for existence. So no need to use both os.path.isfile and os.path.exists.

what is the difference between [] and list()

    In [1]: a = ['abc', ' A', ' B', 'C']
       ...: a[0]
    Out[1]: 'abc'
    
    In [2]: [a[0]]
    Out[2]: ['abc']
    
    In [3]: list(a[0])
    Out[3]: ['a', 'b', 'c']
    
    In [4]: [a[0]] * 2
    Out[4]: ['abc', 'abc']
    
    In [5]: list(a[0]) * 2
    Out[5]: ['a', 'b', 'c', 'a', 'b', 'c']
    

What is the difference between os.environ['foo'] vs os.getenv('foo')

If the variable 'foo' is not defined, os.environ['foo'] gives an exception but os.getenv('foo') returns an empty string.

If the variable is defined, both commands give its value.

    $winpty python
    Python 3.6.1 |Anaconda 4.4.0 (64-bit)| (default, May 11 2017, 13:25:24) [MSC v.1900 64 bit (AMD64)] on win32
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import os
    >>> var = 'TEMP'
    >>> var in os.environ
    True
    >>> os.getenv(var)
    'C:\\Windows\\TEMP'
    >>> os.environ[var]
    'C:\\Windows\\TEMP'
    >>> os.getenv(var) is os.environ[var]
    True
    >>> var = 'blah'
    >>> var in os.environ
    False
    >>> os.getenv(var)
    >>> os.getenv(var) is None
    True
    >>> os.environ[var]
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "C:\ProgramData\Continuum\Anaconda\envs\py36\lib\os.py", line 669, in __getitem__
        raise KeyError(key) from None
    KeyError: 'blah'
    

rename directories

tags | rename directory, move a directory only if it exists, apply a function to each row in a pandas dataframe

    to_rename = pd.DataFrame({
        'src': ['/path/to/src1', '/path/to/src2', '/path/to/src3', '/path/to/src4'],
        'dst': ['/path/to/dst1', '/path/to/dst2', '/path/to/dst3', '/path/to/dst4']
    })
    to_rename.apply(lambda row: os.rename(row['src'], row['dst']) if os.path.exists(row['src']) else None, axis=1)
    

show an environment variable

    import os
    try:
        user_paths = os.environ['FOO'].split(os.pathsep)
    except KeyError:
        user_paths = []
    

For example, to show the PYTHONPATH in the python interpreter

    import os
    os.environ['PYTHONPATH'].split(os.pathsep)
    

Note: The separator changes depending on the OS. For example, Linux uses ':' which Windows uses ';'.

check the python version

    >>> import sys
    >>> print(sys.version)
    3.6.6 |Anaconda custom (64-bit)| (default, Jun 28 2018, 11:27:44) [MSC v.1900 64 bit (AMD64)]
    

Add a directory to the module search path

    import sys
    foo_dir = "C:\\path\\to\dir"
    if foo_dir not in sys.path:
        sys.path.append(foo_dir)
    

demonstrates | import python module into a jupyter notebook

append to a file

This is useful when writing logs. For example

    #! /usr/bin/env python3
    import os
    from datetime import datetime
    a = datetime.now().strftime("%Y%m%d_%H%M%S")
    out_dir = '/path/to/logdir'
    if not os.path.exists(out_dir):
        os.makedirs(out_dir, exist_ok=True)
    out_file_name = "log.txt"
    out_file_path = os.path.join(out_dir, out_file_name)
    with open(out_file_path, 'a') as fh:
        fh.write(a + "\n")
    

frequently used functions

Use case function Additional notes
get current directory cwd = os.getcwd()
change working directory os.chdir(path) For example os.chdir('C:\Temp\foo')
import modules from a different directory
  • sys.path.append('/path/to/dir') - directory searched last
  • sys.path.insert(0, '/path/to/dir') - directory searched first
Add parent directory to the module search path sys.path.append('..') Module Search Path
Get the PATH
  • os.getenv('PATH')
  • os.getenv('PATH').split(os.pathsep)
get basename os.path.basename(path) path is a string. It cannot be None.

frequently used pdb commands

    until NNN - continue until line NNN
    

script to plot a column of data from command line

See https://gitlab.com/d3k2mk7/rutils/blob/master/python3/plot_utils/plot_by_index.py

raw strings and normpath

    a = ['test_cases\test1.csv', r'test_cases\test1.csv', 'test_cases/test1.csv', r'test_cases/test1.csv']
    a
    Out[86]: 
    ['test_cases\test1.csv',
     'test_cases\\test1.csv',
     'test_cases/test1.csv',
     'test_cases/test1.csv']
    
    b = [os.path.normpath(p) for p in a]
    b
    Out[88]: 
    ['test_cases\test1.csv',
     'test_cases\\test1.csv',
     'test_cases\\test1.csv',
     'test_cases\\test1.csv']
    
    c = [eval("r'%s'" %p) for p in a]
    c
    Out[94]: 
    ['test_cases\test1.csv',
     'test_cases\\test1.csv',
     'test_cases/test1.csv',
     'test_cases/test1.csv']
    
    d = [p.encode('string_escape') for p in a]
    d
    Out[96]: 
    ['test_cases\\test1.csv',
     'test_cases\\\\test1.csv',
     'test_cases/test1.csv',
     'test_cases/test1.csv']
    

random links

visualization

The image can also be accessed via https://static1.squarespace.com/static/50363cf324ac8e905e7df861/t/5b86d8a7562fa768a6edfec8/1535563952409/Screen+Shot+2018-08-29+at+10.30.01+AM.png?format=750w

using sort

Using Python 3.6.6

    >>> x=["e","a","é","f"]
    >>> x.sort()
    >>> x
    ['a', 'e', 'f', 'é']
    
    >>> x=["a","A","b","B"]
    >>> x.sort()
    >>> x
    ['A', 'B', 'a', 'b']
    

using map

    >>> ','.join(map(str,[1,2,3]))
    '1,2,3'
    

getsizeof pitfalls

sys.getsizeof() does not add up the size of data that is referenced. For example

    In [1]: import sys
       ...: a = [1]*1000
       ...: b = [2]*1000000
       ...: c = {}; c['a'] = a
       ...: d = {}; d['a'] = a; d['b'] = b
    
    In [2]: [sys.getsizeof(a), sys.getsizeof(b), sys.getsizeof(c), sys.getsizeof(d)]
    Out[2]: [8064, 8000064, 240, 240]
    

When counting the size of c and d, the size of a, b are not included since the dictionaries contain those values via references. That is why we get the same size for c and d.

n choose r

tags | nCr

    In [1]: from scipy.special import comb
       ...: comb(12,3)
    Out[1]: 220.0
    

print a floating point number as dollars and cents

This prints

  • a '$' before a positive number and '-$' before a negative number
  • rounds it to two decimal points
  • adds comma as a thousands separator
    "{}${:,.2f}".format(["","-"][amount<0], abs(amount))
    

For example

    >>> amount = float('214498.63723333334')
    >>> "{}${:,.2f}".format(["","-"][amount<0], abs(amount))
    '$214,498.64'
    
    >>> amount = float('-214498.63723333334')
    >>> "{}${:,.2f}".format(["","-"][amount<0], abs(amount))
    '-$214,498.64'
    

If the '$' symbol is not needed

    "{:,.2f}".format(amount)
    
    >>> amount = float('214498.63723333334')
    >>> "{:,.2f}".format(amount)
    '214,498.64'
    
    >>> amount = float('-214498.63723333334')
    >>> "{:,.2f}".format(amount)
    '-214,498.64'
    

If neither the '$' symbol nor the thousands separator are needed

    "{:.2f}".format(amount)
    
    >>> amount = float('214498.63723333334')
    >>> "{:.2f}".format(amount)
    '214498.64'
    
    >>> amount = float('-214498.63723333334')
    >>> "{:.2f}".format(amount)
    '-214498.64'
    

Ref:- https://stackoverflow.com/questions/21208376/converting-float-to-dollars-and-cents

floating point pitfalls

    $ ipython
    In [1]: x = 651370000000
       ...: x / 3.1416 * 3.1416
    Out[1]: 651370000000.0
    
    In [2]: invd = 1 / 3.1416
       ...: x * invd * 3.1416
    Out[2]: 651370000000.0001
    

Ref:- https://lemire.me/blog/2019/03/12/multiplying-by-the-inverse-is-not-the-same-as-the-division/

latest python release

Ref:- https://www.python.org/downloads/

  • Python 3.7.4 released 2019-07-08 (checked on 2019-09-16)
  • Python 3.6.9 released 2019-07-02 (checked on 2019-09-16)
  • Python 2.7.16 released 2019-03-04 (checked on 2019-09-16)

useful articles

useful links

radix 2 representation of a number

    format(n, 'b')
    
    >>> format(13, 'b')
    '1101'
    >>> type(format(13, 'b'))
    <class 'str'>
    

tags | convert an integer to binary format

To do it the other way (convert a binary string to integer)

    int(s, 2)
    
    >>> int('1101', 2)
    13
    

notes on sys.argv

  • argv[0] - script name
  • len(argv) = number of arguments.

For example, if you call a python script as

    foo.py arg1 arg2
    

len(argv) will be 3. argv[0] = foo.py, argv[1] = arg1, argv[2] = arg2.

hash string to integer

tags | deterministic hashing, cut length of hash to N digits

Starting from Python 3.3, hash randomization is turned on by default as a security feature. A random seed is set when python process is started. As a result, the python hash() will return different values on each run. For example

    $ python -c "print(hash('Coffee'))"
    -6802492772937910521
    
    $ python -c "print(hash('Coffee'))"
    2506337549030620102
    
    $ python --version
    Python 3.6.5 :: Anaconda, Inc.
    

To do it in a deterministic way:

    from hashlib import md5
    
    def get_hash(s, length=None):
        h = md5(str(s).encode('utf-8')).hexdigest()
        if length:
            h = h[:length]
        res = int(h, base=16)
        return res
    

Sample usage:

    In [2]: get_hash('Coffee')
    Out[2]: 79804158907860816670176857872944973754
    
    In [3]: get_hash('Coffee', length=10)
    Out[3]: 257861144725
    
    In [4]: get_hash('Coffee', length=5)
    Out[4]: 245915
    

Ref:-

https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.util.hash_array.html

  • pandas.util.hash_pandas_object - Return a data hash of the Index/Series/DataFrame

https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.util.hash_pandas_object.html

Tasks

first common element

tags | find the first element common to two lists

Q: Given two lists x, y find the first element in x that is present in y?

A: This does it in O(N) time with a set (hash map)

next((i for i in x if i in set_y), None)

Sample run:

    $ ipython
    
    In [1]: x = [1,2,3,4]
       ...: y = [2,5,6,7]
       ...: set_y = set(y)
       ...: c = next((i for i in x if i in set_y), None)
       ...: print(c)
    2
    
    In [2]: x = [1,3,4,5]
       ...: c = next((i for i in x if i in set_y), None)
       ...: print(c)
    5
    
    In [3]: x = [1,3,4,8]
       ...: c = next((i for i in x if i in set_y), None)
       ...: print(c)
    None
    

Ref:-

get file extension

    os.path.splitext(x)[1]
    

Experiments:

    $ ipython
    Python 3.6.5 |Anaconda, Inc.| (default, Mar 29 2018, 13:32:41) [MSC v.1900 64 bit (AMD64)]
    Type 'copyright', 'credits' or 'license' for more information
    IPython 6.4.0 -- An enhanced Interactive Python. Type '?' for help.
    
    In [1]: import os
       ...: files = ['foo.pdf', 'foo.', 'foo',
       ...:          '/dir/foo.pdf', '/dir/foo.', '/dir/foo',
       ...:          '/dir.fun/foo.pdf', '/dir.fun/foo.', '/dir.fun/foo',
       ...:          '.foo', '.foo.bar']
       ...: roots = [os.path.splitext(x)[0] for x in files]
       ...: exts = [os.path.splitext(x)[1] for x in files]
       ...: for i in zip(files, roots, exts):
       ...:     print(i)
       ...:
    ('foo.pdf', 'foo', '.pdf')
    ('foo.', 'foo', '.')
    ('foo', 'foo', '')
    ('/dir/foo.pdf', '/dir/foo', '.pdf')
    ('/dir/foo.', '/dir/foo', '.')
    ('/dir/foo', '/dir/foo', '')
    ('/dir.fun/foo.pdf', '/dir.fun/foo', '.pdf')
    ('/dir.fun/foo.', '/dir.fun/foo', '.')
    ('/dir.fun/foo', '/dir.fun/foo', '')
    ('.foo', '.foo', '')
    ('.foo.bar', '.foo', '.bar')
    

Ref:- https://docs.python.org/3/library/os.path.html#os.path.splitext


which package

Read and write parquet files

pyarrow

Links

Notes:

  • No support for avro.

Read and write avro files

fastavro

Links

Notes:

  • faster

Virtual Environments

The PyPI version of virtualenv works in most environments. As of Python 3.3, the venv virtual environment module is included as part of the standard library. However, some problems with venv have been reported on Ubuntu. Since virtualenv works with Python 3.6 (and as far back as Python 2.6) and on Ubuntu, use virtualenv.

Setup virtual environment

Using pip in Linux

    pip install -U virtualenv
    virtualenv -p /path/to/a/python.exe /path/to/env_name
    source /path/to/env_name/bin/activate
    
    # do your work
    dectivate
    

Uisng Python in Linux

    python3.6 -m pip install -U virtualenv
    python3.6 -m virtualenv env_name
    source env_name/bin/activate
    
    # do your work
    dectivate
    

Using pip in Windows

    pip install -U virtualenv
    virtualenv -p /path/to/a/python.exe /path/to/env_name
    /path/to/env_name/Scripts/activate.bat
    
    # do your work
    dectivate
    

Using Python in Windows

    python3.6 -m pip install -U virtualenv
    python3.6 -m virtualenv env_name
    env_name/Scripts/activate.bat
    
    # do your work
    dectivate
    

check if

check if a file exists

    import os
    if not os.path.isfile(ofile):
        print("File", ofile, "does not exist")
    

Ref:- http://stackoverflow.com/questions/82831/how-do-i-check-whether-a-file-exists-using-python

check if a key is exists in a dictionary

    if 'key1' in dict:
      print "blah"
    else:
      print "boo"
    

Ref: http://stackoverflow.com/questions/1602934/check-if-a-given-key-already-exists-in-a-dictionary

check if two values are not none

    if all(v is not None for v in [A, B, C, D, E]):
    

check if something is not None

In:

    a = None
    if a is not None:
        print('kama')
    else:
        print('raju')
    

Out:

    raju
    

In:

    a = 2
    if a is not None:
        print('kama')
    else:
        print('raju')
    

Out:

    kama
    

check if a list is sorted

     % cat is_list_sorted.py
    def is_list_sorted(l):
        return all( [l[i] < l[i+1] for i in range(len(l)-1)] )
    
    a = [10,1,8,2,5,3,7,9,6,4]
    print(is_list_sorted(a))
    
    b = sorted(a)
    print(is_list_sorted(b))
    
     % python3 -u is_list_sorted.py
    False
    True
    

Ref:- https://stackoverflow.com/questions/3755136/pythonic-way-to-check-if-a-list-is-sorted-or-not

check if a string can be converted to float

    def is_float_by_except(s):
        try:
            float(s)
            return True
        except ValueError:
            return False
    

check if a rest server is accessible

    import requests
    
    def test_rest_server(url):
        # url will be of the form
        # http://host.name:port
        res = requests.get(url)
        res.raise_for_status()    # Raises a HTTPError if the status is 4xx, 5xx
    

See also:-

how if condition evaluates it

tags | check if something evaluates to true or false, check if something is none

Takeaway:

    None, False, 0, '', [] are all evaluated as False in an if condition.
    

Experiment:

    def is_true(x):
        if x:
            return True
        else:
            return False
        
    def is_none(x):
        if x is None:
            return True
        else:
            return False
        
    def is_false(x):
        if not x:
            return True
        else:
            return False
        
    def is_not_none(x):
        if x is not None:
            return True
        else:
            return False
    
    a = [None, False, 0, '', [], True, ' ', 1, ['']]
    print('{!s:5}'.format('elem'),
          '{!s:7}'.format('is_true'), '{!s:7}'.format('is_none'),
          '{!s:8}'.format('is_false'), '{!s:11}'.format('is_not_none'),
          )
    for x in a:
        print('{!r:5}'.format(x),
              '{!r:7}'.format(is_true(x)), '{!r:7}'.format(is_none(x)),
              '{!r:8}'.format(is_false(x)), '{!r:11}'.format(is_not_none(x))
             )
    

Output:

    elem  is_true is_none is_false is_not_none
    None  False   True    True     False      
    False False   False   True     True       
    0     False   False   True     True       
    ''    False   False   True     True       
    []    False   False   True     True       
    True  True    False   False    True       
    ' '   True    False   False    True       
    1     True    False   False    True       
    ['']  True    False   False    True       
    

Code: https://github.com/KamarajuKusumanchi/notebooks/blob/master/python/how%20if%20condition%20evaluates%20it.ipynb

programming notes

string formatting in print statements

tags | specify width of a string

Code snippet:

    a = [None, False, 0, '', [], True, ' ', 1, ['']]
    print('{!s:5}'.format('elem'),
          '{!s:7}'.format('is_true'), '{!s:7}'.format('is_none'),
          '{!s:8}'.format('is_false'), '{!s:11}'.format('is_not_none'),
          )
    for x in a:
        print('{!r:5}'.format(x),
              '{!r:7}'.format(is_true(x)), '{!r:7}'.format(is_none(x)),
              '{!r:8}'.format(is_false(x)), '{!r:11}'.format(is_not_none(x))
             )
    

Output:

    elem  is_true is_none is_false is_not_none
    None  False   True    True     False      
    False False   False   True     True       
    0     False   False   True     True       
    ''    False   False   True     True       
    []    False   False   True     True       
    True  True    False   False    True       
    ' '   True    False   False    True       
    1     True    False   False    True       
    ['']  True    False   False    True       
    

where the functions is_true, is_none etc., are defined in https://github.com/KamarajuKusumanchi/notebooks/blob/master/python/how%20if%20condition%20evaluates%20it.ipynb

string formatting in assert statements

    In [1]: assert '10' == 10, 'not equal {} blah {}'.format(10, 1)
    ---------------------------------------------------------------------------
    AssertionError                            Traceback (most recent call last)
    <ipython-input-1-31a4cefc7818> in <module>()
    ----> 1 assert '10' == 10, 'not equal {} blah {}'.format(10, 1)
    
    AssertionError: not equal 10 blah 1
    
    In [2]: assert '10' == 10, 'not equal {} blah {}'.format(10, '100')
    ---------------------------------------------------------------------------
    AssertionError                            Traceback (most recent call last)
    <ipython-input-2-8bee5a255efa> in <module>()
    ----> 1 assert '10' == 10, 'not equal {} blah {}'.format(10, '100')
    
    AssertionError: not equal 10 blah 100
    
    In [3]: assert '10' == 10, 'not equal %d blah %s' % (10, '100')
    ---------------------------------------------------------------------------
    AssertionError                            Traceback (most recent call last)
    <ipython-input-3-fa41c2fc7bb1> in <module>()
    ----> 1 assert '10' == 10, 'not equal %d blah %s' % (10, '100')
    
    AssertionError: not equal 10 blah 100
    

See also:- https://www.geeksforgeeks.org/python-format-function/

pretty format list comprehension statements

    import pandas as pd
    week_days = [dt.strftime('%Y%m%d') for dt
                 in pd.date_range(end='20200101', periods=8)
                 if dt.weekday() < 5]
    print(week_days)
    

Ref:- https://stackoverflow.com/questions/311588/how-to-indent-python-list-comprehensions

script execution

To execute a python script in the interpreter

    exec(open('myscript.py').read(), globals())
    

But this does not allow to pass any arguments to the script. For that use

    import sys
    import subprocess
    subprocess.call([sys.executable, 'foo.py', arg1])
    

For windows, the filepath can be specified as 'C:/path/to/foo.py'

lambda if

Ref:- https://stackoverflow.com/questions/1585322/is-there-a-way-to-perform-if-in-pythons-lambda

Task:- Copy files in a dataframe column to a destination folder.

We need to check if the file exists before copying it since otherwise copy_file will throw an error.

    from distutils.file_util import copy_file
    import os
    
    os.makedirs(out_dir)
    df['file'].apply(lambda x: copy_file(x, out_dir) if os.path.isfile(x) else None)
    

integer division

Integer division in Python returns the floor of the result instead of truncating towards zero like C. The reason behind this choice is explained in http://python-history.blogspot.com/2010/08/why-pythons-integer-division-floors.html .

Consider for example integer_division.py and integer_division.cpp

     % python ./integer_division.py
    a, b, a//b, a%b
    5, 2, 2,  1
    -5, 2, -3,  1
    5, -2, -3,  -1
    -5, -2, 2,  -1
    
     % g++ ./integer_division.cpp
     % ./a.out
    a, b, a/b, a%b
    5, 2, 2, 1
    -5, 2, -2, -1
    5, -2, -2, 1
    -5, -2, 2, -1
    

regex for comma separated integers

search | comma delimited integers

    import re
    re.search('(\d+(?:,\d+)*)', "1,2,3,4").groups() 
    ('1,2,3,4',)
    
    re.search('(\d+(?:,\d+)*)', "1,2,3,4f").groups()
    ('1,2,3,4',)
    

If non capturing groups are not used, then you get

    re.search('(\d+(,\d+)*)', "1,2,3,4").groups() 
    ('1,2,3,4', ',4')
    

The inner group gets the latest match

    re.search('(\d+(,\d+))', "1,2,3,4").groups()
    ('1,2', ',2')
    re.search('(\d+(,\d+){2})', "1,2,3,4").groups()
    ('1,2,3', ',3')
    re.search('(\d+(,\d+)*)', "1,2,3,4").groups()
    ('1,2,3,4', ',4')
    

regex to check for dates

tags | check for digit

    >>> import re
    >>> file = 'foo_20200321.txt'
    >>> file_split = re.split(r'(\d{8})', file, 1)
    >>> file_split
    ['foo_', '20200321', '.txt']
    

Notes:

sort run_1 run_2 run_10

tags | pass custom function to sorted

The task here is to sort a list of strings of form 'run_N' on N where N is an integer. For example, if we have ['run_9', 'run_2', 'run_10', 'run_1'], the output should be ['run_1', 'run_2', 'run_9', 'run_10']

The default string sorting using the sorted function will not work since 'run_10' is smaller than 'run_2' in pure string comparison.

    $ ipython
    
    In [1]: a = ['run_9', 'run_2', 'run_10', 'run_1']
    
    In [2]: sorted(a)
    Out[2]: ['run_1', 'run_10', 'run_2', 'run_9']
    

Instead one has to extract the integer, N, and use it as the key.

    In [1]: a = ['run_9', 'run_2', 'run_10', 'run_1']
    
    In [2]: import re
    
    In [3]: pattern = 'run_' + '(\d+)$'
    
    In [4]: sorted(a, key=lambda x: int(re.search(pattern, x).group(1)))
    Out[4]: ['run_1', 'run_2', 'run_9', 'run_10']
    

extract something out of a string

     % python3
    Python 3.5.3 (default, Sep 27 2018, 17:25:39)
    [GCC 6.3.0 20170516] on linux
    
    >>> import re
    >>> a = "GeoIP Country Edition: US, United States"
    >>> b = "blah blah"
    >>> pattern = "GeoIP Country Edition: (\w+), "
    >>> re.match(pattern, a)
    <_sre.SRE_Match object; span=(0, 27), match='GeoIP Country Edition: US, '>
    >>> re.match(pattern, b)
    >>> re.match(pattern, a) is None
    False
    >>> re.match(pattern, b) is None
    True
    >>> re.search(pattern, a)
    <_sre.SRE_Match object; span=(0, 27), match='GeoIP Country Edition: US, '>
    >>> re.search(pattern, b)
    >>> re.search(pattern, a).group(1)
    'US'
    >>> re.search(pattern, b).group(1)
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    AttributeError: 'NoneType' object has no attribute 'group'
    >>> c = re.search(pattern, a).group(1) if re.match(pattern, a) else None
    >>> c
    'US'
    >>> c = re.search(pattern, b).group(1) if re.match(pattern, b) else None
    >>> c
    >>> c is None
    True
    

reload a module

For python 3

    import importlib
    importlib.reload(modulename)
    

For python 2

    reload(foo)
    

Note:- if you are importing functions from a module using

    from foo import *
    

then reload(foo) will throw a NameError.

    NameError: name 'foo' is not defined
    

This happens because there is no object bound to foo. As a work around, I do

    from foo import *
    import foo
    reload(foo)
    

See https://stackoverflow.com/questions/13455836/nameerror-when-using-reload for more details.

split string and trim white space

    s = "blah, lots  ,  of ,  spaces, here "
    [x.strip() for x in s.split(',')]
    
    Out[18]: ['blah', 'lots', 'of', 'spaces', 'here']
    

Ref:- http://stackoverflow.com/questions/4071396/split-by-comma-and-strip-whitespace-in-python


convert two lists into a dictionary

Use dict with zip to construct a dictionary from two lists. The number of elements in dictionary will be the minimum of the number of elements in each list. The extra elements will be discarded.

     % python3          
    Python 3.5.2+ (default, Sep 22 2016, 12:18:14) 
    [GCC 6.2.0 20160914] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    
    >>> a = [1, 2, 3]; b = ['a', 'b', 'c']; c = dict(zip(a, b)); print(a); print(b); print(c)
    [1, 2, 3]
    ['a', 'b', 'c']
    {1: 'a', 2: 'b', 3: 'c'}
    
    >>> a = [1, 2, 3, 4]; b = ['a', 'b', 'c']; c = dict(zip(a, b)); print(a); print(b); print(c)
    [1, 2, 3, 4]
    ['a', 'b', 'c']
    {1: 'a', 2: 'b', 3: 'c'}
    
    >>> a = [1, 2, 3]; b = ['a', 'b', 'c', 'd']; c = dict(zip(a, b)); print(a); print(b); print(c)
    [1, 2, 3]
    ['a', 'b', 'c', 'd']
    {1: 'a', 2: 'b', 3: 'c'}
    


debug from command line

To pass command line arguments and invoke the debugger directly

    python -u -m pdb mymodule.py --arg1_key arg1_val --arg2_key arg2_val
    

To do it inside python interpreter

    import pdb
    import mymodule
    pdb.run('mymodule.myfunc()')
    

Ref:-

tags | Debugging python programs from the command line

step through a python script

tags | debug Add the following lines at the beginning of the script

    import pdb
    pdb.set_trace()
    

Ref:- https://pythonconquerstheuniverse.wordpress.com/2009/09/10/debugging-in-python/

returning values

One way to return multiple values in a function is to use a dictionary.

    return {'key1':value1,'key2':value2,'key3':value3}
    

or

    result = {'key1':value1}
    result['key2'] = value2
    result['key3'] = value3
    return(result)
    

Ref:-

if else if syntax

    if x < 0:
        x = 0
        print('Negative changed to zero')
    elif x == 0:
        print('Zero')
    elif x == 1:
        print('Single')
    else:
        print('More')
    

check for missing data

    if (x is not None and x > 0):
        print('x is not none and positive')
    

using if else in a line

    >>> a = 5 if 2 > 3 else 4
    >>> a
    4
    >>> a = 5 if 3 > 2 else 4
    >>> a
    5
    

interval comparison

Python supports chained comparisons such as x < y <= z. It is equivalent to x < y and y <= z, except that y is evaluated only once (but in both cases z is not evaluated at all when x < y is found to be false).

In general, if a, b, c, …, y, z are expressions and op1, op2, …, opN are comparison operators, then a op1 b op2 c ... y opN z is equivalent to a op1 b and b op2 c and ... y opN z, except that each expression is evaluated at most once.

Note that a op1 b op2 c doesn’t imply any kind of comparison between a and c, so that, e.g., x < y > z is perfectly legal (though perhaps not pretty).

Ref:-

tags | a < b < c, check if a value is in between two values, multiple logical operators

Remove duplicates

To remove duplicates from a column of a dataframe

    unique_id = df['id'].drop_duplicates()
    

Ref:- http://pandas.pydata.org/pandas-docs/version/0.17.0/generated/pandas.Series.drop_duplicates.html

tags | unique

Replace newline characters with space

    % python3
    Python 3.4.3 (default, Mar  3 2015, 15:56:43)
    [GCC 4.4.7 20120313 (Red Hat 4.4.7-4)] on linux
    
    >>> a = '''kama
    ... raju'''
    >>> a
    'kama\nraju'
    >>> a.replace("\n", " ")
    'kama raju'
    

overlay one column with another

Say you have two columns 'a' and 'b' in a dataframe. To copy the values from column 'a' to column 'b' whenever the values in column 'b' are missing.

    df.loc[ pd.isnull(df['b']), 'b'] = \
      df.loc[ pd.isnull(df['b']), 'a']
    

tags | pandas update data based on a condition, indexing and selecting data, find and replace values in a python dataframe, replace values if null, assign values by index, search and replace data based on another column using python pandas, example on using loc, python pandas boolean array filter

specify the path of a file

To specify the path of a file relative to the location of the module

    fname = os.path.join(os.path.dirname(__file__), 'foo.txt')
    

__file__ gives the path of current module.

show all variables in python interpreter

use dir()

    >>> dir()
    ['__builtins__', '__doc__', '__loader__', '__name__', '__package__', '__spec__']
    >>> import pandas as pd
    >>> dir()
    ['__builtins__', '__doc__', '__loader__', '__name__', '__package__', '__spec__', 'pd']
    >>> df = pd.DataFrame([[1, 2], [3, 4]], columns=list('AB'))
    >>> dir()
    ['__builtins__', '__doc__', '__loader__', '__name__', '__package__', '__spec__', 'df', 'pd']
    >>> df
       A  B
    0  1  2
    1  3  4
    

sql query string

    sql_query = '''
    select *
      from alpha..beta
     where foo in ('{foo}')
       and bar in ('{boo}')
    '''
    res = db_query(sql_query.format(foo = "', '".join(foo),
                                    bar = "', '".join(bar))
                  )
    
    

where db_query is a custom function to query the database.

connecting to databases

Using pyodbc to connect to MSSQL

        import pyodbc
        import pandas as pd
        # declare user, password, server, database
        conn_str = 'DRIVER={SQL Server};SERVER=' + server + ';DATABASE=' + database
        if user is not None and password is not None:
            conn_str += ';UID=' + user + ';PWD=' + password
        else:
            conn_str += ';Trusted_Connection=yes'
        connection = pyodbc.connect(conn_str)
        data = pd.io.sql.read_sql(<SQL goes here>, connection)
    

Using SQLAlchemy

    from urllib import quote_plus as urlquote
    
    import sqlalchemy as sqla
    import pandas as pd
    
    def create_engine(conn_str):
        url = sqla.engine.url.make_url(conn_str)
        if url.drivername == 'oracle+cx_oracle':
            return sqla.create_engine(url, arraysize=100000)
        else:
            return sqla.create_engine(url)
    

Then for MSSQL

    # declare user, password, server, database
    driver = 'ODBC Driver 11 for SQL Server'
    
    if user is not None and password is not None:
        conn_str = "mssql+pyodbc://{}:{}@{}/{}?driver={}".format(
            user, urlquote(password), server, database, urlquote(driver)
        )
    else:
        # use windows authentication
        conn_str = "mssql+pyodbc://{}/{}?driver={};trusted_connection=yes".format(
            server, database, urlquote(driver)
        )
    engine = create_engine(conn_string)
    data = pd.read_sql_query('<SQL goes here>', engine)
    

and for Oracle

    # declare user, password, server
    conn_str = "oracle+cx_oracle://{}:{}@{}".format(
        user, urlquote(password), server
    )
    engine = create_engine(conn_string)
    data = pd.read_sql_query('<SQL goes here>', engine)
    

writing an if else condition in list comprehension

Syntax

    [ a if C else b for i in items]
    

is equivalent to

    for i in items:
      if (C):
        a
      else:
        b
    

For example, to figure out if a package is installed one can do

    import pandas as pd
    import apt
    cache = apt.Cache()
    
    a = pd.DataFrame({'package': ['python-apt', 'foo', 'dpkg', '0xffff']})
    status = [
        True if ((pkg in cache) and cache[pkg].is_installed) else False
        for pkg in a['package']
    ]
    
    print(status)
    

which will show

    [True, False, True, False]
    

where python-apt, dpkg are installed on the system, '0xffff' is a valid package but not installed and 'foo' is not a valid package name.

Ref: http://stackoverflow.com/questions/4406389/if-else-in-a-list-comprehension

Stackoverflow links

get the username

    >>> import getpass
    >>> getpass.getuser()
    'rajulocal'
    

exit with error

One way is to throw an exception. See http://stackoverflow.com/questions/2052390/manually-raising-throwing-an-exception-in-python for more details.

exit with control d in git bash

Use ipython to exit using control d in git bash.

control d does not work with regular python. There you have to use either exit() or ctrl z <enter>.

enter multiline statements in ipython

args and kwargs

args = arguments kwargs = keyword arguments

    def f(*args, **kwargs):
      print('args: ', args, ' kwargs: ', kwargs)
    
    >>> f('a')
    args:  ('a',)  kwargs:  {}
    >>> f(ar='a')
    args:  ()  kwargs:  {'ar': 'a'}
    >>> f(1,2,param=3)
    args:  (1, 2)  kwargs:  {'param': 3}
    
    >>> f(1,[2,3],param=3)
    args:  (1, [2, 3])  kwargs:  {'param': 3}
    
    >>> f(1, [2,3], [4,5,6], param=3, first='kama', last='raju')
    args:  (1, [2, 3], [4, 5, 6])  kwargs:  {'param': 3, 'last': 'raju', 'first': 'kama'}
    

See also: http://stackoverflow.com/questions/682504/what-is-a-clean-pythonic-way-to-have-multiple-constructors-in-python?noredirect=1&lq=1


using kargs

Sample usage

    def foo(alpha, beta, **kwargs):
        debug = kwargs.get('debug', False)
    

multiline statements

    df = pd.read_csv(fname)\
    .pipe(func_foo)\
    .pipe(func_bar)
    

calling python script from bash

print stack trace

To print stack trace from within the program

    import traceback
    print 'printing stack'
    traceback.print_stack()
    

Ref:- https://docs.python.org/2/library/traceback.html

print exception and continue

http://www.kamaraju.xyz/dk/python_notes#print_exception_and_continue

print exception to log file

http://www.kamaraju.xyz/dk/python_notes#print_exception_to_log_file

parse config files

Example 1:

     % cat config.ini
    [COUNTRIES]
    DENY = AG AL
    ALLOW = IN US
    
     % cat config.py
    import configparser
    import os
    
    Config = configparser.ConfigParser()
    Config.read(os.path.join(os.path.dirname(__file__), 'config.ini'))
    

To use it

     % python3
    Python 3.5.3 (default, Jan 19 2017, 14:11:04)
    [GCC 6.3.0 20170118] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>> from config import Config
    >>> Config['COUNTRIES']['DENY']
    'AG AL'
    >>> Config['COUNTRIES']['ALLOW']
    'IN US'
    

Porting windows code to Linux

Things to look out for when porting python code from windows to Linux

  • use os.path.sep instead of "\\" for parsing directories in a filepath
  • use os.path.join when constructing file name. For example instead of dir_name + "\\" + file_name, use os.path.join(dir_name, file_name)
  • exception during reset or similar

Download a webpage

version 1:-

    import requests
    
    
    def get_html(url):
        # Get html from url
        response = requests.get(url)
        return response.text
    

Version 2:- Same as version 1, but has user_agent hardcoded

    import requests
    
    
    def get_html(url):
        # Get html from url
        user_agent = 'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.139 Safari/537.36'
        header = {'User-Agent': user_agent}
        response = requests.get(url, headers=header)
        return response.text
    

what is **{}

    $python
    Python 3.6.6 |Anaconda, Inc.| (default, Jun 28 2018, 11:27:44) [MSC v.1900 64 bit (AMD64)] on win32
    
    >>> def foo(**bar):
    ...   print(bar)
    ...
    
    >>> foo(**{'a':2})
    {'a': 2}
    
    >>> foo(**{'a':2, 'b':3})
    {'a': 2, 'b': 3}
    
    >>> foo()
    {}
    

See:-

using timeit

    $ python -mtimeit "l=[]"
    10000000 loops, best of 3: 0.0274 usec per loop
    
    $ python -mtimeit "l=list()"
    10000000 loops, best of 3: 0.125 usec per loop
    

pep8 recommendations

  • use 4-space indentation, and no tabs
  • 79 characters per line
  • use CamelCase for classes and lower_case_with_underscores for functions and methods

See:-

python web interpreters

share variables across different files

tags | globals, singleton, variables shared across modules

See | https://instructobit.com/tutorial/108/How-to-share-global-variables-between-files-in-Python


See also

os.makedirs

os.makedirs is like "mkdir -p". It will create intermediate directories if necessary.

get username in windows

    os.getlogin()
    

returns the user that is executing the script.

Extract first N elements from a generator

    $python
    Python 3.6.8 |Anaconda, Inc.| (default, Dec 30 2018, 01:22:34)
    [GCC 7.3.0] on linux
    
    >>> generator = (i for i in range(10))
    
    >>> type(generator)
    <class 'generator'>
    
    >>> list(next(generator) for _ in range(4))
    [0, 1, 2, 3]
    >>> list(next(generator) for _ in range(4))
    [4, 5, 6, 7]
    >>> list(next(generator) for _ in range(4))
    [8, 9]
    >>> list(next(generator) for _ in range(4))
    []
    

Notice how the last two calls give only the remaining elements. If you use [] instead of list(), a StopIteration exception will be thrown in those cases.

    >>> generator = (i for i in range(10))
    >>> [next(generator) for _ in range(4)]
    [0, 1, 2, 3]
    >>> [next(generator) for _ in range(4)]
    [4, 5, 6, 7]
    >>> [next(generator) for _ in range(4)]
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "<stdin>", line 1, in <listcomp>
    StopIteration
    

timing a function

    start_time = time.time()
    call_function_of_interest()
    end_time = time.time()
    print('time elapsed = ', end_time - start_time)
    

For python2

    print 'time elapsed = ', end_time - start_time
    print 'took %.f sec to finish' %(end_time - start_time)
    

For Python 3 using logger

    start = time.time()
    call_function_of_interest()
    elapsed = time.time() - start
    logger.info('function finished in %.2f seconds' % elapsed)
    

tags | time a function, tic toc

Subprocess related

grep of git status

tags | git status grep cut subprocess Popen PIPE, demonstrates | convert bytes to string

    def grep_on_git_status():
        git_st = Popen(['git', 'status', '--porcelain'], stdout=PIPE, stderr=PIPE)
        grep = Popen(['grep', '^??'], stdin=git_st.stdout, stdout=PIPE)
        cut = Popen(['cut', '-f', '2', '-d', ' '], stdin=grep.stdout, stdout=PIPE)
        # The cut.stdout is a gigantic byte string.
        # The readlines() will break it into a listof byte strings
        files = cut.stdout.readlines()
        # strip the newline characters at the end
        files = [x.decode('utf-8').strip() for x in files]
        return files
        
    files = grep_on_git_status()
    print(type(files))
    print(files)
    

Ref:-

notes on subprocess.run

pretty formatting subprocess commands

    import subprocess
    
    pkges = ['pandas',
             'numpy']
    
    for pkg in pkges:
        cmd = ['conda',
               'list',
               '-n',
               'base',
               pkg]
        
        result = subprocess.run(cmd,
                                stdout=subprocess.PIPE,
                                stderr=subprocess.PIPE,
                                universal_newlines=True)
        print(result.stdout)
    

sample output:

    # packages in environment at C:\ProgramData\Continuum\Anaconda:
    #
    # Name                    Version                   Build  Channel
    pandas                    0.23.0           py36h830ac7b_0  
    
    # packages in environment at C:\ProgramData\Continuum\Anaconda:
    #
    # Name                    Version                   Build  Channel
    numpy                     1.14.3           py36h9fa60d3_1  
    numpy-base                1.14.3           py36h555522e_1  
    numpydoc                  0.8.0                    py36_0  
    

Code snippets

Add a directory to the beginning of PATH in Windows

    backup_path = os.environ['PATH']
    if sys.platform == 'win32':
        new_dir = os.path.join('foo', 'bar')
        os.environ['PATH'] = "%s;%s" % (new_dir, backup_path)
    

small experiments

infinity

    % python3
    Python 3.5.3 (default, Jan 19 2017, 14:11:04) 
    [GCC 6.3.0 20170118] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>> float('inf')
    inf
    >>> type(float('inf'))
    <class 'float'>
    >>> float('inf') > 5
    True
    >>> float('inf') > -float('inf')
    True
    >>> float('inf') is float('inf')
    False
    >>> float('inf') == float('inf')
    True
    

Operators related

logical operators

    x or y, x and y, not x
    

Example

        if (a < 1 or a > 100):
            print('a is not in [1,100]')
    

Ref:- https://docs.python.org/3/library/stdtypes.html

bitwise logic operators

    &, |, ^, and ~
    

Example

    np.sum((inches > 0.5 & (inches < 1))
    

Parenthesis are important here because of operator precedence rules. Without the parenthesis, the expression will be evaluated as Example

    inches (> 0.5 &) < 1
    

Another way to express the above logic

    np.sum(~( (inches <= 0.5) | (inches >= 1) ))
    

sinc A AND B is same as NOT (NOT A OR NOT B)

operator precedence

+= and -=

These (+=, -=, *= and /=) are called augmented arithmetic assignments. They are described in https://docs.python.org/3/reference/datamodel.html#emulating-numeric-types -> scroll down to the section on __iadd__, __isub__ functions.

the i semantically means "in-place", which means that they modify the object (or reference in the case of numerics) without having to additionally assign them.

conditional operator

tags | conditional expressions, ternary

    a if condition else b
    

What is the difference between <> and !=

In python 3, there is no <> operator.

In python 2, they both mean the same. != is preferred; <> is obsolescent.

Ref:-

numpy

Frequently used numpy functions

use case solution
average np.average
sum np.sum

OOP related

referring to class variables in methods

tags | use classmethod to access class variables

    $ cat foo.py
    class Days():
        SUN = "Sunday"
        MON = "Monday"
        TUE = "Tuesday"
        WED = "Wednesday"
        THU = "Thursday"
        FRI = "Friday"
        SAT = "Saturday"
    
        @classmethod
        def weekdays(cls):
            return [cls.MON, cls.TUE, cls.WED, cls.THU, cls.FRI]
    
        @classmethod
        def weekends(cls):
            return [cls.SAT, cls.SUN]
    

To use them in bar.py

    from foo import Days
    a = Days.SUN
    b = Days.weekdays()
    c = Days.weekends()
    

See also | https://stackoverflow.com/a/709024/6305733

Challenges

Rotate matrix anti clockwise

Given a 2-D array

    [[1,2,3,4],
     [5,6,7,8],
     [9,10,11,12]]
    

convert it to

    [[4,8,12],
     [3,7,11],
     [2,6,10],
     [1,5,9]]
    
    >>> a = [[1,2,3,4], [5,6,7,8], [9,10,11,12]]
    >>> a
    [[1, 2, 3, 4], [5, 6, 7, 8], [9, 10, 11, 12]]
    >>> b = list(zip(*a))[::-1]
    >>> b
    [(4, 8, 12), (3, 7, 11), (2, 6, 10), (1, 5, 9)]
    >>> c = [list(elem) for elem in b]
    >>> c
    [[4, 8, 12], [3, 7, 11], [2, 6, 10], [1, 5, 9]]
    

Rotate matrix clockwise

Given a 2-D array

    [[1,2,3,4],
     [5,6,7,8],
     [9,10,11,12]]
    

convert it to

    [[9,5,1],
     [10,6,2],
     [11,7,3],
     [12,8,4]]
    
    >>> a = [[1,2,3,4], [5,6,7,8], [9,10,11,12]]
    >>> a
    [[1, 2, 3, 4], [5, 6, 7, 8], [9, 10, 11, 12]]
    >>> b = list(zip(*a[::-1]))
    >>> b
    [(9, 5, 1), (10, 6, 2), (11, 7, 3), (12, 8, 4)]
    >>> c = [list(elem) for elem in b]
    >>> c
    [[9, 5, 1], [10, 6, 2], [11, 7, 3], [12, 8, 4]]
    

index of an item in a sorted list

Use bisect module to find out where an item can be inserted in a sorted list. The return value of bisect is suitable for use as the first parameter to list.insert() assuming that original list is already sorted.

    >>> a = [10,1,8,2,5,3,7,9,6,4]
    >>> a
    [10, 1, 8, 2, 5, 3, 7, 9, 6, 4]
    >>> a.sort()
    >>> a
    [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
    >>> x = 5.5
    >>> from bisect import bisect
    >>> bisect(a,x)
    5
    >>> a.insert( bisect(a,x), x)
    >>> a
    [1, 2, 3, 4, 5, 5.5, 6, 7, 8, 9, 10]
    

Instead of doing bisect and insert, you can also insort which does both.

    >>> a = [10,1,8,2,5,3,7,9,6,4]
    >>> a
    [10, 1, 8, 2, 5, 3, 7, 9, 6, 4]
    >>> a.sort()
    >>> a
    [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
    >>> x = 5.5
    >>> from bisect import insort
    >>> insort(a,x)
    >>> a
    [1, 2, 3, 4, 5, 5.5, 6, 7, 8, 9, 10]
    

Ref:- https://docs.python.org/3/library/bisect.html

questions

decorators

In https://github.com/RomelTorres/alpha_vantage/blob/develop/alpha_vantage/techindicators.py , I see things like

        @av._output_format
        @av._call_api_on_func
        def get_sma(self, symbol, interval='daily', time_period=20, series_type='close'):
    

What do those @av decorators do? Read about this later.

books

Think Python

Book info:

The second edition uses Python 3. The first edition uses Python 2.

Instructions for working with this code are in the preface.

Review:

  • rating: 5 out of 5.
  • level: beginner
  • easy to read, easy to understand, no fluff, no nonsense.
  • highly recommended.