Python notes
From raju
Note:- This page will be migrated to http://www.kamaraju.xyz/dk/python_notes. Once the migration is done, it will be deleted.
Troubleshooting
no module named error when importing from subdirectories
- Make sure that each directory contains __init__.py
ImportError: DLL load failed while importing win32api: The specified module could not be found.
Problem
>>> import win32api Traceback (most recent call last): File "<stdin>", line 1, in <module> ImportError: DLL load failed while importing win32api: The specified module could not be found.
Solution: Install pywin32 package
Utility functions
print command before executing it
command line arguments
Utility Scripts
- Python program that summarizes the shape of a data structure - https://github.com/AllenDowney/ThinkPython2/blob/master/code/structshape.py
Code Templates
Template code for regression testing a Python3 application
Check pep8 conformance via unittest
Functionality testing of python scripts
using printf
indent = " " printf("%s average = %.4g" %(indent, df['foo'].mean()) ) printf("%s N = %d" %(indent, df.shape[0]) )
See also:-
expand %s in a string with variable values
>>> a = '%s:micro soft, %s:google' >>> b = 'MSFT'; c = 'GOOG'; >>> a '%s:micro soft, %s:google' >>> b 'MSFT' >>> c 'GOOG' >>> d = a % (b, c) >>> d 'MSFT:micro soft, GOOG:google'
string construction
Use a tuple while constructing a string with multiple placeholders
>>> 'select * from table_name where column in (%s,%s)' % ('val1','val2') 'select * from table_name where column in (val1,val2)'
Using an array will give an error
>>> 'select * from table_name where column in (%s,%s)' % ['val1','val2'] Traceback (most recent call last): File "<ipython-input-66-xxxxxxxxxxxx>", line 1, in <module> 'select * from table_name where column in (%s,%s)' % ['val1','val2'] TypeError: not enough arguments for format string
pep8 write long strings
user_agent = 'Mozilla/5.0 (Windows NT 6.1; Win64; x64) ' \ 'AppleWebKit/537.36 (KHTML, like Gecko) ' \ 'Chrome/66.0.3359.139 Safari/537.36'
break up long lines
Do
if cond1 and \ cond2:
Instead of
if cond1 and cond2:
For formulas
# Yes: easy to match operators with operands income = (gross_wages + taxable_interest + (dividends - qualified_dividends) - ira_deduction - student_loan_interest)
Ref:- https://www.python.org/dev/peps/pep-0008/#indentation
print objects
class SimpleRepr(object): """ A mixin implementing a simple __repr__. It gives the class name, the (shortened) id, and all of the attributes. """ def __repr__(self): return "<{klass} @{id:x} {attrs}>".format( klass=self.__class__.__name__, id=id(self) & 0xFFFFFF, attrs=" ".join("{}={!r}".format(k, v) for k, v in self.__dict__.items()), )
Use it as follows
class DataWrapper(SimpleRepr): def __init__(self, args): args_dict = vars(args) for (k,v) in args_dict.items(): setattr(self, k, v) if __name__ == '__main__': # parse arguments using argparser data_wrapper = DataWrapper(parsed_args)
Ref:-
- https://stackoverflow.com/a/44595303/6305733 - for __repr__
- https://stackoverflow.com/a/24474314/6305733 - for reading command line arguments into a data wrapper object
catching exception
try: # Do something except Exception as e: self.logger.warnings("doing %s returned exception %s", str(foo), str(e))
raising exception
The raise statement causes an exception. To cause a LookupError, which is a built-in exception used to indicate that a lookup operation failed.
def reverse_lookup(d, v): for k in d: if d[k] == v: return k raise LookupError()
The effect when you raise an exception is the same as when Python raises one: it prints a traceback and an error message.
When you raise an exception, you can provide a detailed error message as an optional argument. For example:
raise LookupError('value does not appear in the dictionary')
logging
simple example
import logging import sys logger = logging.getLogger(__name__) if __name__ == '__main__': logging.basicConfig( level=logging.INFO, format='%(asctime)s %(name)s %(levelname)s %(message)s') logger.info('python executable is %s', sys.executable)
logging variables
To log a string
>>> s = 'foo' ... logger.info('s is %s' % s) 2020-07-18 11:28:45,226 __main__ INFO s is foo
or
>>> s = 'foo' ... logger.info('s is ' + s) 2020-07-18 11:29:05,198 __main__ INFO s is foo
To log a boolean variable
>>> a = True ... logger.info('a is %r' % a) 2020-07-18 11:25:31,937 __main__ INFO a is True
To log multiple boolean variables
>>> a = True ... b = False ... logger.info('a is %r' % a) ... logger.info('a and b are %r and %r respectively' % (a, b)) 2020-07-18 11:26:45,771 __main__ INFO a is True 2020-07-18 11:26:45,771 __main__ INFO a and b are True and False respectively
list files
Recursively list all python files under a directory
list all directories except one
$ ls bar/ bar.txt baz/ baz.txt foo/ foo.txt script.py $ cat script.py import os root_dir = '.' dirs = [d for d in os.listdir(root_dir) if os.path.isdir(os.path.join(root_dir, d)) and d != 'bar'] print(dirs) $ python ./script.py ['baz', 'foo']
list directories
- os.listdir(foo) - list all files and directories under foo. Not recursive. Just shows the top first level stuff.
- List only directories
all_dirs = [d for d in os.listdir(root_dir) if os.path.isdir(os.path.join(root_dir, d))]
list files in descending order of size
def large_files_first(dirPath): allFiles = (os.path.join(dirPath, file) for file in os.listdir(dirPath)) sortedFiles = sorted(allFiles, key=os.path.getsize, reverse=True) return sortedFiles
Sample run:
- Create the files
base64 /dev/urandom | head -c 1k > one.txt base64 /dev/urandom | head -c 2k > two.txt base64 /dev/urandom | head -c 3k > three.txt
Check their sizes
$ du -b *.txt 1024 one.txt 3072 three.txt 2048 two.txt
The default listing
$ ipython In [1]: import os ...: dirPath = os.getcwd() ...: allFiles = [os.path.join(dirPath, file) for file in os.listdir(dirPath)] ...: allFiles Out[1]: ['C:\\Users\\kkusuman\\x\\x17\\one.txt', 'C:\\Users\\kkusuman\\x\\x17\\three.txt', 'C:\\Users\\kkusuman\\x\\x17\\two.txt']
Listing the big ones first
In [1]: import os ...: def large_files_first(dirPath): ...: allFiles = (os.path.join(dirPath, file) for file in os.listdir(dirPath)) ...: sortedFiles = sorted(allFiles, key=os.path.getsize, reverse=True) ...: return sortedFiles ...: ...: dirPath = os.getcwd() ...: sortedFiles = large_files_first(dirPath) ...: sortedFiles Out[1]: ['C:\\Users\\kkusuman\\x\\x17\\three.txt', 'C:\\Users\\kkusuman\\x\\x17\\two.txt', 'C:\\Users\\kkusuman\\x\\x17\\one.txt']
String interpolation
print current function name
print('I am in {} function'.format(__name__))
Ref:- https://stackoverflow.com/questions/251464/how-to-get-a-function-name-as-a-string
listing all the variables in format()
tags | string interpolation using format()
In [1]: a = 'kama' ...: b = 'raju' ...: c = "foo_{alpha}_bar_{beta}.txt".format(alpha=a, beta=b) ...: print([a, b, c]) ['kama', 'raju', 'foo_kama_bar_raju.txt']
In the format function, you have to specify the variable names and its values. Otherwise it gives a KeyError.
In [2]: c = "foo_{alpha}_bar_{beta}.txt".format(a, b) --------------------------------------------------------------------------- KeyError Traceback (most recent call last) <ipython-input-2-34f487dabb30> in <module>() ----> 1 c = "foo_{alpha}_bar_{beta}.txt".format(a, b) KeyError: 'alpha'
using dictionary in format()
In [1]: d = {'alpha': 'kama', 'beta': 'raju'} ...: d Out[1]: {'alpha': 'kama', 'beta': 'raju'} In [2]: c = "foo_{alpha}_bar_{beta}.txt".format(**d) ...: c Out[2]: 'foo_kama_bar_raju.txt'
using locals() in format()
def buildString(user, name = 'john', age=22): userId = user.getUserId() return "Name: {name}, age: {age}, userid:{userId}".format(**locals())
Tested it using Python 3.4.3
Advantage:
- This is better than
return "Name: {name}, age: {age}, userid:{userId}".format(name=name, age=age, userId=userId)
which is ugly.
Disadvantage:
- Do not use it when the format string is user supplied since then it would open up access to every local variable making the application susceptible to injection attacks.
Ref:-
% vs. .format()
.format() is recommended over %. .format() was introduced in Python 2.6.
Ref:- https://stackoverflow.com/questions/5082452/string-formatting-vs-format
Tips
configure vim
" To add the proper PEP8 indentation au BufRead,BufNewFile *.py,*.pyw set tabstop=4 au BufRead,BufNewFile *.py,*.pyw set softtabstop=4 au BufRead,BufNewFile *.py,*.pyw set shiftwidth=4 au BufRead,BufNewFile *.py,*.pyw set textwidth=79 au BufRead,BufNewFile *.py,*.pyw set expandtab au BufRead,BufNewFile *.py,*.pyw set autoindent au BufRead,BufNewFile *.py,*.pyw set fileformat=unix
tested using vim 7.2
Ref:- https://realpython.com/blog/python/vim-and-python-a-match-made-in-heaven/
Copying files
recursively copy files and overwrite if necessary
tags | copying files, copytree overwrite
tags | preserve timestamp when copying files
Use distutils.dir_util.copytree
Sample code snippet
from distutils.dir_util import copy_tree if os.path.isdir(src): copy_tree(src, dst)
This will recursively copy whatever is in src to dst (ex:- src/foo to dst/foo, src/foo/bar/baz.txt to dst/foo/bar/baz.txt etc.,)
If same file exists in both, the one in src overwrites the one in dst.
If a file exists only in dst, but not in src, it is not touched.
Notes:
- src has to be a directory
- dst will be created if it does not already exist
- all child directories under dst will be created as needed
- dst can be an existing directory but not an existing file
- timestamps are preserved by default
Ref:-
- https://docs.python.org/3/distutils/apiref.html#module-distutils.dir_util
- https://docs.python.org/2/distutils/apiref.html#module-distutils.dir_util
Why not shutil.copytree?
- shutil.copytree throws an error if dst directory already exists.
WindowsError: [Error 183] Cannot create a file when that file already exists: '/path/to/dst'
distutils.dir_util.copy_tree does not care if that is the case. It will simply overwrite the files inside it.
copying files
To copy a single file, you can use shutil.copyfile, shutil.copy(), shtuil.copy2()
syntax
shutil.copyfile(src, dst, *, follow_symlinks=True) shutil.copy(src, dst, *, follow_symlinks=True) shutil.copy2(src, dst, *, follow_symlinks=True)
shtuil.copyfile()
destination will be overwritten if it already exists. dst has to be a file and cannot be a directory.
For example
shutil.copyfile('C:/Users/kkusuman/x/x1/junk1.txt', 'C:/Users/kkusuman/x/x1/junk2.txt') shutil.copyfile('C:\\Users\\kkusuman\\x\\x1\\junk1.txt', 'C:\\Users\kkusuman\\x\\x1\\junk2.txt')
shtuil.copy()
dst can either be a file or a directory. It copies the data and file's permission mode. Other metadata such as file's creation and modification times are not preserved.
shtuil.copy2()
same as shutil.copy() except that it preserves all metadata.
Ref:- https://docs.python.org/3/library/shutil.html#shutil.copy
Delete files
deleting files
def remove_file(fname): if os.path.isfile(fname): try: os.remove(fname) except Exception as exc: print exc pass
delete directory recursively
import os from distutils.dir_util import remove_tree if os.path.isdir(fname): remove_tree('/path/to/foo')
disttuils.dir_util.remove_tree works only on a directory and the directory should exist. Otherwise, it will throw an exception.
If you try to delete a file using remove_tree, it will throw an exception
NotADirectoryError: [WinError 267] The directory name is invalid: 'file.txt'
writing files
write a list of strings to a file
Get the input
input_file = 'foo.txt' with open(input_file) as fh: # read first three lines into a list header = [fh.readline() for line in range(3)]
write it
output_file = 'bar.txt' with open(output_file) as fh: for line in header: fh.write('%s' % line)
tags | array of strings
Dummy
String for SQL where clause
write either to file or to stdout
Approach 1: cat_files() shows how to write binary data, cat_files_with_same_header() shows how write normal data. In both cases, output is written to dest if it is a file path. If it is None or sys.stdout, output is written to stdout.
Tested on | Python 3.8.2
def cat_files(dest, sources): """Cat multiple files into dest.""" if dest is None or dest is sys.stdout: # https://bugs.python.org/issue4571 says to use sys.stdout.buffer # when writing binary data to stdout. fdst = sys.stdout.buffer else: fdst = open(dest, 'wb') for src in sources: with open(src, 'rb') as fsrc: shutil.copyfileobj(fsrc, fdst) if fdst is not sys.stdout.buffer: fdst.close() def cat_files_with_same_header(dest, sources): """Cat multiple files into dest. Ignore headers in all the files except the first.""" if dest is None or dest is sys.stdout: fdst = sys.stdout else: fdst = open(dest, 'w') first_src = sources.pop(0) with open(first_src, 'r') as fsrc: header = fsrc.readline() fdst.write(header) shutil.copyfileobj(fsrc, fdst) for src in sources: with open(src, 'r') as fsrc: # read but ignore the header header = fsrc.readline() shutil.copyfileobj(fsrc, fdst) if fdst is not sys.stdout: fdst.close()
Sample calls
cat_files(dest_file, source_files) cat_files(None, source_files) cat_files(sys.stdout, source_files) cat_files_with_same_header(dest_file, source_file) cat_files_with_same_header(None, source_file) cat_files_with_same_header(sys.stdout, source_file)
Ref:- https://github.com/KamarajuKusumanchi/rutils/blob/master/python3/cat_files.py - latest version of the functions.
Approach 2:
/* 2020-06-13: Delete this approach if you are happy with the first approach. */
import os odir = "foo" ofile = os.path.join(odir, "bar.txt") with open(ofile, "w") as fh: print('writing', ofile) data_dumper(data, fh) ... def data_dumper(data, fh) fh.write(data) ...
To print data onto stdout
import sys data_dumper(data, sys.stdout)
See also: http://stackoverflow.com/questions/17602878/how-to-handle-both-with-open-and-sys-stdout-nicely
Approach 3:
/* 2020-06-13: Delete this approach if you are happy with the first approach. */
def dump_foo(arg1, arg2, ..., param={}) import sys # do something # dump to a file if some parameters are specified otherwise use stdout if (('out_dir' in param) & ('out_file' in param)): out_dir = param['out_dir'] if not os.path.exists(out_dir): os.makedirs(out_dir, exist_ok=True) out_file = os.path.join(out_dir, param['out_file']) fh = open(out_file, "w") print('writing', out_file) else: fh = sys.stdout; fh.write("I am in dump_foo()\n") if fh is not sys.stdout: fh.close()
list of alphabets
from string import ascii_letters print(ascii_letters)
abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ
Ref:- https://docs.python.org/3/library/string.html
Tutorials
The advantage of the pdf version is that all the content is in a single file. The advantage of html is that python code is shown with syntax highlighting.
Profiling Python
Install python on windows
- Instructions to install python on windows are at http://learnpythonthehardway.org/book/ex0.html
documentation links
- unittest - Unit testing framework
range like functions
- range function - https://docs.python.org/3/library/stdtypes.html#range
- randint - https://docs.python.org/3/library/random.html#random.randint
- randrange - https://docs.python.org/3/library/random.html#random.randrange
- random integer in [a, b]
- randint(a, b)
- randrange(a, b+1)
- To generate random numbers between 0 and N (including both)
- randint(0, N)
range(N) => 0 through N-1; count = N
set PYTHONPATH in windows
Use ';' as the separator. For example
PYTHONPATH="DIR1;DIR2;$PYTHONPATH" python foo.py --arg1 bar
So it may look
PYTHONPATH="C:\Temp\utils;$PYTHONPATH" python foo.py --file 'C:\Temp\input\bar.csv'
Run python from a windows script
call it foo.cmd
@echo off :: Anaconda path set ANACONDA_PATH=C:\Path\to\Continuum\Anaconda2 :: Report env variables. :: set set PATH=%ANACONDA_PATH%;%ANACONDA_PATH%\Scripts;%PATH% :: Report the path set PATH call activate myenv c:/Windows/System32/where.exe python python --version python -c "import pandas as pd; print(pd.__version__)" call deactivate
try-except-else
Pseudo-code to understand the try-except-else-finally structure.
try: try_this(whatever) except SomeException as the_exception: handle_SomeException(the_exception) # Handle a instance of SomeException or a subclass of it. except Exception as the_exception: generic_handle(the_exception) # Handle any other exception that inherits from Exception # - doesn't include GeneratorExit, KeyboardInterrupt, SystemExit # Avoid bare `except:` else: # there was no exception whatsoever return something() # if no exception, the "something()" gets evaluated, # but the return will not be executed due to the return in the # finally block below. finally: # this block will execute no matter what, even if no exception, # after "something" is eval'd but before that value is returned # but even if there is an exception. # a return here will hijack the return functionality. e.g.: return True # hijacks the return in the else clause above
Ref:-
- https://stackoverflow.com/questions/16138232/is-it-a-good-practice-to-use-try-except-else-in-python/ - Very good discussion on the try-except-else-finally structure. There are two useful posts in this thread https://stackoverflow.com/a/16138864/6305733 , https://stackoverflow.com/a/31626974/6305733. Each of them is a bit long but well worth reading in entirety. The above pseudocode is copied from https://stackoverflow.com/a/31626974/6305733 .
- https://realpython.com/python-exceptions/ - A bit long but explains everything in an easy to understand fashion.
os.path.isfile vs os.path.exists
- os.path.isfile - is it an existing regular file?
- os.path.exist - does it exist?
os.path.isfile also checks for existence. So no need to use both os.path.isfile and os.path.exists.
what is the difference between [] and list()
In [1]: a = ['abc', ' A', ' B', 'C'] ...: a[0] Out[1]: 'abc' In [2]: [a[0]] Out[2]: ['abc'] In [3]: list(a[0]) Out[3]: ['a', 'b', 'c'] In [4]: [a[0]] * 2 Out[4]: ['abc', 'abc'] In [5]: list(a[0]) * 2 Out[5]: ['a', 'b', 'c', 'a', 'b', 'c']
What is the difference between os.environ['foo'] vs os.getenv('foo')
If the variable 'foo' is not defined, os.environ['foo'] gives an exception but os.getenv('foo') returns an empty string.
If the variable is defined, both commands give its value.
$winpty python Python 3.6.1 |Anaconda 4.4.0 (64-bit)| (default, May 11 2017, 13:25:24) [MSC v.1900 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import os >>> var = 'TEMP' >>> var in os.environ True >>> os.getenv(var) 'C:\\Windows\\TEMP' >>> os.environ[var] 'C:\\Windows\\TEMP' >>> os.getenv(var) is os.environ[var] True >>> var = 'blah' >>> var in os.environ False >>> os.getenv(var) >>> os.getenv(var) is None True >>> os.environ[var] Traceback (most recent call last): File "<stdin>", line 1, in <module> File "C:\ProgramData\Continuum\Anaconda\envs\py36\lib\os.py", line 669, in __getitem__ raise KeyError(key) from None KeyError: 'blah'
rename directories
tags | rename directory, move a directory only if it exists, apply a function to each row in a pandas dataframe
to_rename = pd.DataFrame({ 'src': ['/path/to/src1', '/path/to/src2', '/path/to/src3', '/path/to/src4'], 'dst': ['/path/to/dst1', '/path/to/dst2', '/path/to/dst3', '/path/to/dst4'] }) to_rename.apply(lambda row: os.rename(row['src'], row['dst']) if os.path.exists(row['src']) else None, axis=1)
show an environment variable
import os try: user_paths = os.environ['FOO'].split(os.pathsep) except KeyError: user_paths = []
For example, to show the PYTHONPATH in the python interpreter
import os os.environ['PYTHONPATH'].split(os.pathsep)
Note: The separator changes depending on the OS. For example, Linux uses ':' which Windows uses ';'.
check the python version
>>> import sys >>> print(sys.version) 3.6.6 |Anaconda custom (64-bit)| (default, Jun 28 2018, 11:27:44) [MSC v.1900 64 bit (AMD64)]
Add a directory to the module search path
import sys foo_dir = "C:\\path\\to\dir" if foo_dir not in sys.path: sys.path.append(foo_dir)
demonstrates | import python module into a jupyter notebook
append to a file
This is useful when writing logs. For example
#! /usr/bin/env python3 import os from datetime import datetime a = datetime.now().strftime("%Y%m%d_%H%M%S") out_dir = '/path/to/logdir' if not os.path.exists(out_dir): os.makedirs(out_dir, exist_ok=True) out_file_name = "log.txt" out_file_path = os.path.join(out_dir, out_file_name) with open(out_file_path, 'a') as fh: fh.write(a + "\n")
frequently used functions
Use case | function | Additional notes |
get current directory | cwd = os.getcwd() | |
change working directory | os.chdir(path) | For example os.chdir('C:\Temp\foo') |
import modules from a different directory |
|
|
Add parent directory to the module search path | sys.path.append('..') | Module Search Path |
Get the PATH |
|
|
get basename | os.path.basename(path) | path is a string. It cannot be None. |
frequently used pdb commands
until NNN - continue until line NNN
script to plot a column of data from command line
See https://gitlab.com/d3k2mk7/rutils/blob/master/python3/plot_utils/plot_by_index.py
raw strings and normpath
a = ['test_cases\test1.csv', r'test_cases\test1.csv', 'test_cases/test1.csv', r'test_cases/test1.csv'] a Out[86]: ['test_cases\test1.csv', 'test_cases\\test1.csv', 'test_cases/test1.csv', 'test_cases/test1.csv'] b = [os.path.normpath(p) for p in a] b Out[88]: ['test_cases\test1.csv', 'test_cases\\test1.csv', 'test_cases\\test1.csv', 'test_cases\\test1.csv'] c = [eval("r'%s'" %p) for p in a] c Out[94]: ['test_cases\test1.csv', 'test_cases\\test1.csv', 'test_cases/test1.csv', 'test_cases/test1.csv'] d = [p.encode('string_escape') for p in a] d Out[96]: ['test_cases\\test1.csv', 'test_cases\\\\test1.csv', 'test_cases/test1.csv', 'test_cases/test1.csv']
random links
- Google's Python Class - https://developers.google.com/edu/python/ . It uses python2.
visualization
- https://github.com/seemantobarua/dviz/blob/master/python/line%20chart%20demo.ipynb - Generate a line chart that shows the period over which the monarchs ruled the British empire.
- https://www.ben-evans.com/benedictevans/2018/8/29/tesla-software-and-disruption -> "Figure 11: BNEF lithium-ion battery price survey results - volume-weighted average" shows a nice bar chart with % decreases at the top. I think this is a very nice way of showing the percent changes,
The image can also be accessed via https://static1.squarespace.com/static/50363cf324ac8e905e7df861/t/5b86d8a7562fa768a6edfec8/1535563952409/Screen+Shot+2018-08-29+at+10.30.01+AM.png?format=750w
using sort
Using Python 3.6.6
>>> x=["e","a","é","f"] >>> x.sort() >>> x ['a', 'e', 'f', 'é'] >>> x=["a","A","b","B"] >>> x.sort() >>> x ['A', 'B', 'a', 'b']
using map
>>> ','.join(map(str,[1,2,3])) '1,2,3'
getsizeof pitfalls
sys.getsizeof() does not add up the size of data that is referenced. For example
In [1]: import sys ...: a = [1]*1000 ...: b = [2]*1000000 ...: c = {}; c['a'] = a ...: d = {}; d['a'] = a; d['b'] = b
In [2]: [sys.getsizeof(a), sys.getsizeof(b), sys.getsizeof(c), sys.getsizeof(d)] Out[2]: [8064, 8000064, 240, 240]
When counting the size of c and d, the size of a, b are not included since the dictionaries contain those values via references. That is why we get the same size for c and d.
n choose r
tags | nCr
In [1]: from scipy.special import comb ...: comb(12,3) Out[1]: 220.0
print a floating point number as dollars and cents
This prints
- a '$' before a positive number and '-$' before a negative number
- rounds it to two decimal points
- adds comma as a thousands separator
"{}${:,.2f}".format(["","-"][amount<0], abs(amount))
For example
>>> amount = float('214498.63723333334') >>> "{}${:,.2f}".format(["","-"][amount<0], abs(amount)) '$214,498.64' >>> amount = float('-214498.63723333334') >>> "{}${:,.2f}".format(["","-"][amount<0], abs(amount)) '-$214,498.64'
If the '$' symbol is not needed
"{:,.2f}".format(amount)
>>> amount = float('214498.63723333334') >>> "{:,.2f}".format(amount) '214,498.64' >>> amount = float('-214498.63723333334') >>> "{:,.2f}".format(amount) '-214,498.64'
If neither the '$' symbol nor the thousands separator are needed
"{:.2f}".format(amount)
>>> amount = float('214498.63723333334') >>> "{:.2f}".format(amount) '214498.64' >>> amount = float('-214498.63723333334') >>> "{:.2f}".format(amount) '-214498.64'
Ref:- https://stackoverflow.com/questions/21208376/converting-float-to-dollars-and-cents
floating point pitfalls
$ ipython In [1]: x = 651370000000 ...: x / 3.1416 * 3.1416 Out[1]: 651370000000.0 In [2]: invd = 1 / 3.1416 ...: x * invd * 3.1416 Out[2]: 651370000000.0001
Ref:- https://lemire.me/blog/2019/03/12/multiplying-by-the-inverse-is-not-the-same-as-the-division/
latest python release
Ref:- https://www.python.org/downloads/
- Python 3.7.4 released 2019-07-08 (checked on 2019-09-16)
- Python 3.6.9 released 2019-07-02 (checked on 2019-09-16)
- Python 2.7.16 released 2019-03-04 (checked on 2019-09-16)
useful articles
- https://datascience.blog.wzb.eu/2018/02/02/vectorization-and-parallelization-in-python-with-numpy-and-pandas/ - Vectorization and parallelization in Python with numpy and pandas. Very easy to read, explains concepts very well.
- http://nedbatchelder.com/text/names1.html - article on how names, values, assignment and mutability works in Python. The concepts are explained very well. Highly recommended for C/C++ programmers looking to learn Python.
- https://stackoverflow.com/questions/20021693/how-to-pass-argparse-arguments-to-a-class - contains some ideas on how to use argparse in OOP context.
useful links
- https://wiki.python.org/moin/HowTo/Sorting - Easy to read and understand, well written.
- logging tutorial
- https://docs.python.org/2/howto/logging.html - for python 2
- https://docs.python.org/3/howto/logging.html - for python 3
- Run a single unittest from command line - https://stackoverflow.com/questions/15971735/running-single-test-from-unittest-testcase-via-command-line
- Disable one unittest temporarily - https://stackoverflow.com/questions/2066508/disable-individual-python-unit-tests-temporarily
- https://www.dataquest.io/blog/python-generators-tutorial/ - Python Generators Tutorial. It explains the concepts very well. A bit wordy at time but easy to understand.
- https://www.programiz.com/python-programming/decorator - python decorators
- https://www.programiz.com/python-programming/property - using @property in OOP with Python
- string formatting - https://docs.python.org/3/library/string.html
radix 2 representation of a number
format(n, 'b')
>>> format(13, 'b') '1101' >>> type(format(13, 'b')) <class 'str'>
tags | convert an integer to binary format
To do it the other way (convert a binary string to integer)
int(s, 2)
>>> int('1101', 2) 13
notes on sys.argv
- argv[0] - script name
- len(argv) = number of arguments.
For example, if you call a python script as
foo.py arg1 arg2
len(argv) will be 3. argv[0] = foo.py, argv[1] = arg1, argv[2] = arg2.
hash string to integer
tags | deterministic hashing, cut length of hash to N digits
Starting from Python 3.3, hash randomization is turned on by default as a security feature. A random seed is set when python process is started. As a result, the python hash() will return different values on each run. For example
$ python -c "print(hash('Coffee'))" -6802492772937910521 $ python -c "print(hash('Coffee'))" 2506337549030620102 $ python --version Python 3.6.5 :: Anaconda, Inc.
To do it in a deterministic way:
from hashlib import md5 def get_hash(s, length=None): h = md5(str(s).encode('utf-8')).hexdigest() if length: h = h[:length] res = int(h, base=16) return res
Sample usage:
In [2]: get_hash('Coffee') Out[2]: 79804158907860816670176857872944973754 In [3]: get_hash('Coffee', length=10) Out[3]: 257861144725 In [4]: get_hash('Coffee', length=5) Out[4]: 245915
Ref:-
- https://docs.python.org/3/library/hashlib.html
- pandas.util.hash_array - Given a 1d array, returns an array of deterministic integers.
https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.util.hash_array.html
- pandas.util.hash_pandas_object - Return a data hash of the Index/Series/DataFrame
https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.util.hash_pandas_object.html
Tasks
first common element
tags | find the first element common to two lists
Q: Given two lists x, y find the first element in x that is present in y?
A: This does it in O(N) time with a set (hash map)
next((i for i in x if i in set_y), None)
Sample run:
$ ipython In [1]: x = [1,2,3,4] ...: y = [2,5,6,7] ...: set_y = set(y) ...: c = next((i for i in x if i in set_y), None) ...: print(c) 2 In [2]: x = [1,3,4,5] ...: c = next((i for i in x if i in set_y), None) ...: print(c) 5 In [3]: x = [1,3,4,8] ...: c = next((i for i in x if i in set_y), None) ...: print(c) None
Ref:-
- https://stackoverflow.com/questions/16118621/first-common-element-from-two-lists - jamylak comment contains some code that times various approaches and shows that the above is fastest under different scenarios.
- https://www.programiz.com/python-programming/methods/built-in/next
get file extension
os.path.splitext(x)[1]
Experiments:
$ ipython Python 3.6.5 |Anaconda, Inc.| (default, Mar 29 2018, 13:32:41) [MSC v.1900 64 bit (AMD64)] Type 'copyright', 'credits' or 'license' for more information IPython 6.4.0 -- An enhanced Interactive Python. Type '?' for help. In [1]: import os ...: files = ['foo.pdf', 'foo.', 'foo', ...: '/dir/foo.pdf', '/dir/foo.', '/dir/foo', ...: '/dir.fun/foo.pdf', '/dir.fun/foo.', '/dir.fun/foo', ...: '.foo', '.foo.bar'] ...: roots = [os.path.splitext(x)[0] for x in files] ...: exts = [os.path.splitext(x)[1] for x in files] ...: for i in zip(files, roots, exts): ...: print(i) ...: ('foo.pdf', 'foo', '.pdf') ('foo.', 'foo', '.') ('foo', 'foo', '') ('/dir/foo.pdf', '/dir/foo', '.pdf') ('/dir/foo.', '/dir/foo', '.') ('/dir/foo', '/dir/foo', '') ('/dir.fun/foo.pdf', '/dir.fun/foo', '.pdf') ('/dir.fun/foo.', '/dir.fun/foo', '.') ('/dir.fun/foo', '/dir.fun/foo', '') ('.foo', '.foo', '') ('.foo.bar', '.foo', '.bar')
Ref:- https://docs.python.org/3/library/os.path.html#os.path.splitext
which package
Read and write parquet files
pyarrow
Links
- Documentation - https://arrow.apache.org/docs/python/
- Source code - https://github.com/apache/arrow.
Notes:
- No support for avro.
Read and write avro files
fastavro
Links
- Documentation - https://fastavro.readthedocs.io/
- Source code - https://github.com/fastavro/fastavro
Notes:
- faster
Virtual Environments
The PyPI version of virtualenv works in most environments. As of Python 3.3, the venv virtual environment module is included as part of the standard library. However, some problems with venv have been reported on Ubuntu. Since virtualenv works with Python 3.6 (and as far back as Python 2.6) and on Ubuntu, use virtualenv.
Setup virtual environment
Using pip in Linux
pip install -U virtualenv virtualenv -p /path/to/a/python.exe /path/to/env_name source /path/to/env_name/bin/activate # do your work dectivate
Uisng Python in Linux
python3.6 -m pip install -U virtualenv python3.6 -m virtualenv env_name source env_name/bin/activate # do your work dectivate
Using pip in Windows
pip install -U virtualenv virtualenv -p /path/to/a/python.exe /path/to/env_name /path/to/env_name/Scripts/activate.bat # do your work dectivate
Using Python in Windows
python3.6 -m pip install -U virtualenv python3.6 -m virtualenv env_name env_name/Scripts/activate.bat # do your work dectivate
check if
check if a file exists
import os if not os.path.isfile(ofile): print("File", ofile, "does not exist")
Ref:- http://stackoverflow.com/questions/82831/how-do-i-check-whether-a-file-exists-using-python
check if a key is exists in a dictionary
if 'key1' in dict: print "blah" else: print "boo"
Ref: http://stackoverflow.com/questions/1602934/check-if-a-given-key-already-exists-in-a-dictionary
check if two values are not none
if all(v is not None for v in [A, B, C, D, E]):
check if something is not None
In:
a = None if a is not None: print('kama') else: print('raju')
Out:
raju
In:
a = 2 if a is not None: print('kama') else: print('raju')
Out:
kama
check if a list is sorted
% cat is_list_sorted.py def is_list_sorted(l): return all( [l[i] < l[i+1] for i in range(len(l)-1)] ) a = [10,1,8,2,5,3,7,9,6,4] print(is_list_sorted(a)) b = sorted(a) print(is_list_sorted(b))
% python3 -u is_list_sorted.py False True
Ref:- https://stackoverflow.com/questions/3755136/pythonic-way-to-check-if-a-list-is-sorted-or-not
check if a string can be converted to float
def is_float_by_except(s): try: float(s) return True except ValueError: return False
check if a rest server is accessible
import requests def test_rest_server(url): # url will be of the form # http://host.name:port res = requests.get(url) res.raise_for_status() # Raises a HTTPError if the status is 4xx, 5xx
See also:-
how if condition evaluates it
tags | check if something evaluates to true or false, check if something is none
Takeaway:
None, False, 0, '', [] are all evaluated as False in an if condition.
Experiment:
def is_true(x): if x: return True else: return False def is_none(x): if x is None: return True else: return False def is_false(x): if not x: return True else: return False def is_not_none(x): if x is not None: return True else: return False a = [None, False, 0, '', [], True, ' ', 1, ['']] print('{!s:5}'.format('elem'), '{!s:7}'.format('is_true'), '{!s:7}'.format('is_none'), '{!s:8}'.format('is_false'), '{!s:11}'.format('is_not_none'), ) for x in a: print('{!r:5}'.format(x), '{!r:7}'.format(is_true(x)), '{!r:7}'.format(is_none(x)), '{!r:8}'.format(is_false(x)), '{!r:11}'.format(is_not_none(x)) )
Output:
elem is_true is_none is_false is_not_none None False True True False False False False True True 0 False False True True '' False False True True [] False False True True True True False False True ' ' True False False True 1 True False False True [''] True False False True
programming notes
string formatting in print statements
tags | specify width of a string
Code snippet:
a = [None, False, 0, '', [], True, ' ', 1, ['']] print('{!s:5}'.format('elem'), '{!s:7}'.format('is_true'), '{!s:7}'.format('is_none'), '{!s:8}'.format('is_false'), '{!s:11}'.format('is_not_none'), ) for x in a: print('{!r:5}'.format(x), '{!r:7}'.format(is_true(x)), '{!r:7}'.format(is_none(x)), '{!r:8}'.format(is_false(x)), '{!r:11}'.format(is_not_none(x)) )
Output:
elem is_true is_none is_false is_not_none None False True True False False False False True True 0 False False True True '' False False True True [] False False True True True True False False True ' ' True False False True 1 True False False True [''] True False False True
where the functions is_true, is_none etc., are defined in https://github.com/KamarajuKusumanchi/notebooks/blob/master/python/how%20if%20condition%20evaluates%20it.ipynb
string formatting in assert statements
In [1]: assert '10' == 10, 'not equal {} blah {}'.format(10, 1) --------------------------------------------------------------------------- AssertionError Traceback (most recent call last) <ipython-input-1-31a4cefc7818> in <module>() ----> 1 assert '10' == 10, 'not equal {} blah {}'.format(10, 1) AssertionError: not equal 10 blah 1 In [2]: assert '10' == 10, 'not equal {} blah {}'.format(10, '100') --------------------------------------------------------------------------- AssertionError Traceback (most recent call last) <ipython-input-2-8bee5a255efa> in <module>() ----> 1 assert '10' == 10, 'not equal {} blah {}'.format(10, '100') AssertionError: not equal 10 blah 100 In [3]: assert '10' == 10, 'not equal %d blah %s' % (10, '100') --------------------------------------------------------------------------- AssertionError Traceback (most recent call last) <ipython-input-3-fa41c2fc7bb1> in <module>() ----> 1 assert '10' == 10, 'not equal %d blah %s' % (10, '100') AssertionError: not equal 10 blah 100
See also:- https://www.geeksforgeeks.org/python-format-function/
pretty format list comprehension statements
import pandas as pd week_days = [dt.strftime('%Y%m%d') for dt in pd.date_range(end='20200101', periods=8) if dt.weekday() < 5] print(week_days)
Ref:- https://stackoverflow.com/questions/311588/how-to-indent-python-list-comprehensions
script execution
To execute a python script in the interpreter
exec(open('myscript.py').read(), globals())
But this does not allow to pass any arguments to the script. For that use
import sys import subprocess subprocess.call([sys.executable, 'foo.py', arg1])
For windows, the filepath can be specified as 'C:/path/to/foo.py'
lambda if
Ref:- https://stackoverflow.com/questions/1585322/is-there-a-way-to-perform-if-in-pythons-lambda
Task:- Copy files in a dataframe column to a destination folder.
We need to check if the file exists before copying it since otherwise copy_file will throw an error.
from distutils.file_util import copy_file import os os.makedirs(out_dir) df['file'].apply(lambda x: copy_file(x, out_dir) if os.path.isfile(x) else None)
integer division
Integer division in Python returns the floor of the result instead of truncating towards zero like C. The reason behind this choice is explained in http://python-history.blogspot.com/2010/08/why-pythons-integer-division-floors.html .
Consider for example integer_division.py and integer_division.cpp
% python ./integer_division.py a, b, a//b, a%b 5, 2, 2, 1 -5, 2, -3, 1 5, -2, -3, -1 -5, -2, 2, -1 % g++ ./integer_division.cpp % ./a.out a, b, a/b, a%b 5, 2, 2, 1 -5, 2, -2, -1 5, -2, -2, 1 -5, -2, 2, -1
regex for comma separated integers
search | comma delimited integers
import re re.search('(\d+(?:,\d+)*)', "1,2,3,4").groups() ('1,2,3,4',) re.search('(\d+(?:,\d+)*)', "1,2,3,4f").groups() ('1,2,3,4',)
If non capturing groups are not used, then you get
re.search('(\d+(,\d+)*)', "1,2,3,4").groups() ('1,2,3,4', ',4')
The inner group gets the latest match
re.search('(\d+(,\d+))', "1,2,3,4").groups() ('1,2', ',2') re.search('(\d+(,\d+){2})', "1,2,3,4").groups() ('1,2,3', ',3') re.search('(\d+(,\d+)*)', "1,2,3,4").groups() ('1,2,3,4', ',4')
regex to check for dates
tags | check for digit
>>> import re >>> file = 'foo_20200321.txt' >>> file_split = re.split(r'(\d{8})', file, 1) >>> file_split ['foo_', '20200321', '.txt']
Notes:
- The 'r' in the front of r'(\d{8})' tells Python that it is a raw string literal. They are explained in https://stackoverflow.com/questions/2081640/what-exactly-do-u-and-r-string-flags-do-and-what-are-raw-string-literals
sort run_1 run_2 run_10
tags | pass custom function to sorted
The task here is to sort a list of strings of form 'run_N' on N where N is an integer. For example, if we have ['run_9', 'run_2', 'run_10', 'run_1'], the output should be ['run_1', 'run_2', 'run_9', 'run_10']
The default string sorting using the sorted function will not work since 'run_10' is smaller than 'run_2' in pure string comparison.
$ ipython In [1]: a = ['run_9', 'run_2', 'run_10', 'run_1'] In [2]: sorted(a) Out[2]: ['run_1', 'run_10', 'run_2', 'run_9']
Instead one has to extract the integer, N, and use it as the key.
In [1]: a = ['run_9', 'run_2', 'run_10', 'run_1'] In [2]: import re In [3]: pattern = 'run_' + '(\d+)$' In [4]: sorted(a, key=lambda x: int(re.search(pattern, x).group(1))) Out[4]: ['run_1', 'run_2', 'run_9', 'run_10']
extract something out of a string
% python3 Python 3.5.3 (default, Sep 27 2018, 17:25:39) [GCC 6.3.0 20170516] on linux >>> import re >>> a = "GeoIP Country Edition: US, United States" >>> b = "blah blah" >>> pattern = "GeoIP Country Edition: (\w+), " >>> re.match(pattern, a) <_sre.SRE_Match object; span=(0, 27), match='GeoIP Country Edition: US, '> >>> re.match(pattern, b) >>> re.match(pattern, a) is None False >>> re.match(pattern, b) is None True >>> re.search(pattern, a) <_sre.SRE_Match object; span=(0, 27), match='GeoIP Country Edition: US, '> >>> re.search(pattern, b) >>> re.search(pattern, a).group(1) 'US' >>> re.search(pattern, b).group(1) Traceback (most recent call last): File "<stdin>", line 1, in <module> AttributeError: 'NoneType' object has no attribute 'group' >>> c = re.search(pattern, a).group(1) if re.match(pattern, a) else None >>> c 'US' >>> c = re.search(pattern, b).group(1) if re.match(pattern, b) else None >>> c >>> c is None True
reload a module
For python 3
import importlib importlib.reload(modulename)
For python 2
reload(foo)
Note:- if you are importing functions from a module using
from foo import *
then reload(foo) will throw a NameError.
NameError: name 'foo' is not defined
This happens because there is no object bound to foo. As a work around, I do
from foo import * import foo reload(foo)
See https://stackoverflow.com/questions/13455836/nameerror-when-using-reload for more details.
split string and trim white space
s = "blah, lots , of , spaces, here " [x.strip() for x in s.split(',')] Out[18]: ['blah', 'lots', 'of', 'spaces', 'here']
Ref:- http://stackoverflow.com/questions/4071396/split-by-comma-and-strip-whitespace-in-python
convert two lists into a dictionary
Use dict with zip to construct a dictionary from two lists. The number of elements in dictionary will be the minimum of the number of elements in each list. The extra elements will be discarded.
% python3 Python 3.5.2+ (default, Sep 22 2016, 12:18:14) [GCC 6.2.0 20160914] on linux Type "help", "copyright", "credits" or "license" for more information. >>> a = [1, 2, 3]; b = ['a', 'b', 'c']; c = dict(zip(a, b)); print(a); print(b); print(c) [1, 2, 3] ['a', 'b', 'c'] {1: 'a', 2: 'b', 3: 'c'} >>> a = [1, 2, 3, 4]; b = ['a', 'b', 'c']; c = dict(zip(a, b)); print(a); print(b); print(c) [1, 2, 3, 4] ['a', 'b', 'c'] {1: 'a', 2: 'b', 3: 'c'} >>> a = [1, 2, 3]; b = ['a', 'b', 'c', 'd']; c = dict(zip(a, b)); print(a); print(b); print(c) [1, 2, 3] ['a', 'b', 'c', 'd'] {1: 'a', 2: 'b', 3: 'c'}
debug from command line
To pass command line arguments and invoke the debugger directly
python -u -m pdb mymodule.py --arg1_key arg1_val --arg2_key arg2_val
To do it inside python interpreter
import pdb import mymodule pdb.run('mymodule.myfunc()')
Ref:-
tags | Debugging python programs from the command line
step through a python script
tags | debug Add the following lines at the beginning of the script
import pdb pdb.set_trace()
Ref:- https://pythonconquerstheuniverse.wordpress.com/2009/09/10/debugging-in-python/
returning values
One way to return multiple values in a function is to use a dictionary.
return {'key1':value1,'key2':value2,'key3':value3}
or
result = {'key1':value1} result['key2'] = value2 result['key3'] = value3 return(result)
Ref:-
- https://stackoverflow.com/a/9752970/6305733 shows many ways on how to return multiple values from a function.
if else if syntax
if x < 0: x = 0 print('Negative changed to zero') elif x == 0: print('Zero') elif x == 1: print('Single') else: print('More')
check for missing data
if (x is not None and x > 0): print('x is not none and positive')
using if else in a line
>>> a = 5 if 2 > 3 else 4 >>> a 4 >>> a = 5 if 3 > 2 else 4 >>> a 5
interval comparison
Python supports chained comparisons such as x < y <= z. It is equivalent to x < y and y <= z, except that y is evaluated only once (but in both cases z is not evaluated at all when x < y is found to be false).
In general, if a, b, c, …, y, z are expressions and op1, op2, …, opN are comparison operators, then a op1 b op2 c ... y opN z is equivalent to a op1 b and b op2 c and ... y opN z, except that each expression is evaluated at most once.
Note that a op1 b op2 c doesn’t imply any kind of comparison between a and c, so that, e.g., x < y > z is perfectly legal (though perhaps not pretty).
Ref:-
- https://docs.python.org/3/reference/expressions.html#comparisons
- https://stackoverflow.com/questions/18755059/is-abc-valid-python
- https://stackoverflow.com/questions/24436150/how-does-interval-comparison-work
- https://stackoverflow.com/questions/13628791/how-do-i-check-whether-an-int-is-between-the-two-numbers
tags | a < b < c, check if a value is in between two values, multiple logical operators
Remove duplicates
To remove duplicates from a column of a dataframe
unique_id = df['id'].drop_duplicates()
Ref:- http://pandas.pydata.org/pandas-docs/version/0.17.0/generated/pandas.Series.drop_duplicates.html
tags | unique
Replace newline characters with space
% python3 Python 3.4.3 (default, Mar 3 2015, 15:56:43) [GCC 4.4.7 20120313 (Red Hat 4.4.7-4)] on linux >>> a = '''kama ... raju''' >>> a 'kama\nraju' >>> a.replace("\n", " ") 'kama raju'
overlay one column with another
Say you have two columns 'a' and 'b' in a dataframe. To copy the values from column 'a' to column 'b' whenever the values in column 'b' are missing.
df.loc[ pd.isnull(df['b']), 'b'] = \ df.loc[ pd.isnull(df['b']), 'a']
tags | pandas update data based on a condition, indexing and selecting data, find and replace values in a python dataframe, replace values if null, assign values by index, search and replace data based on another column using python pandas, example on using loc, python pandas boolean array filter
specify the path of a file
To specify the path of a file relative to the location of the module
fname = os.path.join(os.path.dirname(__file__), 'foo.txt')
__file__ gives the path of current module.
show all variables in python interpreter
use dir()
>>> dir() ['__builtins__', '__doc__', '__loader__', '__name__', '__package__', '__spec__'] >>> import pandas as pd >>> dir() ['__builtins__', '__doc__', '__loader__', '__name__', '__package__', '__spec__', 'pd'] >>> df = pd.DataFrame([[1, 2], [3, 4]], columns=list('AB')) >>> dir() ['__builtins__', '__doc__', '__loader__', '__name__', '__package__', '__spec__', 'df', 'pd'] >>> df A B 0 1 2 1 3 4
sql query string
sql_query = ''' select * from alpha..beta where foo in ('{foo}') and bar in ('{boo}') ''' res = db_query(sql_query.format(foo = "', '".join(foo), bar = "', '".join(bar)) )
where db_query is a custom function to query the database.
connecting to databases
Using pyodbc to connect to MSSQL
import pyodbc import pandas as pd # declare user, password, server, database conn_str = 'DRIVER={SQL Server};SERVER=' + server + ';DATABASE=' + database if user is not None and password is not None: conn_str += ';UID=' + user + ';PWD=' + password else: conn_str += ';Trusted_Connection=yes' connection = pyodbc.connect(conn_str) data = pd.io.sql.read_sql(<SQL goes here>, connection)
Using SQLAlchemy
from urllib import quote_plus as urlquote import sqlalchemy as sqla import pandas as pd def create_engine(conn_str): url = sqla.engine.url.make_url(conn_str) if url.drivername == 'oracle+cx_oracle': return sqla.create_engine(url, arraysize=100000) else: return sqla.create_engine(url)
Then for MSSQL
# declare user, password, server, database driver = 'ODBC Driver 11 for SQL Server' if user is not None and password is not None: conn_str = "mssql+pyodbc://{}:{}@{}/{}?driver={}".format( user, urlquote(password), server, database, urlquote(driver) ) else: # use windows authentication conn_str = "mssql+pyodbc://{}/{}?driver={};trusted_connection=yes".format( server, database, urlquote(driver) ) engine = create_engine(conn_string) data = pd.read_sql_query('<SQL goes here>', engine)
and for Oracle
# declare user, password, server conn_str = "oracle+cx_oracle://{}:{}@{}".format( user, urlquote(password), server ) engine = create_engine(conn_string) data = pd.read_sql_query('<SQL goes here>', engine)
writing an if else condition in list comprehension
Syntax
[ a if C else b for i in items]
is equivalent to
for i in items: if (C): a else: b
For example, to figure out if a package is installed one can do
import pandas as pd import apt cache = apt.Cache() a = pd.DataFrame({'package': ['python-apt', 'foo', 'dpkg', '0xffff']}) status = [ True if ((pkg in cache) and cache[pkg].is_installed) else False for pkg in a['package'] ] print(status)
which will show
[True, False, True, False]
where python-apt, dpkg are installed on the system, '0xffff' is a valid package but not installed and 'foo' is not a valid package name.
Ref: http://stackoverflow.com/questions/4406389/if-else-in-a-list-comprehension
Stackoverflow links
- Same function name with different arguments | method overloading | variable number of arguments - http://stackoverflow.com/a/5079766/6305733
get the username
>>> import getpass >>> getpass.getuser() 'rajulocal'
exit with error
One way is to throw an exception. See http://stackoverflow.com/questions/2052390/manually-raising-throwing-an-exception-in-python for more details.
exit with control d in git bash
Use ipython to exit using control d in git bash.
control d does not work with regular python. There you have to use either exit() or ctrl z <enter>.
enter multiline statements in ipython
args and kwargs
args = arguments kwargs = keyword arguments
def f(*args, **kwargs): print('args: ', args, ' kwargs: ', kwargs) >>> f('a') args: ('a',) kwargs: {} >>> f(ar='a') args: () kwargs: {'ar': 'a'} >>> f(1,2,param=3) args: (1, 2) kwargs: {'param': 3} >>> f(1,[2,3],param=3) args: (1, [2, 3]) kwargs: {'param': 3} >>> f(1, [2,3], [4,5,6], param=3, first='kama', last='raju') args: (1, [2, 3], [4, 5, 6]) kwargs: {'param': 3, 'last': 'raju', 'first': 'kama'}
See also: http://stackoverflow.com/questions/682504/what-is-a-clean-pythonic-way-to-have-multiple-constructors-in-python?noredirect=1&lq=1
using kargs
Sample usage
def foo(alpha, beta, **kwargs): debug = kwargs.get('debug', False)
multiline statements
df = pd.read_csv(fname)\ .pipe(func_foo)\ .pipe(func_bar)
calling python script from bash
print stack trace
To print stack trace from within the program
import traceback print 'printing stack' traceback.print_stack()
Ref:- https://docs.python.org/2/library/traceback.html
print exception and continue
http://www.kamaraju.xyz/dk/python_notes#print_exception_and_continue
print exception to log file
http://www.kamaraju.xyz/dk/python_notes#print_exception_to_log_file
parse config files
Example 1:
% cat config.ini [COUNTRIES] DENY = AG AL ALLOW = IN US
% cat config.py import configparser import os Config = configparser.ConfigParser() Config.read(os.path.join(os.path.dirname(__file__), 'config.ini'))
To use it
% python3 Python 3.5.3 (default, Jan 19 2017, 14:11:04) [GCC 6.3.0 20170118] on linux Type "help", "copyright", "credits" or "license" for more information. >>> from config import Config >>> Config['COUNTRIES']['DENY'] 'AG AL' >>> Config['COUNTRIES']['ALLOW'] 'IN US'
Porting windows code to Linux
Things to look out for when porting python code from windows to Linux
- use os.path.sep instead of "\\" for parsing directories in a filepath
- use os.path.join when constructing file name. For example instead of dir_name + "\\" + file_name, use os.path.join(dir_name, file_name)
- exception during reset or similar
Download a webpage
version 1:-
import requests def get_html(url): # Get html from url response = requests.get(url) return response.text
Version 2:- Same as version 1, but has user_agent hardcoded
import requests def get_html(url): # Get html from url user_agent = 'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.139 Safari/537.36' header = {'User-Agent': user_agent} response = requests.get(url, headers=header) return response.text
what is **{}
$python Python 3.6.6 |Anaconda, Inc.| (default, Jun 28 2018, 11:27:44) [MSC v.1900 64 bit (AMD64)] on win32 >>> def foo(**bar): ... print(bar) ... >>> foo(**{'a':2}) {'a': 2} >>> foo(**{'a':2, 'b':3}) {'a': 2, 'b': 3} >>> foo() {}
See:-
- https://docs.python.org/dev/tutorial/controlflow.html#keyword-arguments
- https://docs.python.org/dev/tutorial/controlflow.html#arbitrary-argument-lists
- https://docs.python.org/dev/tutorial/controlflow.html#unpacking-argument-lists
using timeit
$ python -mtimeit "l=[]" 10000000 loops, best of 3: 0.0274 usec per loop $ python -mtimeit "l=list()" 10000000 loops, best of 3: 0.125 usec per loop
pep8 recommendations
- use 4-space indentation, and no tabs
- 79 characters per line
- use CamelCase for classes and lower_case_with_underscores for functions and methods
See:-
python web interpreters
- https://repl.it - Try this at home. Does it work?
- https://pypyjs.org - fast. Pandas package is not installed.
- http://www.codeskulptor.org/ - no pandas
- http://py3.codeskulptor.org/ - no pandas
tags | globals, singleton, variables shared across modules
See | https://instructobit.com/tutorial/108/How-to-share-global-variables-between-files-in-Python
See also
- https://docs.python.org/3/faq/programming.html#how-do-i-share-global-variables-across-modules
- https://stackoverflow.com/questions/13034496/using-global-variables-between-files
- https://stackoverflow.com/questions/10851906/python-3-unboundlocalerror-local-variable-referenced-before-assignment
os.makedirs
os.makedirs is like "mkdir -p". It will create intermediate directories if necessary.
get username in windows
os.getlogin()
returns the user that is executing the script.
Extract first N elements from a generator
$python Python 3.6.8 |Anaconda, Inc.| (default, Dec 30 2018, 01:22:34) [GCC 7.3.0] on linux >>> generator = (i for i in range(10)) >>> type(generator) <class 'generator'> >>> list(next(generator) for _ in range(4)) [0, 1, 2, 3] >>> list(next(generator) for _ in range(4)) [4, 5, 6, 7] >>> list(next(generator) for _ in range(4)) [8, 9] >>> list(next(generator) for _ in range(4)) []
Notice how the last two calls give only the remaining elements. If you use [] instead of list(), a StopIteration exception will be thrown in those cases.
>>> generator = (i for i in range(10)) >>> [next(generator) for _ in range(4)] [0, 1, 2, 3] >>> [next(generator) for _ in range(4)] [4, 5, 6, 7] >>> [next(generator) for _ in range(4)] Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 1, in <listcomp> StopIteration
timing a function
start_time = time.time() call_function_of_interest() end_time = time.time() print('time elapsed = ', end_time - start_time)
For python2
print 'time elapsed = ', end_time - start_time print 'took %.f sec to finish' %(end_time - start_time)
For Python 3 using logger
start = time.time() call_function_of_interest() elapsed = time.time() - start logger.info('function finished in %.2f seconds' % elapsed)
tags | time a function, tic toc
grep of git status
tags | git status grep cut subprocess Popen PIPE, demonstrates | convert bytes to string
def grep_on_git_status(): git_st = Popen(['git', 'status', '--porcelain'], stdout=PIPE, stderr=PIPE) grep = Popen(['grep', '^??'], stdin=git_st.stdout, stdout=PIPE) cut = Popen(['cut', '-f', '2', '-d', ' '], stdin=grep.stdout, stdout=PIPE) # The cut.stdout is a gigantic byte string. # The readlines() will break it into a listof byte strings files = cut.stdout.readlines() # strip the newline characters at the end files = [x.decode('utf-8').strip() for x in files] return files files = grep_on_git_status() print(type(files)) print(files)
Ref:-
- https://stackoverflow.com/questions/22115629/cut-command-in-python-via-subprocess-module - tells that there is no need to double quote the space character in the cut command.
notes on subprocess.run
- https://github.com/KamarajuKusumanchi/notebooks/blob/master/python/notes%20on%20subprocess.run.ipynb
- https://docs.python.org/3/library/subprocess.html
pretty formatting subprocess commands
import subprocess pkges = ['pandas', 'numpy'] for pkg in pkges: cmd = ['conda', 'list', '-n', 'base', pkg] result = subprocess.run(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=True) print(result.stdout)
sample output:
# packages in environment at C:\ProgramData\Continuum\Anaconda: # # Name Version Build Channel pandas 0.23.0 py36h830ac7b_0 # packages in environment at C:\ProgramData\Continuum\Anaconda: # # Name Version Build Channel numpy 1.14.3 py36h9fa60d3_1 numpy-base 1.14.3 py36h555522e_1 numpydoc 0.8.0 py36_0
Code snippets
Add a directory to the beginning of PATH in Windows
backup_path = os.environ['PATH'] if sys.platform == 'win32': new_dir = os.path.join('foo', 'bar') os.environ['PATH'] = "%s;%s" % (new_dir, backup_path)
small experiments
infinity
% python3 Python 3.5.3 (default, Jan 19 2017, 14:11:04) [GCC 6.3.0 20170118] on linux Type "help", "copyright", "credits" or "license" for more information. >>> float('inf') inf >>> type(float('inf')) <class 'float'> >>> float('inf') > 5 True >>> float('inf') > -float('inf') True >>> float('inf') is float('inf') False >>> float('inf') == float('inf') True
logical operators
x or y, x and y, not x
Example
if (a < 1 or a > 100): print('a is not in [1,100]')
Ref:- https://docs.python.org/3/library/stdtypes.html
bitwise logic operators
&, |, ^, and ~
Example
np.sum((inches > 0.5 & (inches < 1))
Parenthesis are important here because of operator precedence rules. Without the parenthesis, the expression will be evaluated as Example
inches (> 0.5 &) < 1
Another way to express the above logic
np.sum(~( (inches <= 0.5) | (inches >= 1) ))
sinc A AND B is same as NOT (NOT A OR NOT B)
operator precedence
+= and -=
These (+=, -=, *= and /=) are called augmented arithmetic assignments. They are described in https://docs.python.org/3/reference/datamodel.html#emulating-numeric-types -> scroll down to the section on __iadd__, __isub__ functions.
the i semantically means "in-place", which means that they modify the object (or reference in the case of numerics) without having to additionally assign them.
conditional operator
tags | conditional expressions, ternary
a if condition else b
What is the difference between <> and !=
In python 3, there is no <> operator.
In python 2, they both mean the same. != is preferred; <> is obsolescent.
Ref:-
- https://docs.python.org/2.7/reference/lexical_analysis.html#operators
- https://stackoverflow.com/questions/40211270/is-there-a-difference-between-and-operators-in-python
numpy
Frequently used numpy functions
use case | solution |
average | np.average |
sum | np.sum |
referring to class variables in methods
tags | use classmethod to access class variables
$ cat foo.py class Days(): SUN = "Sunday" MON = "Monday" TUE = "Tuesday" WED = "Wednesday" THU = "Thursday" FRI = "Friday" SAT = "Saturday" @classmethod def weekdays(cls): return [cls.MON, cls.TUE, cls.WED, cls.THU, cls.FRI] @classmethod def weekends(cls): return [cls.SAT, cls.SUN]
To use them in bar.py
from foo import Days a = Days.SUN b = Days.weekdays() c = Days.weekends()
See also | https://stackoverflow.com/a/709024/6305733
Challenges
Rotate matrix anti clockwise
Given a 2-D array
[[1,2,3,4], [5,6,7,8], [9,10,11,12]]
convert it to
[[4,8,12], [3,7,11], [2,6,10], [1,5,9]]
>>> a = [[1,2,3,4], [5,6,7,8], [9,10,11,12]] >>> a [[1, 2, 3, 4], [5, 6, 7, 8], [9, 10, 11, 12]] >>> b = list(zip(*a))[::-1] >>> b [(4, 8, 12), (3, 7, 11), (2, 6, 10), (1, 5, 9)] >>> c = [list(elem) for elem in b] >>> c [[4, 8, 12], [3, 7, 11], [2, 6, 10], [1, 5, 9]]
Rotate matrix clockwise
Given a 2-D array
[[1,2,3,4], [5,6,7,8], [9,10,11,12]]
convert it to
[[9,5,1], [10,6,2], [11,7,3], [12,8,4]]
>>> a = [[1,2,3,4], [5,6,7,8], [9,10,11,12]] >>> a [[1, 2, 3, 4], [5, 6, 7, 8], [9, 10, 11, 12]] >>> b = list(zip(*a[::-1])) >>> b [(9, 5, 1), (10, 6, 2), (11, 7, 3), (12, 8, 4)] >>> c = [list(elem) for elem in b] >>> c [[9, 5, 1], [10, 6, 2], [11, 7, 3], [12, 8, 4]]
index of an item in a sorted list
Use bisect module to find out where an item can be inserted in a sorted list. The return value of bisect is suitable for use as the first parameter to list.insert() assuming that original list is already sorted.
>>> a = [10,1,8,2,5,3,7,9,6,4] >>> a [10, 1, 8, 2, 5, 3, 7, 9, 6, 4] >>> a.sort() >>> a [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] >>> x = 5.5 >>> from bisect import bisect >>> bisect(a,x) 5 >>> a.insert( bisect(a,x), x) >>> a [1, 2, 3, 4, 5, 5.5, 6, 7, 8, 9, 10]
Instead of doing bisect and insert, you can also insort which does both.
>>> a = [10,1,8,2,5,3,7,9,6,4] >>> a [10, 1, 8, 2, 5, 3, 7, 9, 6, 4] >>> a.sort() >>> a [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] >>> x = 5.5 >>> from bisect import insort >>> insort(a,x) >>> a [1, 2, 3, 4, 5, 5.5, 6, 7, 8, 9, 10]
Ref:- https://docs.python.org/3/library/bisect.html
questions
decorators
In https://github.com/RomelTorres/alpha_vantage/blob/develop/alpha_vantage/techindicators.py , I see things like
@av._output_format @av._call_api_on_func def get_sma(self, symbol, interval='daily', time_period=20, series_type='close'):
What do those @av decorators do? Read about this later.
books
Think Python
Book info:
- Think Python 2nd Edition by Allen B. Downey
- website: https://greenteapress.com/wp/think-python-2e/
The second edition uses Python 3. The first edition uses Python 2.
- http://amzn.to/1WmXPCx
- Download link: http://greenteapress.com/thinkpython2/thinkpython2.pdf
- Read online: http://greenteapress.com/thinkpython2/html/index.html
- Example programs and solutions to some exercises: https://github.com/AllenDowney/ThinkPython2/tree/master/code
Instructions for working with this code are in the preface.
Review:
- rating: 5 out of 5.
- level: beginner
- easy to read, easy to understand, no fluff, no nonsense.
- highly recommended.