Grep installed.py
From raju
Introduction
The grep_installed.py script can be used to show the list of installed packages that are in one distribution but not in another. For example, it can be used to figure out the list of installed packages that are present in Debian Stable but not in either Debian Testing or Debian Unstable. The script can handle both symbolic distribution names such as Stable/Testing/Unstable and actual distribution names such as Jessie/Stretch/Sid etc., The script can process "apt-cache search" output. The output from the script can be shown in different "report" formats.
The rest of the article shows how this script can be useful in practice.
Software
Latest version of grep_installed.py - https://gitlab.com/d3k2mk7/rutils/blob/master/bin/grep_installed.py
Simple use cases
Use case 1
To filter "apt-cache search" output and show information only on the installed packages, simply pipe the output to the script.
apt-cache search <word1> <word2> ... | grep_installed.py
For example, on my machine, there are 138 packages with python and apt in their descriptions. But only 11 of them are installed.
% apt-cache search python apt | wc -l 138 % apt-cache search python apt | grep_installed.py | wc -l 11 % apt-cache search python apt | grep_installed.py needrestart - check which daemons need to be restarted after library upgrades python-apt - Python interface to libapt-pkg python-apt-common - Python interface to libapt-pkg (locales) python-apt-doc - Python interface to libapt-pkg (API documentation) ...
Use case 2
Instead of apt-cache search, the package list can also be specified manually. For example
% echo "spyder spyder3" | tr ' ' '\n' | grep_installed.py spyder3
which shows that between spyder and spyder3, only spyder3 is installed on my machine. Since the script requires each input line to start with a pacakge name, we use tr to convert the spaces to new line characters.
Complex use cases
The examples given so far do not involve any distribution specific logic. For that we need a three step approach.
- Get the list of available packages for each distribution and cache it. This needs to be done only once per distribution unless the user wants to update the cache itself.
- Query the cache.
- (Optional) clear the cache.
Cache the list of available packages
Run the script with --update-cache to update the package lists for Debian Stable, Testing and Unstable.
% grep_installed.py --update-cache Processed: http://httpredir.debian.org/debian/dists/stable/main/binary-amd64/Packages.gz Processed: http://httpredir.debian.org/debian/dists/stable/contrib/binary-amd64/Packages.gz Processed: http://httpredir.debian.org/debian/dists/stable/non-free/binary-amd64/Packages.gz writing: /home/rajulocal/.cache/grep_installed/stable.gz Processed: http://httpredir.debian.org/debian/dists/testing/main/binary-amd64/Packages.gz Processed: http://httpredir.debian.org/debian/dists/testing/contrib/binary-amd64/Packages.gz Processed: http://httpredir.debian.org/debian/dists/testing/non-free/binary-amd64/Packages.gz writing: /home/rajulocal/.cache/grep_installed/testing.gz Processed: http://httpredir.debian.org/debian/dists/unstable/main/binary-amd64/Packages.gz Processed: http://httpredir.debian.org/debian/dists/unstable/contrib/binary-amd64/Packages.gz Processed: http://httpredir.debian.org/debian/dists/unstable/non-free/binary-amd64/Packages.gz writing: /home/rajulocal/.cache/grep_installed/unstable.gz
Here /home/rajulocal is my home directory and the output will change accordingly when another user runs it.
To update the package lists of other distributions, specify them as arguments to either --include-dists or --exclude-dists. For example, the following will update the package lists of Jessie, Stretch and Sid.
% grep_installed.py --update-cache --include-dists jessie --exclude-dists 'stretch,sid' Processed: http://httpredir.debian.org/debian/dists/jessie/main/binary-amd64/Packages.gz Processed: http://httpredir.debian.org/debian/dists/jessie/contrib/binary-amd64/Packages.gz Processed: http://httpredir.debian.org/debian/dists/jessie/non-free/binary-amd64/Packages.gz writing: /home/rajulocal/.cache/grep_installed/jessie.gz Processed: http://httpredir.debian.org/debian/dists/stretch/main/binary-amd64/Packages.gz Processed: http://httpredir.debian.org/debian/dists/stretch/contrib/binary-amd64/Packages.gz Processed: http://httpredir.debian.org/debian/dists/stretch/non-free/binary-amd64/Packages.gz writing: /home/rajulocal/.cache/grep_installed/stretch.gz Processed: http://httpredir.debian.org/debian/dists/sid/main/binary-amd64/Packages.gz Processed: http://httpredir.debian.org/debian/dists/sid/contrib/binary-amd64/Packages.gz Processed: http://httpredir.debian.org/debian/dists/sid/non-free/binary-amd64/Packages.gz writing: /home/rajulocal/.cache/grep_installed/sid.gz
Clear the cache
To clear the cache, run the script with --clear-cache option. This will remove all *.xz files in the cache directory (~/.cache/grep_installed) and the directory itself (if it is empty).
% grep_installed.py --clear-cache removing /home/rajulocal/.cache/grep_installed/testing.xz removing /home/rajulocal/.cache/grep_installed/stable.xz removing /home/rajulocal/.cache/grep_installed/stretch.xz removing /home/rajulocal/.cache/grep_installed/unstable.xz removing /home/rajulocal/.cache/grep_installed/jessie.xz removing /home/rajulocal/.cache/grep_installed/sid.xz removing /home/rajulocal/.cache/grep_installed
Use case 3
The script can be used to check the availability of a bunch of packages across different distributions. For example,
% grep_installed.py --update-cache % echo "python3.4 python3.5" | tr ' ' '\n' | grep_installed.py --report matrix | column -ts ',' data pkg is_installed stable testing unstable python3.4 python3.4 True True False False python3.5 python3.5 True False True True
shows that python3.4 is currently available in stable but not in testing or unstable where as python3.5 is the opposite (available in testing and unstable, but not in stable).
Here we pass the packages of interest one per line (tr ' ' '\n') to the script and ask it to produce a "matrix" report. The output is further processed by column (part of bddmainutils package) to pretty print it.
On systems without the column command, one can use the "tabulate" report. This relies on the tabulate module in python3-tabulate package. A sample run looks as
% grep_installed.py --update-cache % echo "python3.4 python3.5" | tr ' ' '\n' | grep_installed.py --report tabulate data pkg is_installed stable testing unstable python3.4 python3.4 1 1 0 0 python3.5 python3.5 1 0 1 1
where '1' represents True, '0' represents False.
To do the same but for different set of distributions, specify them as arguments to either of --include-dists or --exclude-dists. For example, the following will show the availability matrix for Jessie, Stretch and Sid.
% grep_installed.py --update-cache --include-dists jessie --exclude-dists 'stretch,sid' % echo "python3.4 python3.5" | tr ' ' '\n' | \ grep_installed.py --report matrix --include-dists jessie --exclude-dists 'stretch,sid' | column -ts ',' data pkg is_installed jessie stretch sid python3.4 python3.4 True True False False python3.5 python3.5 True False True True
where the first command updates the cache and the next command queries the cache.
Note that in this particular example, it does not matter whether the distributions are specified as part of --include-dists or --exclude-dists. But as you will see soon, it does matter in some other cases.
Use case 4
For a given set of packages, list all the installed packages, that are available in Stable, but are not available in either Testing or Unstable.
% grep_installed.py --update-cache % echo "python3.4 python3.5" | tr ' ' '\n' | grep_installed.py --include-dists stable --exclude-dists 'testing,unstable' python3.4
To do the same with actual distribution names where packages are in jessie but not in either stretch or sid
% grep_installed.py --update-cache --include-dists jessie --exclude-dists 'stretch,sid' % echo "python3.4 python3.5" | tr ' ' '\n' | grep_installed.py --include-dists jessie --exclude-dists 'stretch,sid' python3.4
Use case 5
After upgrading from jessie (currently stable) to stretch (currently testing), one might be left with a bunch of packages that are only available in Jessie but no longer available in Stretch or Sid. To get a list of all those outdated packages, you can do
% grep_installed.py --update-cache --include-dists jessie --exclude-dists 'stretch,sid' % dpkg -l | grep "^ii" | awk '{print $2}' | grep_installed.py --include-dists jessie --exclude-dists 'stretch,sid'
Limitations
The script assumes that the list of available packages on all architectures is the same and matches with the package list of amd64. Since this is not always the case, caution should be exercised when using it on non-amd64 architectures.
Notes for developers
Generation of package lists
The list of available packages in each distribution is obtained by parsing http://httpredir.debian.org/debian/dists/<distribution>/<section>/binary-amd64/Packages.xz and aggregating over all three sections of a distribution - main, contrib, non-free. It is then dumped as a compressed file for each distribution. For example, a typical cache directory will look as
% ls ~/.cache/grep_installed -1rt stable.xz testing.xz unstable.xz jessie.xz stretch.xz sid.xz