Grep installed.py

From raju

Introduction

The grep_installed.py script can be used to show the list of installed packages that are in one distribution but not in another. For example, it can be used to figure out the list of installed packages that are present in Debian Stable but not in either Debian Testing or Debian Unstable. The script can handle both symbolic distribution names such as Stable/Testing/Unstable and actual distribution names such as Jessie/Stretch/Sid etc., The script can process "apt-cache search" output. The output from the script can be shown in different "report" formats.

The rest of the article shows how this script can be useful in practice.

Software

Latest version of grep_installed.py - https://gitlab.com/d3k2mk7/rutils/blob/master/bin/grep_installed.py

Simple use cases

Use case 1

To filter "apt-cache search" output and show information only on the installed packages, simply pipe the output to the script.

    apt-cache search <word1> <word2> ... | grep_installed.py
    

For example, on my machine, there are 138 packages with python and apt in their descriptions. But only 11 of them are installed.

     % apt-cache search python apt | wc -l
    138
     % apt-cache search python apt | grep_installed.py | wc -l
    11
     % apt-cache search python apt | grep_installed.py
    needrestart - check which daemons need to be restarted after library upgrades
    python-apt - Python interface to libapt-pkg
    python-apt-common - Python interface to libapt-pkg (locales)
    python-apt-doc - Python interface to libapt-pkg (API documentation)
    ...
    

Use case 2

Instead of apt-cache search, the package list can also be specified manually. For example

     % echo "spyder spyder3" | tr ' ' '\n' | grep_installed.py
    spyder3
    

which shows that between spyder and spyder3, only spyder3 is installed on my machine. Since the script requires each input line to start with a pacakge name, we use tr to convert the spaces to new line characters.

Complex use cases

The examples given so far do not involve any distribution specific logic. For that we need a three step approach.

  1. Get the list of available packages for each distribution and cache it. This needs to be done only once per distribution unless the user wants to update the cache itself.
  2. Query the cache.
  3. (Optional) clear the cache.

Cache the list of available packages

Run the script with --update-cache to update the package lists for Debian Stable, Testing and Unstable.

     % grep_installed.py --update-cache
    Processed: http://httpredir.debian.org/debian/dists/stable/main/binary-amd64/Packages.gz
    Processed: http://httpredir.debian.org/debian/dists/stable/contrib/binary-amd64/Packages.gz
    Processed: http://httpredir.debian.org/debian/dists/stable/non-free/binary-amd64/Packages.gz
    writing: /home/rajulocal/.cache/grep_installed/stable.gz
    Processed: http://httpredir.debian.org/debian/dists/testing/main/binary-amd64/Packages.gz
    Processed: http://httpredir.debian.org/debian/dists/testing/contrib/binary-amd64/Packages.gz
    Processed: http://httpredir.debian.org/debian/dists/testing/non-free/binary-amd64/Packages.gz
    writing: /home/rajulocal/.cache/grep_installed/testing.gz
    Processed: http://httpredir.debian.org/debian/dists/unstable/main/binary-amd64/Packages.gz
    Processed: http://httpredir.debian.org/debian/dists/unstable/contrib/binary-amd64/Packages.gz
    Processed: http://httpredir.debian.org/debian/dists/unstable/non-free/binary-amd64/Packages.gz
    writing: /home/rajulocal/.cache/grep_installed/unstable.gz
    

Here /home/rajulocal is my home directory and the output will change accordingly when another user runs it.

To update the package lists of other distributions, specify them as arguments to either --include-dists or --exclude-dists. For example, the following will update the package lists of Jessie, Stretch and Sid.

     % grep_installed.py --update-cache --include-dists jessie --exclude-dists 'stretch,sid'
    Processed: http://httpredir.debian.org/debian/dists/jessie/main/binary-amd64/Packages.gz
    Processed: http://httpredir.debian.org/debian/dists/jessie/contrib/binary-amd64/Packages.gz
    Processed: http://httpredir.debian.org/debian/dists/jessie/non-free/binary-amd64/Packages.gz
    writing: /home/rajulocal/.cache/grep_installed/jessie.gz
    Processed: http://httpredir.debian.org/debian/dists/stretch/main/binary-amd64/Packages.gz
    Processed: http://httpredir.debian.org/debian/dists/stretch/contrib/binary-amd64/Packages.gz
    Processed: http://httpredir.debian.org/debian/dists/stretch/non-free/binary-amd64/Packages.gz
    writing: /home/rajulocal/.cache/grep_installed/stretch.gz
    Processed: http://httpredir.debian.org/debian/dists/sid/main/binary-amd64/Packages.gz
    Processed: http://httpredir.debian.org/debian/dists/sid/contrib/binary-amd64/Packages.gz
    Processed: http://httpredir.debian.org/debian/dists/sid/non-free/binary-amd64/Packages.gz
    writing: /home/rajulocal/.cache/grep_installed/sid.gz
    

Clear the cache

To clear the cache, run the script with --clear-cache option. This will remove all *.xz files in the cache directory (~/.cache/grep_installed) and the directory itself (if it is empty).

    % grep_installed.py --clear-cache
    removing /home/rajulocal/.cache/grep_installed/testing.xz
    removing /home/rajulocal/.cache/grep_installed/stable.xz
    removing /home/rajulocal/.cache/grep_installed/stretch.xz
    removing /home/rajulocal/.cache/grep_installed/unstable.xz
    removing /home/rajulocal/.cache/grep_installed/jessie.xz
    removing /home/rajulocal/.cache/grep_installed/sid.xz
    removing /home/rajulocal/.cache/grep_installed
    

Use case 3

The script can be used to check the availability of a bunch of packages across different distributions. For example,

     % grep_installed.py --update-cache
     % echo "python3.4 python3.5" | tr ' ' '\n' | grep_installed.py --report matrix | column -ts ','
    data       pkg        is_installed  stable  testing  unstable
    python3.4  python3.4  True          True    False    False
    python3.5  python3.5  True          False   True     True
    

shows that python3.4 is currently available in stable but not in testing or unstable where as python3.5 is the opposite (available in testing and unstable, but not in stable).

Here we pass the packages of interest one per line (tr ' ' '\n') to the script and ask it to produce a "matrix" report. The output is further processed by column (part of bddmainutils package) to pretty print it.

On systems without the column command, one can use the "tabulate" report. This relies on the tabulate module in python3-tabulate package. A sample run looks as

     % grep_installed.py --update-cache
     % echo "python3.4 python3.5" | tr ' ' '\n' | grep_installed.py --report tabulate
    data       pkg          is_installed    stable    testing    unstable
    python3.4  python3.4               1         1          0           0
    python3.5  python3.5               1         0          1           1
    

where '1' represents True, '0' represents False.

To do the same but for different set of distributions, specify them as arguments to either of --include-dists or --exclude-dists. For example, the following will show the availability matrix for Jessie, Stretch and Sid.

     % grep_installed.py --update-cache --include-dists jessie --exclude-dists 'stretch,sid'
     % echo "python3.4 python3.5" | tr ' ' '\n' | \
    grep_installed.py --report matrix --include-dists jessie --exclude-dists 'stretch,sid' | column -ts ','
    
    data       pkg        is_installed  jessie  stretch  sid
    python3.4  python3.4  True          True    False    False
    python3.5  python3.5  True          False   True     True
    

where the first command updates the cache and the next command queries the cache.

Note that in this particular example, it does not matter whether the distributions are specified as part of --include-dists or --exclude-dists. But as you will see soon, it does matter in some other cases.

Use case 4

For a given set of packages, list all the installed packages, that are available in Stable, but are not available in either Testing or Unstable.

     % grep_installed.py --update-cache
     % echo "python3.4 python3.5" | tr ' ' '\n' | grep_installed.py --include-dists stable --exclude-dists 'testing,unstable'
    python3.4
    

To do the same with actual distribution names where packages are in jessie but not in either stretch or sid

     % grep_installed.py --update-cache --include-dists jessie --exclude-dists 'stretch,sid'
     % echo "python3.4 python3.5" | tr ' ' '\n' | grep_installed.py --include-dists jessie --exclude-dists 'stretch,sid'
    python3.4
    

Use case 5

After upgrading from jessie (currently stable) to stretch (currently testing), one might be left with a bunch of packages that are only available in Jessie but no longer available in Stretch or Sid. To get a list of all those outdated packages, you can do

     % grep_installed.py --update-cache --include-dists jessie --exclude-dists 'stretch,sid'
     % dpkg -l | grep "^ii" | awk '{print $2}' | grep_installed.py --include-dists jessie --exclude-dists 'stretch,sid'
    

Limitations

The script assumes that the list of available packages on all architectures is the same and matches with the package list of amd64. Since this is not always the case, caution should be exercised when using it on non-amd64 architectures.

Notes for developers

Generation of package lists

The list of available packages in each distribution is obtained by parsing http://httpredir.debian.org/debian/dists/<distribution>/<section>/binary-amd64/Packages.xz and aggregating over all three sections of a distribution - main, contrib, non-free. It is then dumped as a compressed file for each distribution. For example, a typical cache directory will look as

     % ls ~/.cache/grep_installed -1rt
    stable.xz
    testing.xz
    unstable.xz
    jessie.xz
    stretch.xz
    sid.xz