Find usage

From raju

Find recent files

    find dir -type f -mtime -1 -ls
    find . -iname '*.sw*' -atime -10 -type f

Find files containing two expressions

Search the C++ source code and header files to find files that have two keywords - ifstream and string

    find /path/to/dir \( -iname '*.C' -o -iname '*.H' \) -print0  | xargs -0 grep -lZ ifstream  | xargs -0 grep -l string

All intermediate grep statements should use the -Z option so that the output from grep can be passed to "xargs -0".

list files with two different extensions

To search for both .jpg and .png files

    find /path/to/dir \( -iname '*.jpg' -o -iname '*.png' \)

To do the same in two directories

    find /path/to/dir1 /path/to/dir2 \( -iname '*.jpg' -o -iname '*.png' \)

list all files in a directory that do not end with a particular extension

=> find . -maxdepth 1 -not -iname "*.H"

find all git folders under a directory

find . -name .git -type d -prune


list a directory but exclude directories and vim swap files

find . -maxdepth 1 ! -type d ! -name "*.sw?"

grep files in a directory but exclude subdirectories, files matching a pattern

find . -maxdepth 1 ! -type d ! -name "*.sw?" ! -name "*_RK_????????_??????" ! -name "*_blame_????????_??????" -exec grep -inH --color key_word {} +

Did I ever call awk from a perl script?

rajulocal@hogwarts ~/work/rutils
 % find . -iname '*.pl' -exec grep awk -inH --color {} +
./perl/    my $cmd = "dpkg -l \*$keyword\* | grep ^ii | awk '{print \$2}'";

List all the R programs I accessed in the last 60 days

% find . -iname '*.R' -atime -60 -type f

Find all the log files and delete them

First make sure that you are matching on the correct set of files

find . -iname '*.log'

Then use any of the following commands

find . -iname '*.log' -exec rm {} +
find . -iname '*.log' -print0 | xargs -0 rm
find . -iname '*.log' -delete

The exact command depends on the installation since some versions of find do not have -print0, -delete etc., In any case, try to avoid

find . -iname '*.log' -exec rm {} \;

which spawns an external process for each and every matched file.

The advantage of using "-exec rm" approach is that you can pass additional options to the rm command. For example

find . -iname '*.log' -exec rm -i {} \;

will delete the files interactively.

tags | remove files

Aggregate data across all files modified in the last 10 minutes

 find . -mmin -10 -type f -print0 | xargs -0 tail -n +2 -q | sort | uniq

Find old files and move them to a directory

First make sure that you are matching on the correct set of files

find path/to/source/dir \( -iname 'pattern1' -o -iname 'pattern2' \) -mtime +6

Then append it with the mv command as below

find path/to/source/dir \( -iname 'pattern1' -o -iname 'pattern2' \) -mtime +6 -exec mv -ivt path/to/dest/dir {} +

Find files modified after a certain date

The code snippet below will find files modified on or after a certain date and copies them from an nfs directory to hdfs.

    # The -mtime in find is too coarse. If you have files modified 6 hours ago and
    # 8 hours ago, they can't be distinguished with "find -mtime". One way to work
    # around this is to create a temp file and look for files newer than that.
    touch --date $doi $tmp_file
    find /path/to/nfs/dir -iname '*.foo' \
    -newer $tmp_file -exec hadoop fs -put -f '{}' \
    /path/to/hdfs/dir \;
    rm -f $tmp_file

Find files newer than files in another directory

This will find all files in raw_dir that are newer than the files in processed_dir.

    find raw_dir -newer `ls processed_dir/*.txt -rt | tail -n1` -iname '*.txt'

gunzip files on nfs and copy to hadoop

    files_to_copy=`find $dir -maxdepth 1 -iname '*.avro.gz' -newer $tmp_file -print
    for i in $files_to_copy
        unzipped_file=${tmp_nfs_dir}/`basename $i | sed -e 's/\.gz$//'`
        gunzip $i -c > $unzipped_file
        hadoop fs -put -f $unzipped_file $hdfs_dir
        echo "$unzipped_file -> $hdfs_dir"
        rm -f $unzipped_file

tags | process the output of find

iterate over files from find where the files have spaces

Say we have some files with spaces in them

    bash-4.1$ ls -1
    foo (2).csv

To iterate over them, change the IFS

    bash-4.1$ OLDIFS="$IFS"
    bash-4.1$ IFS=$'\n'
    bash-4.1$ a=`find . -iname '*.csv'`
    bash-4.1$ for i in $a; do echo $i; done
    ./foo (2).csv
    bash-4.1$ IFS="$OLDIFS"

If you try to loop over the files without changing the IFS, it will break on the spaces.

    bash-4.1$ a=`find . -iname '*.csv'`
    bash-4.1$ for i in $a; do echo $i; done