Find usage

From raju

search all python files in a directory for a string

find . -iname '*.py' -exec grep foo -inH --color {} +

Find recent files

find dir -type f -mtime -1 -ls
find . -iname '*.sw*' -atime -10 -type f

Do not show the starting directory in the output

By default, find shows the starting directory name in the output. For example

$find x -iname '*elastic*'

To remove it, use -printf '%P\n'

$find x -iname '*elastic*' -printf '%P\n'

To display only the file names without printing the subdirectories, use the '%f' modifier

$find x -iname '*elastic*' -printf '%f\n'

Tested on GNU findutils 4.6.0

copy all but one file

To deploy all shell scripts and python scripts except the deploying script itself

find . \( -iname '*.sh' -o -iname '*.py' \) \
    -not -name \
    -exec cp -vt $DEPLOY_DIR {} +

You can also use ! instead of -not

find . \( -iname '*.sh' -o -iname '*.py' \) ! -name -exec cp -vt $DEPLOY_DIR {} +


Find files containing two expressions

Search the C++ source code and header files to find files that have two keywords - ifstream and string

find /path/to/dir \( -iname '*.C' -o -iname '*.H' \) -print0  | xargs -0 grep -lZ ifstream  | xargs -0 grep -l string

All intermediate grep statements should use the -Z option so that the output from grep can be passed to "xargs -0".

list files with two different extensions

To search for both .jpg and .png files

find /path/to/dir \( -iname '*.jpg' -o -iname '*.png' \)

The expression within \( ... \) should be delimited by spaces on both sides.

To do the same in two directories

find /path/to/dir1 /path/to/dir2 \( -iname '*.jpg' -o -iname '*.png' \)

list all files in a directory that do not end with a particular extension

=> find . -maxdepth 1 -not -iname "*.H"

find all git folders under a directory

find . -name .git -type d -prune


list a directory but exclude directories and vim swap files

find . -maxdepth 1 ! -type d ! -name "*.sw?"

grep files in a directory but exclude subdirectories, files matching a pattern

find . -maxdepth 1 ! -type d ! -name "*.sw?" ! -name "*_RK_????????_??????" ! -name "*_blame_????????_??????" -exec grep -inH --color key_word {} +

Did I ever call awk from a perl script?

rajulocal@hogwarts ~/work/rutils
 % find . -iname '*.pl' -exec grep awk -inH --color {} +
./perl/    my $cmd = "dpkg -l \*$keyword\* | grep ^ii | awk '{print \$2}'";

List all the R programs I accessed in the last 60 days

% find . -iname '*.R' -atime -60 -type f

Find all the log files and delete them

First make sure that you are matching on the correct set of files

find . -iname '*.log'

Then use any of the following commands

find . -iname '*.log' -exec rm {} +
find . -iname '*.log' -print0 | xargs -0 rm
find . -iname '*.log' -delete

To delete interactively

find . -iname '*.log' -exec rm -i {} +

The exact command depends on the installation since some versions of find do not have -print0, -delete etc., In any case, try to avoid

find . -iname '*.log' -exec rm {} \;

which spawns an external process for each and every matched file.

The advantage of using "-exec rm" approach is that you can pass additional options to the rm command. For example

find . -iname '*.log' -exec rm -i {} \;

will delete the files interactively.

tags | remove files

delete output directories older than a month

echo "Deleting directories older than a month in $OUTPUT_ROOT"

# Check for directory existence before trying to remove the files.
# Otherwise, in the offchance that the variable is empty, find will search the
# current directory and will try to remove files there.
if [ -d "$OUTPUT_ROOT" ]; then
    find "$OUTPUT_ROOT" -maxdepth 1 -type d -ctime +30 -exec rm -rfv {} +

Give write permission to the user for all top level directories under a directory

find . -maxdepth 1 -type d -exec chmod u+w {} +

Aggregate data across all files modified in the last 10 minutes

 find . -mmin -10 -type f -print0 | xargs -0 tail -n +2 -q | sort | uniq

Find old files and move them to a directory

First make sure that you are matching on the correct set of files

find path/to/source/dir \( -iname 'pattern1' -o -iname 'pattern2' \) -mtime +6

Then append it with the mv command as below

find path/to/source/dir \( -iname 'pattern1' -o -iname 'pattern2' \) -mtime +6 -exec mv -ivt path/to/dest/dir {} +

Find files modified after a certain date

The code snippet below will find files modified on or after a certain date and copies them from an nfs directory to hdfs.

# The -mtime in find is too coarse. If you have files modified 6 hours ago and
# 8 hours ago, they can't be distinguished with "find -mtime". One way to work
# around this is to create a temp file and look for files newer than that.

touch --date $doi $tmp_file

find /path/to/nfs/dir -iname '*.foo' \
-newer $tmp_file -exec hadoop fs -put -f '{}' \
/path/to/hdfs/dir \;

rm -f $tmp_file

Find files newer than files in another directory

This will find all files in raw_dir that are newer than the files in processed_dir.

find raw_dir -newer `ls processed_dir/*.txt -rt | tail -n1` -iname '*.txt'

gunzip files on nfs and copy to hadoop

files_to_copy=`find $dir -maxdepth 1 -iname '*.avro.gz' -newer $tmp_file -print
for i in $files_to_copy
    unzipped_file=${tmp_nfs_dir}/`basename $i | sed -e 's/\.gz$//'`
    gunzip $i -c > $unzipped_file
    hadoop fs -put -f $unzipped_file $hdfs_dir
    echo "$unzipped_file -> $hdfs_dir"
    rm -f $unzipped_file

tags | process the output of find

iterate over files from find where the files have spaces

Say we have some files with spaces in them

bash-4.1$ ls -1
foo (2).csv

To iterate over them, change the IFS

bash-4.1$ OLDIFS="$IFS"
bash-4.1$ IFS=$'\n'
bash-4.1$ a=`find . -iname '*.csv'`
bash-4.1$ for i in $a; do echo $i; done
./foo (2).csv
bash-4.1$ IFS="$OLDIFS"

If you try to loop over the files without changing the IFS, it will break on the spaces.

bash-4.1$ a=`find . -iname '*.csv'`
bash-4.1$ for i in $a; do echo $i; done


useful links