Linux notes

From raju

dummy

head tail usage

head -n -5 FILE    print everything but the last 5 lines
head -n +5 FILE    print only the first 5 lines
tail -n -5 FILE    print only the last 5 lines
tail -n +5 FILE    print from 5th line onwards

print last but one line

    cat FILE | tail -n -2 | head -n +1
    

here documents

tags | EOF, heredocs

Example 1:

To remove leading zeros from otherwise integer values using command line and standard input

    $cat << EOF | bc
    > 001
    > 002
    > 003
    > EOF
    1
    2
    3
    

Example 2:

Use 'EOF' to not expand the shell variables inside a here document.

    $ cat << EOF
    > echo $HOME
    > EOF
    echo /home/raju
    
    $ cat << 'EOF'
    > echo $HOME
    > EOF
    echo $HOME
    

Tips

convert jpg to pdf

Change the default configuration of imagemagick. Has to be done only once.

    sudo sed -i -e 's!<policy domain="coder" rights="none" pattern="PDF" />!<policy domain="coder" rights="read|write" pattern="PDF" />!g' /etc/ImageMagick-6/policy.xml
    

Then

    convert foo.jpg foo.pdf
    

With the default policy.xml, I was getting

     % convert first_day.jpg first_day.pdf
    convert-im6.q16: attempt to perform an operation not allowed by the security policy `PDF' @ error/constitute.c/IsCoderAuthorized/408.
    
     % gs --version
    9.27
    
     % convert --version
    Version: ImageMagick 6.9.10-23 Q16 x86_64 20190101 https://imagemagick.org
    Copyright: © 1999-2019 ImageMagick Studio LLC
    License: https://imagemagick.org/script/license.php
    Features: Cipher DPC Modules OpenMP 
    Delegates (built-in): bzlib djvu fftw fontconfig freetype heic jbig jng jp2 jpeg lcms lqr ltdl lzma openexr pangocairo png tiff webp wmf x xml zlib
    
     % cat /etc/debian_version 
    10.4
    

Ref:- https://askubuntu.com/questions/1127260/imagemagick-convert-not-allowed

redirect standard output and error

To redirect standard output and error into a file

    foo > log.txt 2>&1
    

useful HOWTOs and tutorials

sound related

Increase the Sound Level in a Video - http://blog.expat-it.net/index.cgi/Admin/commandLine/files/increase-sound-level-in-video.html

ffmpeg -i video-in.avi -vol 1536 -vcodec copy video-out.avi

Every 256 in vol increases sound by 100%. So in the above example, the sound will be increased by 600%.

wget tips

using wget to download a file via php script

wget http://www.vim.org/scripts/download_script.php?src_id=21680 -O NrrwRgn-0.32.vmb -q

Listing options in this order is more easy to remember (get the link, store it here and be quiet). Compare this to

wget -O csv-0.30.vmb -q http://vim.sourceforge.net/scripts/download_script.php?src_id=21684

where the thought process is not as smooth.

Ref:- https://raam.org/2007/using-wget-to-run-a-php-script/

execute external commands

To execute an external command in gdb

(gdb) help shell
Execute the rest of the line as a shell command.
With no arguments, run an inferior shell.
(gdb) shell date
Sun Jan 18 15:04:19 EST 2015

enter tab character in the shell

    <CTRL-v><Tab>
    

will insert a tab character on the command-line.

find usage articles

Articles on using find

List directories by size

To sort the directories by disk usage

    $ du -ah --max-depth=1 | sort -hr | head -n 10
    1.3G    .
    143M    ./agg_5
    143M    ./agg_4
    143M    ./agg_3
    143M    ./agg_2
    143M    ./agg_1
    122M    ./run_3
    121M    ./run_5
    121M    ./run_4
    121M    ./run_2
    

or

    rajulocal@hogwarts ~ % du -a -h --max-depth=1 | sort -h | tail -n 10
    225M    ./compile_static_gdb_debian_way
    228M    ./.local
    370M    ./songs
    501M    ./youtube_videos
    849M    ./x
    944M    ./.cache
    1.2G    ./work
    1.7G    ./.adobe
    8.0G    ./software
    15G     .
    

tips available on the net

add a string to the end of each line

We can use awk to add a given string to the end of each line in a file. For example,

% cat ~/x/junk21.txt
a
an
the
they
their
% cat ~/x/junk21.txt| awk '{print $0 " str .=$()-"}'
a str .=$()-
an str .=$()-
the str .=$()-
they str .=$()-
their str .=$()-

search the symbols

nm DebuggableExecutable -C

check if the DISPLAY variable is set correctly

Use "xset q" and check the error code. For example if it is working

% xset q
<output suppressed>
% echo $?
0

If it is not working

% xset q
xset:  unable to open display ""
% echo $?
1

What type of OS am I running?

Use OSTYPE environment variable to check the type of operating system.

% printenv | grep OS
...
OSTYPE=linux-gnu
...

or simply

% printenv OSTYPE   
linux-gnu

top usage

  • Run in batch mode
 top -c -u [username] -b -n 1

-c -u [username] is optional -b is for batch mode -n 1 is to stop after one iteration

  • track the memory usage of a process over time
 top -b -p [pid] < /dev/null >& top.out &

Look at the VIRT field.

updatedb and locate

  • To generate the database
 updatedb -U <dir1> -l 0 -o <dir1.db>
 updatedb -U <dir2> -l 0 -o <dir2.db>
  • To search the database
 locate -d <dir1.db> -d <dir2.db> -e -i <expr>

Looks like the updatedb options have changed?

 mkdir -p ~/.locate
 cd ~/.locate
 updatedb --netpaths='dir1' --output=dir1.db

tested using | updatedb 4.6.0

Open a pdf at a specific page

To open at the 10th page of a pdf. Note that this is not same as opening it at page 10 by label.

    acroread /a page=10 file.pdf
    evince -i 10 file.pdf
    okular -p 10 file.pdf
    

To open at page 10 by label.

    evince -p 10 file.pdf
    

cat show filename

Use "tail -n +1" to show the filename when displaying its contents. For example

    tail -n +1 file1.txt file2.txt file3.txt
    

Ouptut:

    ==> file1.txt <==
    <contents of file1.txt>
    
    ==> file2.txt <==
    <contents of file2.txt>
    
    ==> file3.txt <==
    <contents of file3.txt>
    

print the type of file system

    df -T /path/to/dir
    

This can tell if the directory is a local directory (ex:- ext4 filesytem) or a remote mount (ex:- nfs).

create a file of specific size with random ascii data

Method 1:

    base64 /dev/urandom | head -c 1024 > file.txt
    

Replace 1024 with the number of bytes. You can also use 1k, 1m, 1g (or 1K, 1M, 1G) etc., as arguments to create files of 1 kilo byte (i.e. 1024 bytes), mega byte and giga byte respectively.

Method 2:

    tr -dc A-Za-z0-9 < /dev/urandom | head -c 1024 > file.txt
    

create a zip file from list of files

    zip foo.zip bar.txt baz.txt
    

To unzip

    unzip foo.zip
    

Note that this will not delete the foo.zip.

shell redirect before the command

    $ >fruits echo apple
    
    $ cat fruits
    apple
    
    $ >>fruits echo grape
    
    $ cat fruits
    apple
    grape
    

Ref:-

recursively count number of lines in all files under a directory

    wc `find`
    

Ref:- https://stackoverflow.com/a/8263707/6305733

external links

append output of two commands

Sample commands:

    ( command1 ; command2 ; command3 ) | cat
    { command1 ; command2 ; command3 ; } > outfile.txt
    
    { command1 & command2; }
    { command1 & command2; } > new_file
    { command1 & command2; } > STDOUT_file 2> STDERR_file
    

Ref:- https://unix.stackexchange.com/questions/64736/combine-the-output-of-two-commands-in-bash

echo multiple lines

Can be with cat and a here document

    $ cat << EOF
    > line1
    > line2
    > EOF
    line1
    line2
    

commonly used date time stamps

time stamp for log files

    $ foo > bar_`date +'%Y%m%d_%H%M%S'`
    

stores the output of foo into a file called bar_YYYYmmDDHHMMSS

Standards

XDG

Troubleshooting

slow file listing

tl;dr
Setting LS_COLORS as

    export LS_COLORS='ex=00:su=00:sg=00:ca=00:'
    

will speed up the listing of files in directories with lots of files.