Unit 7 - Standard I/O and Pipes
From Msamir projects
Contents |
Introduction
In Linux philosophy every thing is a file. all hardware for linux nothing but files for example: Keyboard, Screen, Printer, Mouse, Hard Disk, Fingerprint-scanner, ....
One of smart guy now will ask so how come linux differentiate between all this files ?
The answer is with file descriptors, each pieces of this hardware have got unique file descriptors associated with it linux provides each process that is run on it access to three important files. They are standard input, standard output and standard error. And each of these special files (standard input, output and error) have got the file descriptors 0, 1 and 2 respectively.
debian:/dev# ls -ali /dev/std* 2658 lrwxrwxrwx 1 root root 15 2009-06-24 09:44 /dev/stderr -> /proc/self/fd/2 2656 lrwxrwxrwx 1 root root 15 2009-06-24 09:44 /dev/stdin -> /proc/self/fd/0 2657 lrwxrwxrwx 1 root root 15 2009-06-24 09:44 /dev/stdout -> /proc/self/fd/1
But what is standard input and standard output?
the purpose we invite the computer for is to give the machine some information and let the machine do the computing in this information "input" and sure we need the result "output". in typical operation your keyboard is your standard input (stdin), and the screen is the standard output (stdout). if any thing wrong happen screen also will be standard error (stderr). Take care we are in linux world where everything is configurable, we can change all this later on when we need too
Output Redirection
let say we need to have file include numbers from 1 to million, you have two choice
- Like all respectable cave man open file type them from your stander input "keyboard" into file -- meet you in next century --
- Like 21st century man do it simply by computer
let's think with this 21st century man how we can do it,there is already command in linux can print sequence of numbers this command called seq all we have to give to this command where to start and where to end, we still have small problem seq will print this numbers to it's (stout) which is screen we force seq to print them into file so.... what we need called redirecting output which can be achieved by using the ">" (greater-than symbol) this symbol can tell linux to change the stnout to other destination in our case the filename.
debian:/tmp# seq 1 5 1 2 3 4 5 debian:/tmp# seq 1 5 > filename debian:/tmp# cat filename 1 2 3 4 5 debian:/tmp#
the notable thing in ">" behavior is that if file does not exists file name will be created - or overwritten if it already exists, so take care. it is important to note that when creating files, the output redirection operators are interpreted by the shell before the commands are executed. This means that any output files created through redirection are opened first. For this reason, you cannot modify a file in place, like this:
debian:/tmp# cat filename > filename cat: filename: input file is output file debian:/tmp# grep 1 filename > filename debian:/tmp# cat filename debian:/tmp#
If file1 contains something of importance, this command would be a disaster because an empty file1 would overwrite the original. The grep command would be last to execute, resulting in a complete data loss from the original file1 file because the file that replaced it was empty.
Now let's make the task more hard for computer we need this numbers to be order randomly, we have them already in filename all we need is use command sort with -R option to make computer do this job
debian:/tmp# sort -R filename 5 3 4 1 2
pipe
sure you think now do we have to do this in two step create the numbers file then sort them, there is short way which can be done using "|" (pipe) operator which sends the standard output of one command to another command as standard input. so here is how our command will be
debian:/tmp# seq 1 5 | sort -R > filename debian:/tmp# cat filename 1 4 2 5 3
Input Redirection
if you grasp the idea behind how i/o redirection work then you can guess what this command mean and do
debian:/tmp# cat http://www.msamir.net http://www.msamir.net ^C debian:/tmp# cat < filename this is test file you think i say something wrong dsfafsadfdsa
in short cat command take any thing from stdin and give it back in stdout and in our example we make his stdin filename so it give it's content back in stdout
Error Redirection
let's digging more deeper in redirection try to guess what is going on in this example
redhat:/tmp/example# ls redhat:/tmp/example# ls msamir.net ls: cannot access msamir.net: No such file or directory redhat:/tmp/example# ls msamir.net > error.txt ls: cannot access msamir.net: No such file or directory redhat:/tmp/example# cat error.txt redhat:/tmp/example# ls msamir.net 2> error.txt redhat:/tmp/example# cat error.txt ls: cannot access msamir.net: No such file or directory redhat:/tmp/example# touch msamir.net redhat:/tmp/example# ls msamir.net 2> error.txt msamir.net redhat:/tmp/example# cat error.txt redhat:/tmp/example# ls msamir.net 1> error.txt redhat:/tmp/example# cat error.txt msamir.net redhat:/tmp/example#
I am in empty directory as you see ls command list no file in first command. when i try ls none existing file "msamir.net" it give me error messages back, till now nothing special happen. when i try to redirect this error messages to my error.txt file surprisingly find it's not working as some of you might have been expected !!!!
think in it well i never say ">" will redirect every thing i say it will redirect only stdout but linux designer elite of the the hacker know that nothing is perfect and all the time error can happen so thy make for error place to go to allow poor user/programmer to trace error as separate output, by default linux kernel understand ">" as redirection to stdout if we need to tell the kernel something else we will have to give the kernel the file descriptors before the ">" symbol and in our case we need to redirect stderr which have file descriptor 2 so this is what i do in 5th command and also this is why 6th command show that error.txt file now include the error message.
i will leave the reset as exercise for you
as we must understand by now stdout and stderr by default both go to screen let's try change both to go to other files "yes screen was a file too"
msamir@RHCE:~$ find / -name msamir.net > result 2>error ^C msamir@RHCE:~$ cat error find: `/root/Mail': Permission denied find: `/root/.config': Permission denied find: `/root/.ssh': Permission denied msamir@RHCE:~$ cat result /tmp/example/msamir.net msamir@RHCE:~$
input/output can be redirected to the same file:
msamir@RHCE:~$ find / -name msamir.net &>result ^C msamir@RHCE:~$ cat result find: `/root/.gconf': Permission denied find: `/root/Mail': Permission denied find: `/root/.config': Permission denied find: `/root/.ssh': Permission denied find: `/BACKUP/lost+found': Permission denied /tmp/example/msamir.net find: `/etc/ssl/private': Permission denied
All Redirection Cases
redirect stdout to outputfile. # command > outputfile
# command 1> outputfile
Append stdout to file. # command >> file
# command 1>> file
Append stderr to errfile. # command 2>> errfile
Append both stdout and stderr to file. # command >> file 2>&1
Send stderr to errfile. # command 2> errfile
Send both stdout and stderr to allfile. # command > allfile 2>&1
Send stdout to outputfile and stderr to errfile. # command > outputfile 2> errfile
Receive stdin from infile. # command < infile
Pipe stdout from command1 to command2. # command1 | command2
Pipe stdout and stderr from command1 to command2. # command1 2>&1 | command2
tee
In some situation you maybe want to run a program and send its output to a file while at the same time viewing the output on the screen, this is tee utility job and it's purpose
SYNOPSIS
tee [OPTION]... [FILE]...
DESCRIPTION
Copy standard input to each FILE, and also to standard output.
-a, --append
append to the given FILEs, do not overwrite
If a FILE is -, copy again to standard output.
tee examples
msamir@centos:~$ find / -name msamir.net 2>&1 | tee result find: `/root/.gconf': Permission denied . . /tmp/example/msamir.net . . ^C msamir@centos:~$ cat result find: `/root/.gconf': Permission denied . . /tmp/example/msamir.net . . msamir@centos:~$
xargs
xargs is a command on Unix and most Unix-like operating systems. It is useful when one wants to pass a large number of arguments to a command. Until Linux kernel 2.6.23, arbitrarily long lists of parameters could not be passed to a command [1], so xargs will break the list of arguments into sublists small enough to be acceptable.
xargs - build and execute command lines from standard input
user1@suse:~$ find /etc/ld.so.conf.d/ -type f | xargs -n 1 du -sh 4.0K /etc/ld.so.conf.d/libc.conf 4.0K /etc/ld.so.conf.d/i486-linux-gnu.conf
| « | RedHat Linux Essentials Standard I/O and Pipes | » |
| Using the bash Shell | Text Processing Tools |
