Introduction to Data Analysis in Geophysics Using UNIX
Learn how to use UNIX commands such as pipes and xargs in geophysics data analysis. Discover how to manipulate file lists, search for specific strings in files, and run programs on file names efficiently using xargs in a UNIX environment.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
C E R I C E R I- -7 1 04/ C IV L 7 1 04/ C IV L - -8 1 2 6 D ata A nalysis in G eophysics 8 1 2 6 D ata A nalysis in G eophysics C ontinue start U N IX . L ab 1 5 , 1 07 / 1 0/ 1 9
xargs command S o far we have seen how to use pipes to take the output of one command and use it as the input for another command. W e can pipe a list of file names into grep using standard IN to look for files with certain strings in their names for example. lsd -l | grep -v HW | grep '\.m.*$'
xargs command B ut what if we wanted to run some other program on the resulting list of file names, i.e. use the list of files as an argument to the next program rather than input to the next program? G repping for something in the file list, or copying the files, for example. T here are (of course) several ways to do this with U N IX . O ne is to use the U N IX command xargs.
xargs command S ay I wanted to find all my shell scripts with the character string bin in them. F irst a comment on naming things. It is helpful if you name your files so the name tells you something like what kind of file it is. T he extension (the part after the last dot ) is handy for this. S o I name sh/ bash shell scripts something.sh A nd csh/ tcsh shell scripts something.csh or something.tcsh E tc.
xargs command S ay I want to find all my shell scripts that have the string bin in them? I need to loop over all the file names and run grep on each file. T his is one of the things xargs does, it takes standard IN and runs its argument list for each token (things separated by white space).
xargs command ls *sh | xargs grep bin xargs takes the list of files from the ls command and runs grep bin file from the list until it runs out of input. $ ls *sh | xargs grep bin NevadaNEV2mat.sh:#!/bin/sh blewitt2matlab.sh:#!/bin/sh find_all_mfiles.sh:#!/bin/sh get_altamini_noam_itrf_gps.sh:#!/bin/sh
B y default xargs puts the list of stuff coming in at the end of the argument list (in our example after the grep bin ) S ay I wanted to put in a list of stuff to search for in a specific file grep searchitem specificfile T he place I want to put the list is no longer at the end. D o this with an option to xargs: -J where you define a placeholder that says where it goes xargs -J % grep % specificfile
I said there was also another way to do this. A ll the shells are programming languages and you can write programs called scripts in them. T hey have loops, ifs, etc. T he syntax is of course just different enough in each to be a pain.
S o in csh/ tcsh I can do the same thing by writing this % foreach file ( `ls *sh` ) foreach? echo \[$file\] `grep bin $file` foreach? end [altamini4matlab.sh] #!/bin/sh [atomlist.sh] #!/bin/sh [find_newcmt.sh] #!/bin/sh [gmt5ex5.sh] #!/bin/bash [readline_temp.sh] #!/bin/sh /users/robertsmalley/unixside/bin/crx2rnx whtm0010.16d [remove_m64_flag.sh] #!/bin/sh [stderrredirect.sh] #!/bin/sh [test.sh] #!/bin/sh
% foreach file ( `ls *sh` ) foreach? echo \[$file\] `grep bin $file` foreach? end T here are several new things here foreach is the for loop in csh/ tcsh, everything until the end keyword is in the loop. T he foreach? prompts on the interior lines indicate you are entering commands into the body of the loop.
S o in csh/ tcsh I can do the same thing by writing this % foreach file ( `ls *sh` ) foreach? echo \[$file\] `grep bin $file` foreach? end N ext we have the grave or back quotes ` T hese perform command substitution they take the output of the command and just dump it literally into the command line.
S o % foreach file ( `ls *sh` ) Is the same as if I had typed the output of the ls command between the parenthesis. % ls *.sh altamini4matlab.sh find_newcmt.sh readline_temp.sh stderrredirect.sh atomlist.sh gmt5ex5.sh remove_m64_flag.sh test.sh % foreach file ( altamini4matlab.sh atomlist.sh find_newcmt.sh gmt5ex5.sh readline_temp.sh remove_m64_flag.sh stderrredirect.sh test.sh) foreach? echo \[$file\] `grep bin $file` foreach? end [altamini4matlab.sh] #!/bin/sh [atomlist.sh] #!/bin/sh [find_newcmt.sh] #!/bin/sh [gmt5ex5.sh] #!/bin/bash [readline_temp.sh] #!/bin/sh /users/robertsmalley/unixside/bin/crx2rnx whtm0010.16d [remove_m64_flag.sh] #!/bin/sh [stderrredirect.sh] #!/bin/sh [test.sh] #!/bin/sh
B y the way- what is the term file here? % foreach file (`ls *sh` ) foreach? echo \[$file\] `grep bin $file` foreach? end H ow it is referenced tells you. It is an environment variable. T he foreach (or for) defines it for you. Y ou access it with $file. N ote that when working in U N IX if your file names have spaces it is a big problem for this script as each space is used to start a new file name. S o replace spaces with - or _ , etc.
In sh/ bash it looks like this $ for file in `ls *sh` > do > echo [$file] `grep bin $file` > done [altamini4matlab.sh] #!/bin/sh [atomlist.sh] #!/bin/sh [find_newcmt.sh] #!/bin/sh [gmt5ex5.sh] #!/bin/bash [readline_temp.sh] #!/bin/sh /users/robertsmalley/unixside/bin/crx2rnx whtm0010.16d [remove_m64_flag.sh] #!/bin/sh [stderrredirect.sh] #!/bin/sh [test.sh] #!/bin/sh W here the > prompt says you are entering commands into the for loop.
Y ou could type the commands into a file, naming it myshellscript.tcsh or myshellscirpt.sh as appropriate, make it executable, and then just enter it on the command line. cat myshellscript.* #!/bin/sh for file in `ls *sh` do echo [$file] `grep bin $file` done #!/bin/tcsh foreach file ( `ls *sh` ) echo $file `grep bin $file` end R emember to make them executable (chmod)
T hen run them bash$ myshellscript.tcsh altamini4matlab.sh #!/bin/sh test.sh #!/bin/sh bash$ myshellscript.sh [altamini4matlab.sh] #!/bin/sh [test.sh] #!/bin/sh O ne thing to notice my prompt indicates the shell I m using on the command line bash. T he first script is in tcsh that s why I need the #!/bin/tcsh on the first line of the script that sets the shell for the script (and the script only I m back in bash when it returns to the command line.)
H ow to decide between xargs and loops? If you need to do more than one thing with the list of files use the loop and a multiline script, if you only have to do one thing use xargs (you can use either).
I said there was also another way to do this. A ll the shells are programming languages and you can write programs called scripts in them. T hey have loops, ifs, etc. T he syntax between shells is of course just different enough in each to be a pain.
S o in csh/ tcsh I can do the same thing by writing this % foreach file ( `ls *sh` ) foreach? echo \[$file\] `grep bin $file` foreach? end [altamini4matlab.sh] #!/bin/sh [atomlist.sh] #!/bin/sh [find_newcmt.sh] #!/bin/sh [gmt5ex5.sh] #!/bin/bash [readline_temp.sh] #!/bin/sh /users/robertsmalley/unixside/bin/crx2rnx whtm0010.16d [remove_m64_flag.sh] #!/bin/sh [stderrredirect.sh] #!/bin/sh [test.sh] #!/bin/sh
% foreach file ( `ls *sh` ) foreach? echo \[$file\] `grep bin $file` foreach? end T here are several new things here foreach is the for loop in csh/ tcsh, everything until the end keyword is in the loop. T he foreach? prompts on the interior lines of the loop to indicate you are entering commands into the body of the loop.
% foreach file ( `ls *sh` ) foreach? echo \[$file\] `grep bin $file` foreach? end N ext we have the grave or back quotes `stuff`. T hese perform command substitution T hey take the output of the command and just dump it literally into the command line.
S o % foreach file ( `ls *sh` ) Is the same as if I had typed the output of the ls command between the parenthesis. % ls *.sh altamini4matlab.sh find_newcmt.sh readline_temp.sh stderrredirect.sh atomlist.sh gmt5ex5.sh remove_m64_flag.sh test.sh % foreach file (altamini4matlab.sh atomlist.sh find_newcmt.sh gmt5ex5.sh readline_temp.sh remove_m64_flag.sh stderrredirect.sh test.sh) foreach? echo \[$file\] `grep bin $file` foreach? end [altamini4matlab.sh] #!/bin/sh [atomlist.sh] #!/bin/sh [find_newcmt.sh] #!/bin/sh [gmt5ex5.sh] #!/bin/bash [readline_temp.sh] #!/bin/sh /users/robertsmalley/unixside/bin/crx2rnx whtm0010.16d [remove_m64_flag.sh] #!/bin/sh [stderrredirect.sh] #!/bin/sh [test.sh] #!/bin/sh
B y the way- what is the term file here? % foreach file (`ls *sh` ) foreach? echo \[$file\] `grep bin $file` foreach? end H ow it is referenced tells you. It is an environment variable. T he foreach (or for in sh/ bash) defines it for you. Y ou access it with $file. N ote that when working in U N IX if your file names have spaces it is a big problem for this script as each space is used to start a new file name. S o when naming things use - or _ , etc.
In sh/ bash it looks like this $ for file in `ls *sh` > do > echo [$file] `grep bin $file` > done [altamini4matlab.sh] #!/bin/sh [atomlist.sh] #!/bin/sh [find_newcmt.sh] #!/bin/sh [gmt5ex5.sh] #!/bin/bash [readline_temp.sh] #!/bin/sh /users/robertsmalley/unixside/bin/crx2rnx whtm0010.16d [remove_m64_flag.sh] #!/bin/sh [stderrredirect.sh] #!/bin/sh [test.sh] #!/bin/sh W here the > prompt says you are entering commands into the for loop.
Y ou could type the commands into a file, naming it myshellscript.tcsh or myshellscirpt.sh as appropriate, make it executable, and then just enter it on the command line. cat myshellscript.* #!/bin/sh for file in `ls *sh` do echo [$file] `grep bin $file` done #!/bin/tcsh foreach file ( `ls *sh` ) echo $file `grep bin $file` end R emember to make them executable (chmod)
T hen run them bash$ myshellscript.tcsh altamini4matlab.sh #!/bin/sh test.sh #!/bin/sh bash$ myshellscript.sh [altamini4matlab.sh] #!/bin/sh [test.sh] #!/bin/sh O ne thing to notice my prompt indicates the shell I m using on the command line bash. T he first script is in tcsh that s why I need the #!/bin/tcsh on the first line of the script that sets the shell for the script (and the script only I m back in bash when it returns to the command line.)
H ow to decide between xargs and loops? If you need to do more than one thing with the list of files use the loop and a multiline script, if you only have to do one thing use xargs (you can use either).
Q U O T E S W e have been using quotes here and there without saying much about what they do exactly. W e have used them to make filenames with spaces into one token instead of breaking it up on the spaces into multiple tokens for example. (actually what we did was told the shell to not process the stuff in the quotes treat it literally)
Q U O T E S F irst quote is the \ T his is the strongest quote. It quotes or escapes the following character. T ranslation It removes any special metacharacter meaning from the following character or prevents the shell from interpreting the following character
W hat will this do as a prompt? echo Are you sure you want to remove these files? O ne of two things if there are no files whose names are files plus 0 or 1 additional character is will print out Are you sure you want to remove these E lse it will print out. Are you sure you want to remove these and a list of all the files meeting the match files? A s the shell interprets the ? as a wildcard.
Q U O T E S T he next quote is the strong quote: T his quotes all the characters between the first and last quote. S o if you have the following you can quote it 2 ways A B A\ \ \ B O r A B
Q U O T E S T he last quote is the weak quote T his quotes all the characters between the first and last quote except a select set of metacharacters that the shell can still interpret. % echo "Is your home directory $HOME? Is your home directory /home/kreskin/u0/barnett? % echo "Your current directory is `pwd` Your current directory is /home/kreskin/u0/Barnett doesn't expand meta-characters like "*" or "?," but does expand environment variables and does command substitution.
Q U O T E S Q uotes within Q uotes Q uotes within Q uotes Y ou can put either type quote inside the other. T o quote single quotes, use double quotes around it. T o quote double quotes, use single quotes. %echo "Don't do that Don't do that %echo Quote of the day: "TGIF Quote of the day: "TGIF"
W e are going to use these ideas on steroids when we cover the graphics program G eneric M apping T ools or G M T (next!)
