Some Useful Linux or Unix Commands
- Simple Commands
- Bash Scripts
- Good Stuff to have in your .bashrc
- Delete all files in a directory, except the first 100 files
- Call a script with all directories in your current directory
- Apply a script which has multiple inputs to all txt files in a folder
- Append a line to all files in a folder
- Other
- Perl
- How to install new CPAN modules?
- Perl One-Liners
- Sending multiple commands to screen session on different machines
- Extracting multiple multiline patterns between a start and an end tag using regular expressions
- Bash Scripts
- Good Stuff to have in your .bashrc
- Delete all files in a directory, except the first 100 files
- Call a script with all directories in your current directory
- Apply a script which has multiple inputs to all txt files in a folder
- Append a line to all files in a folder
- Other
- Perl
- How to install new CPAN modules?
- Perl One-Liners
- Sending multiple commands to screen session on different machines
- Extracting multiple multiline patterns between a start and an end tag using regular expressions
Simple Commands
- ln
- ln -s /u/to/long/d/ symbolicLinkName
- grep and pipe
- this returns each line in the manual entry of grep that contains the string \"reg\" somewhere in this lineman grep | grep reg
- prints out the contents of the file dict.lsp, the pipe takes these arguments and gives them to grep as input, grep then filters out all the lines that contain the string PRE and gives this input to cat which opens a file and writes all these lines into the file preps.lspcat dict.lsp | grep PRE | cat > preps.lsp
- cut
- Example of 'cut' Explanation: with the first part I output the file preps.lsp to standard output. this is the input to the cut command. This cuts each line into parts seperated by the delimiter of the quoting symbol (needs to be behind a backslash). One can think of the result as a little hash from which I take the second field with the -f2 command. The last pipe returns the output to the cat command which prints the output to the file onlypreps.txtcat preps.lsp | cut -d \" -f2 | cat > onlypreps.txt
- Using this and the second example of grep, I now have a list with all the prepositions of my dictionary, which has 120000 lines looking like this: (setf(gethash "avec" *dict*)'((ADV "avec" NIL NIL ((NIL )) 159.95)(PRE "avec" NIL NIL ((NIL )) 7378.48)))
(setf(gethash "ave" *dict*)'((NOM "ave" m NIL ((NIL )) 5.55)))
(setf(gethash "avelines" *dict*)'((NOM "aveline" f p ((NIL )) 0.07)))
(setf(gethash "avenant" *dict*)'((ADJ "avenant" m s ((NIL )) 1.3)(NOM "avenant" m s ((NIL )) 0.94))) - The first grep step would return only the line: and the second then:(setf(gethash "avec" *dict*)'((ADV "avec" NIL NIL ((NIL )) 159.95)(PRE "avec" NIL NIL ((NIL )) 7378.48)))
- The cut command returns: 'avec'
- Example of 'cut'
- tar
- This code packs all files in a directory into a g-zipped tar package, splits this tar package into multiple files of size 500MB and names them accordinglytar cz hugeDirectory/ | split -b500m - TaredAndZippedDirectory.tgz.split.
- This code packs all files in a directory into a g-zipped tar package, splits this tar package into multiple files of size 500MB and names them accordingly
- scp
- used for securely copying folders from one computer to another
- scp -r folderName rsocher@sub.domain.edu:~/
- this copies the entire folder
- xargs - apply a command on every single file in folder and its subfolders
- find folder/ -maxdepth 2 -type f | grep -E "this|or|that|in|filename" | xargs -l ./getFeat other args
- -exec if you want to fork the piped program for each input, this is especially helpful, if some of the inputs make the program fail. Then xargs would stop completely, but -exec would continue:
- time find positives/new* -maxdepth 2 -type f -exec ./getFeat 2 1 {} \; >> goKart.txt
- screen http://www.linux.com/articles/56443
- screen # to open a new session
Ctrl-a d # to detach, now you can log out of the machine and then log back into this session by:
screen -r # to reattach
screen # open another session
screen -list # list al open sessions
screen -r xxx # reattach screen with the number xxx
screen -S name # opens a scrren with the name for easier reference later - if 'Ctrl-a k' does not quit a screen, just run kill -9 id_of_screen_process
- nohup http://en.wikipedia.org/wiki/Nohup - simple alternative to screen for some cases
- tail - print the end of the file and updates it, whenever the file changes, nice to watch log files
- tail -f filename
- du -h /home/richard - this returns the size of the directory in a format readable by humans
- du -sh * - size of all files and folders in current directory
- http://www.apmaths.uwo.ca/~xli/vim/quickstart.html for the **** vim
- http://www.unixgeeks.org/security/newbie/unix/cron-1.html for a good cron job tutorial
Bash Scripts
Good Stuff to have in your .bashrc
- case $(id -u) in
0)
STARTCOLOUR='\[\e[31m\]';
;;
*)
STARTCOLOUR='\[\e[36m\]';
;;
esac
ENDCOLOR="\[\e[0m\]"
PS1="$STARTCOLOUR\u@\h:\w$ $ENDCOLOR";
export LS_OPTIONS='--color=auto -h'
alias ls='ls $LS_OPTIONS'
export FIGNORE=.svn
alias sls='screen -ls'
alias sn='screen -S'
alias sr='screen -r'
Delete all files in a directory, except the first 100 files
- #!/bin/bash
total=`ls $1 | wc -l`
deleteTail=`expr $total - 100`
rm -v `find $1 | tail -n $deleteTail` - call it after copying it to a file keepOnlyFirst100.sh by: ./keepOnly100.sh directoryName
Call a script with all directories in your current directory
- find . -maxdepth 1 -type d -exec ./keepOnlyFirst100.sh '{}' \;
Apply a script which has multiple inputs to all txt files in a folder
- The 3 inputs here are a number and the input file and output file:
- find *.txt -maxdepth 1 -exec ./avgimg '400' '{}' '{}.jpg' \;
Append a line to all files in a folder
- for FILE in *; do cat $FILE | echo -e "added line for all file\n" >> $FILE; done
Other
- Create a password-less ssh key to login into other machines using the same filesystem (i.e. be able to do ssh otherMachine without a password)
- in your home directory:
- ssh-keygen -t dsa
# press enter 3 times
cd .ssh
cp id_dsa.pub authorized_keys2
Perl
How to install new CPAN modules?
- perl -MCPAN -e shell # go to CPAN install mode
install Bundle::CPAN # update CPAN
reload cpan
install Set::Scalar
Perl One-Liners
Sending multiple commands to screen session on different machines
- #!/usr/bin/perl -w
# This script creates screen sessions, ssh's to machines and executes code on these machines.
# parameters: -s (start) -r (run) -q (quit)
# HowTo:
# 1) change the executed code, property folder and prefix to your values
# 2) select your machines
# 3) on the machine where you want your screen sessions run to start your sessions: ./clusterSubmitJobs.pl -s
# 4) once you're done and want to quit all your sessions: ./clusterSubmitJobs.pl -q
# author: richard socher.org
use strict;
use Getopt::Std;
use List::Util qw[min max];
my %options=();
getopts("srq",\%options);
#------------------
# files to be considered
my $folder = '/folderWithInputFiles';
my $prefix = 'tests_';
my $ext = '.config';
# code to run with files
my $code = './runMyScript.sh -configFile ';
# deprecated by mstat
my @freemachines = ('machine1.yourPlace.edu', 'machine2.yourPlace.edu');
#-------------------
my $full = $folder . $prefix . '*' . $ext;
print "Using files: $full \n";
my @files = <$full*>;
my $numMachines = @freemachines;
my $numFiles = @files;
my $minNum = min($numMachines,$numFiles);
for (my $i = 0; $i < $minNum; $i++) {
if ($options{s}){
print "Creating screen session: freemachines[$i] for \t $files[$i] \n";
system("screen -d -m -S $freemachines[$i]");
system("screen -S $freemachines[$i] -p 0 -X stuff \"ssh $freemachines[$i]\015\"");
}
if ($options{r}){
print "run: screen -S $freemachines[$i] -p 0 -X stuff \"$code $files[$i]\"\n";
system("screen -S $freemachines[$i] -p 0 -X stuff \"$code $files[$i]\015\"");
}
if ($options{q}){
print "screen -S $freemachines[$i] -p 0 -X stuff \"exit\n";
system("screen -S $freemachines[$i] -p 0 -X stuff \"exit\015\"");
system("screen -S $freemachines[$i] -p 0 -X stuff \"exit\015\"");
}
}
Extracting multiple multiline patterns between a start and an end tag using regular expressions
- Here, we want to extract everything between <parse> and </parse>.