Some Useful Linux or Unix Commands

Simple Commands

• ln
  • ln -s /u/to/long/d/ symbolicLinkName
• grep and pipe
  • man grep | grep reg
    
    this returns each line in the manual entry of grep that contains the string \"reg\" somewhere in this line
  • cat dict.lsp | grep PRE | cat > preps.lsp
    
    prints out the contents of the file dict.lsp, the pipe takes these arguments and gives them to grep as input, grep then filters out all the lines that contain the string PRE and gives this input to cat which opens a file and writes all these lines into the file preps.lsp
• cut
  • Example of 'cut'
    cat preps.lsp | cut -d \" -f2 | cat > onlypreps.txt
    
    Explanation: with the first part I output the file preps.lsp to standard output. this is the input to the cut command. This cuts each line into parts seperated by the delimiter of the quoting symbol (needs to be behind a backslash). One can think of the result as a little hash from which I take the second field with the -f2 command. The last pipe returns the output to the cat command which prints the output to the file onlypreps.txt
  • Using this and the second example of grep, I now have a list with all the prepositions of my dictionary, which has 120000 lines looking like this:
    (setf(gethash "avec" *dict*)'((ADV "avec" NIL  NIL  ((NIL )) 159.95)(PRE "avec" NIL  NIL  ((NIL )) 7378.48)))
    (setf(gethash "ave" *dict*)'((NOM "ave" m NIL  ((NIL )) 5.55)))
    (setf(gethash "avelines" *dict*)'((NOM "aveline" f p ((NIL )) 0.07)))
    (setf(gethash "avenant" *dict*)'((ADJ "avenant" m s ((NIL )) 1.3)(NOM "avenant" m s ((NIL )) 0.94)))
  • The first grep step would return only the line:
    (setf(gethash "avec" *dict*)'((ADV "avec" NIL  NIL  ((NIL )) 159.95)(PRE "avec" NIL  NIL  ((NIL )) 7378.48)))
    and the second then:
  • The cut command returns: 'avec'
• tar
  • This code packs all files in a directory into a g-zipped tar package, splits this tar package into multiple files of size 500MB and names them accordingly
    tar cz hugeDirectory/ | split -b500m - TaredAndZippedDirectory.tgz.split.
    
• scp
  • used for securely copying folders from one computer to another
  • scp -r folderName rsocher@sub.domain.edu:~/
    
  • this copies the entire folder
• xargs - apply a command on every single file in folder and its subfolders
  • find folder/ -maxdepth 2 -type f | grep -E "this|or|that|in|filename" | xargs -l ./getFeat other args
    
• -exec if you want to fork the piped program for each input, this is especially helpful, if some of the inputs make the program fail. Then xargs would stop completely, but -exec would continue:
  • time find positives/new* -maxdepth 2 -type f -exec ./getFeat 2 1 {} \; >> goKart.txt
    
• screen http://www.linux.com/articles/56443
  • screen # to open a new session
    Ctrl-a d # to detach, now you can log out of the machine and then log back into this session by:
    screen -r # to reattach
    screen # open another session
    screen -list # list al open sessions
    screen -r xxx # reattach screen with the number xxx
    screen -S name # opens a scrren with the name for easier reference later
  • if 'Ctrl-a k' does not quit a screen, just run kill -9 id_of_screen_process
• nohup http://en.wikipedia.org/wiki/Nohup - simple alternative to screen for some cases
• tail - print the end of the file and updates it, whenever the file changes, nice to watch log files
  • tail -f filename
    
• du -h /home/richard - this returns the size of the directory in a format readable by humans
• du -sh * - size of all files and folders in current directory
• http://www.apmaths.uwo.ca/~xli/vim/quickstart.html for the **** vim
• http://www.unixgeeks.org/security/newbie/unix/cron-1.html for a good cron job tutorial

Bash Scripts

Good Stuff to have in your .bashrc

• case $(id -u) in
      0)
          STARTCOLOUR='\[\e[31m\]';
          ;;
      *)
          STARTCOLOUR='\[\e[36m\]';
          ;;
  esac
  ENDCOLOR="\[\e[0m\]"
  PS1="$STARTCOLOUR\u@\h:\w$ $ENDCOLOR";
  
  export LS_OPTIONS='--color=auto -h'
  alias ls='ls $LS_OPTIONS'
  
  export FIGNORE=.svn
  
  alias sls='screen -ls'
  alias sn='screen -S'
  alias sr='screen -r'

Delete all files in a directory, except the first 100 files

• #!/bin/bash
  total=`ls $1 | wc -l`
  deleteTail=`expr $total - 100`
  rm -v `find $1 | tail -n $deleteTail`
• call it after copying it to a file keepOnlyFirst100.sh by: ./keepOnly100.sh directoryName

Call a script with all directories in your current directory

• find . -maxdepth 1 -type d -exec ./keepOnlyFirst100.sh '{}' \;
  

Apply a script which has multiple inputs to all txt files in a folder

• The 3 inputs here are a number and the input file and output file:
• find *.txt -maxdepth 1 -exec ./avgimg '400' '{}' '{}.jpg' \;
  

Append a line to all files in a folder

• for FILE in *; do cat $FILE | echo -e "added line for all file\n" >> $FILE; done
  

Other

• Create a password-less ssh key to login into other machines using the same filesystem (i.e. be able to do ssh otherMachine without a password)
  • in your home directory:
  • ssh-keygen -t dsa
    # press enter 3 times
    cd .ssh
    cp id_dsa.pub authorized_keys2
    

Perl

How to install new CPAN modules?

• perl -MCPAN -e shell # go to CPAN install mode
  install Bundle::CPAN # update CPAN
  reload cpan
  install Set::Scalar
  

Perl One-Liners

• http://sial.org/howto/perl/one-liner/

Sending multiple commands to screen session on different machines

• #!/usr/bin/perl -w
  
  # This script creates screen sessions, ssh's to machines and executes code on these machines.
  # parameters: -s (start) -r (run) -q (quit)
  # HowTo:
  #  1) change the executed code, property folder and prefix to your values
  #  2) select your machines
  #  3) on the machine where you want your screen sessions run to start your sessions: ./clusterSubmitJobs.pl -s
  #  4) once you're done and want to quit all your sessions: ./clusterSubmitJobs.pl -q
  # author: richard socher.org
  
  use strict;
  use Getopt::Std;
  use List::Util qw[min max];
  my %options=();
  getopts("srq",\%options);
  
  #------------------
  # files to be considered
  my $folder = '/folderWithInputFiles';
  my $prefix = 'tests_';
  my $ext = '.config';
  # code to run with files
  my $code = './runMyScript.sh -configFile ';
  
  # deprecated by mstat
  my @freemachines = ('machine1.yourPlace.edu', 'machine2.yourPlace.edu');
  #-------------------
  
  
  my $full = $folder . $prefix . '*' . $ext;
  print "Using files: $full \n";
  
  my @files = <$full*>;
  my $numMachines = @freemachines;
  my $numFiles = @files;
  my $minNum = min($numMachines,$numFiles);
  
  for (my $i = 0; $i < $minNum; $i++) {
    if ($options{s}){
      print "Creating screen session: freemachines[$i] for \t $files[$i] \n";
      system("screen -d -m -S $freemachines[$i]");
      system("screen -S $freemachines[$i] -p 0 -X stuff \"ssh $freemachines[$i]\015\"");
    }
  
    if ($options{r}){
      print "run: screen -S $freemachines[$i] -p 0 -X stuff \"$code $files[$i]\"\n";
      system("screen -S $freemachines[$i] -p 0 -X stuff \"$code $files[$i]\015\"");
    }
  
    if ($options{q}){
      print "screen -S $freemachines[$i] -p 0 -X stuff \"exit\n";
      system("screen -S $freemachines[$i] -p 0 -X stuff \"exit\015\"");
      system("screen -S $freemachines[$i] -p 0 -X stuff \"exit\015\"");
    }
  }

Extracting multiple multiline patterns between a start and an end tag using regular expressions

• Here, we want to extract everything between <parse> and </parse>.
• #!/usr/bin/perl -w
  
  local $/;
  
  open(DAT, "yourFile.xml") || die("Could not open file!");
  my $content = <DAT>;
  
  while ($content =~ m/<parse>(.*?)<\/parse>/sg){
   print "$1\n"
  };

Add Comment
Sign as Author
Enter code: