Some Useful Linux or Unix Commands

Simple Commands

• |, sort, uniq,
• pushd popd
  • /here/now/$: pushd /there/then
    /there/then/$: ls
    /there/then/$: popd
    /here/now/$:
    
• ln
  • ln -s /u/to/long/d/ symbolicLinkName
• grep
  • man grep
    
    this returns each line in the manual entry of grep that contains the string \"reg\" somewhere in this line
  • grep PRE file.txt > out.lsp
    
    prints out the contents of the file file.txt that contain the string PRE
  • Print all lines except those that contain the word card:
    grep -v card uFiltered.txt > uFilter2.txt
    
• find - prints all filenames with pattern inside, does not print errors such as not having permission to search directory
  • find . -name *pattern* 2>/dev/null
    
• cut
  • Example of 'cut'
    cat preps.lsp | cut -d \" -f2 > onlypreps.txt
    
    Explanation: with the first part I output the file preps.lsp to standard output. this is the input to the cut command. This cuts each line into parts seperated by the delimiter of the quoting symbol (needs to be behind a backslash). One can think of the result as a little hash from which I take the second field with the -f2 command. The last pipe returns the output to the cat command which prints the output to the file onlypreps.txt
  • Using this and the second example of grep, I now have a list with all the prepositions of my dictionary, which has 120000 lines looking like this:
    (setf(gethash "avec" *dict*)'((ADV "avec" NIL  NIL  ((NIL )) 159.95)(PRE "avec" NIL  NIL  ((NIL )) 7378.48)))
    (setf(gethash "ave" *dict*)'((NOM "ave" m NIL  ((NIL )) 5.55)))
    (setf(gethash "avelines" *dict*)'((NOM "aveline" f p ((NIL )) 0.07)))
    (setf(gethash "avenant" *dict*)'((ADJ "avenant" m s ((NIL )) 1.3)(NOM "avenant" m s ((NIL )) 0.94)))
  • The first grep step would return only the line:
    (setf(gethash "avec" *dict*)'((ADV "avec" NIL  NIL  ((NIL )) 159.95)(PRE "avec" NIL  NIL  ((NIL )) 7378.48)))
    and the second then:
  • The cut command returns: 'avec'
• tar
  • This code packs all files in a directory into a g-zipped tar package, splits this tar package into multiple files of size 500MB and names them accordingly
    tar cz hugeDirectory/ | split -b500m - TaredAndZippedDirectory.tgz.split.
    
• scp
  • used for securely copying folders from one computer to another
  • scp -r folderName login@sub.domain.edu:~/
    
  • this copies the entire folder
• xargs - apply a command on every single file in folder and its subfolders
  • find folder/ -maxdepth 2 -type f | grep -E "this|or|that|in|filename" | xargs -l ./getFeat other args
    
• -exec if you want to fork the piped program for each input, this is especially helpful, if some of the inputs make the program fail. Then xargs would stop completely, but -exec would continue:
  • time find positives/new* -maxdepth 2 -type f -exec ./getFeat 2 1 {} \; >> goKart.txt
    
• screen http://www.linux.com/articles/56443
  • screen # to open a new session
    Ctrl-a d # to detach, now you can log out of the machine and then log back into this session by:
    screen -r # to reattach
    screen # open another session
    screen -list # list al open sessions
    screen -r xxx # reattach screen with the number xxx
    screen -S name # opens a scrren with the name for easier reference later
  • If 'Ctrl-a k' does not quit a screen, just run kill -9 id_of_screen_process.
  • If you want to kill all screen sessions, see perl one-liners below.
  • If you want to rename an already running screen session:
    Ctrl+A :sessionname name
• nohup http://en.wikipedia.org/wiki/Nohup - simple alternative to screen for some cases
• tail - print the end of the file and updates it, whenever the file changes, nice to watch log files
  • tail -f filename
    
• du -h /home/richard - this returns the size of the directory in a format readable by humans
• du -sh * - size of all files and folders in current directory
• df -h . - hard drive info of current drive
• split a file every 100 lines, give the resulting files digit identifications of length 10 and prefix batchWin
  split -l 100 -d -a 10 windows.txt batchWin
  
• http://www.apmaths.uwo.ca/~xli/vim/quickstart.html for the **** vim
• http://www.unixgeeks.org/security/newbie/unix/cron-1.html for a good cron job tutorial

Perl

One-Liner: Print only non-uppercase letters

• Go through file and only print words that do not have any uppercase letters.
  perl -ne 'print unless m/[A-Z]/' allWords.txt > allWordsOnlyLowercase.txt

One-Liner: Print one word per line

• Go through file, split line at each space and print words one per line.
  perl -ne 'print join("\n", split(/ /,$_));print("\n")' someText.txt > wordsPerLine.txt

One-Liner: Kill all screen sessions (no remorse)

• Since there's no screen command that would kill all screen sessions regardless of what they're doing, here's a perl one-liner that really kills ALL screen sessions without remorse.
  screen -ls | perl -ne '/(\d+)\./;print $1' | xargs -l kill -9

One-Liner: Return all unique words and their counts

• assuming no punctuation marks:
  perl -ne 'print join("\n", split(/\s+/,$_));print("\n")' documents.txt > wordsOnePerLine.txt
  cat wordsOnePerLine.txt | sort | uniq -c  | sort -n > wordCountsSorted.txt

One-Liner: Delete all special characters

• or in other words, delete every character that is not a letter, white space or line end (replace with nothing)
  perl -pne 's/[^a-zA-Z\s]*//g' text_withSpecial.txt > text_lettersOnly.txt

One-Liner: Lower case everything

• perl -pne 'tr/[A-Z]/[a-z]/' textWithUpperCase.txt > textwithoutuppercase.txt;

One-Liner: Print only one column

• Print only the second column of the data when using tabular as a separator
  perl -ne '@F = split("\t", $_); print "$F[1]";' columnFileWithTabs.txt > justSecondColumn.txt

One-Liner: Print only text between tags

• perl -ne 'if (m/<a>(.*?)</a>/g){print "$1\n"}' textFile
• The same as a script:
• Extracting multiple multiline patterns between a start and an end tag
  • Here, we want to extract everything between <parse> and </parse>.
  • #!/usr/bin/perl -w
    local $/;
    
    open(DAT, "yourFile.xml") || die("Could not open file!");
    my $content = <DAT>;
    
    while ($content =~ m/<parse>(.*?)<\/parse>/sg){
     print "$1\n"
    };

One-Liner: Sort lines by their length

• perl -e 'print sort {length $a <=> length $b} <>' textFile

One-Liner: Print second column, unless it contains a number

• perl -lane 'print $F[1] unless $F[1] =~ m/[0-9]/' wordCounts.txt

More info

• http://affy.blogspot.com/p5be/ch17.htm
• http://sial.org/howto/perl/one-liner/
• http://www.well.ox.ac.uk/~johnb/comp/perl/intro.html

Sending multiple commands to screen session on different machines

• #!/usr/bin/perl -w
  
  # This script creates screen sessions, ssh's to machines and executes code on these machines.
  # parameters: -s (start) -r (run) -q (quit)
  # HowTo:
  #  1) change the executed code, property folder and prefix to your values
  #  2) select your machines
  #  3) on the machine where you want your screen sessions run to start your sessions: ./clusterSubmitJobs.pl -s
  #  4) once you're done and want to quit all your sessions: ./clusterSubmitJobs.pl -q
  # author: richard socher.org
  
  use strict;
  use Getopt::Std;
  use List::Util qw[min max];
  my %options=();
  getopts("srq",\%options);
  
  #------------------
  # files to be considered
  my $folder = '/folderWithInputFiles';
  my $prefix = 'tests_';
  my $ext = '.config';
  # code to run with files
  my $code = './runMyScript.sh -configFile ';
  
  # deprecated by mstat
  my @freemachines = ('machine1.yourPlace.edu', 'machine2.yourPlace.edu');
  #-------------------
  
  
  my $full = $folder . $prefix . '*' . $ext;
  print "Using files: $full \n";
  
  my @files = <$full*>;
  my $numMachines = @freemachines;
  my $numFiles = @files;
  my $minNum = min($numMachines,$numFiles);
  
  for (my $i = 0; $i < $minNum; $i++) {
    if ($options{s}){
      print "Creating screen session: freemachines[$i] for \t $files[$i] \n";
      system("screen -d -m -S $freemachines[$i]");
      system("screen -S $freemachines[$i] -p 0 -X stuff \"ssh $freemachines[$i]\015\"");
    }
  
    if ($options{r}){
      print "run: screen -S $freemachines[$i] -p 0 -X stuff \"$code $files[$i]\"\n";
      system("screen -S $freemachines[$i] -p 0 -X stuff \"$code $files[$i]\015\"");
    }
  
    if ($options{q}){
      print "screen -S $freemachines[$i] -p 0 -X stuff \"exit\n";
      system("screen -S $freemachines[$i] -p 0 -X stuff \"exit\015\"");
      system("screen -S $freemachines[$i] -p 0 -X stuff \"exit\015\"");
    }
  }

How to install new CPAN modules?

• perl -MCPAN -e shell # go to CPAN install mode
  install Bundle::CPAN # update CPAN
  reload cpan
  install Set::Scalar
  

Bash Scripts

Good Stuff to have in your .bashrc

• case $(id -u) in
      0)
          STARTCOLOUR='\[\e[31m\]';
          ;;
      *)
          STARTCOLOUR='\[\e[36m\]';
          ;;
  esac
  ENDCOLOR="\[\e[0m\]"
  PS1="$STARTCOLOUR\u@\h:\w$ $ENDCOLOR";
  
  export LS_OPTIONS='--color=auto -h'
  alias ls='ls $LS_OPTIONS'
  
  export FIGNORE=.svn
  
  alias sls='screen -ls'
  alias sn='screen -S'
  alias sr='screen -r'

Delete all files in a directory, except the first 100 files

• #!/bin/bash
  total=`ls $1 | wc -l`
  deleteTail=`expr $total - 100`
  rm -v `find $1 | tail -n $deleteTail`
• call it after copying it to a file keepOnlyFirst100.sh by: ./keepOnly100.sh directoryName

Call a script with all directories in your current directory

• find . -maxdepth 1 -type d -exec ./keepOnlyFirst100.sh '{}' \;
  

Apply a script which has multiple inputs to all txt files in a folder

• The 3 inputs here are a number and the input file and output file:
• find *.txt -maxdepth 1 -exec ./avgimg '400' '{}' '{}.jpg' \;
  

Append a line to all files in a folder

• for FILE in *; do cat $FILE | echo -e "added line for all file\n" >> $FILE; done
  

Password-less ssh key

• Create a password-less ssh key to login into other machines using the same filesystem (i.e. be able to do ssh otherMachine without a password)
  • in your home directory:
  • ssh-keygen -t dsa
    # press enter 3 times
    cd .ssh
    cp id_dsa.pub authorized_keys2
    

Quick guide for setting up an svn

• cd /.../SVNDatabaseFolder/
  svnadmin create .
  cd /.../whereYouWantTheCheckedOutSVN/
  
  svn checkout file:///.../SVNDatabaseFolder
  # or if you're on a different machine
  svn checkout svn+ssh://machineName/SVNDatabaseFolder/
  # Checked out revision 0.
  # then move stuff into that folder and to svn ci -m "initial commit"
  

Add Comment
Sign as Author
Enter code: