Some Useful Linux or Unix Commands

Simple Commands

• For more details on any of the commands, type
  • man commandName
    # for example:
    man chmod
• Important basic commands (try man with them) cd, ls, cat, |, sort, uniq, nano, chmod, top, paste
• Reverse search through your bash history with ctrl+r, then type any substring of previous commands, then cycle through all lines in your history which have that substring by ctrl+r
• chmod, make this and all subdirectories world readable, writable and executable (only do that in a safe environment)
  chmod -R 777 folderName/
  # make a folder traversable but not readable:
  chmod 711 scr/
  # make a folder only readable/executable to yourself:
  chmod 700 folder
• Finding your process ids and all your own jobs with top
  • top -u username
• pushd popd
  • /here/now/$: pushd /there/then
    /there/then/$: ls
    /there/then/$: popd
    /here/now/$:
• ln
  • ln -s /u/to/long/d/ symbolicLinkName
• grep
  • man grep
    
    this returns each line in the manual entry of grep that contains the string \"reg\" somewhere in this line
  • grep PRE file.txt > out.lsp
    
    prints out the contents of the file file.txt that contain the string PRE
  • Print all lines except those that contain the word card:
    grep -v card uFiltered.txt > uFilter2.txt
    
• find - prints all filenames with pattern inside, does not print errors such as not having permission to search directory
  • find . -name *pattern* 2>/dev/null
    
• cut
  • Example of 'cut'
    cat preps.lsp | cut -d \" -f2 > onlypreps.txt
    
    Explanation: with the first part I output the file preps.lsp to standard output. this is the input to the cut command. This cuts each line into parts seperated by the delimiter of the quoting symbol (needs to be behind a backslash). One can think of the result as a little hash from which I take the second field with the -f2 command. The last pipe returns the output to the cat command which prints the output to the file onlypreps.txt
  • Using this and the second example of grep, I now have a list with all the prepositions of my dictionary, which has 120000 lines looking like this:
    (setf(gethash "avec" *dict*)'((ADV "avec" NIL  NIL  ((NIL )) 159.95)(PRE "avec" NIL  NIL  ((NIL )) 7378.48)))
    (setf(gethash "ave" *dict*)'((NOM "ave" m NIL  ((NIL )) 5.55)))
    (setf(gethash "avelines" *dict*)'((NOM "aveline" f p ((NIL )) 0.07)))
    (setf(gethash "avenant" *dict*)'((ADJ "avenant" m s ((NIL )) 1.3)(NOM "avenant" m s ((NIL )) 0.94)))
  • The first grep step would return only the line:
    (setf(gethash "avec" *dict*)'((ADV "avec" NIL  NIL  ((NIL )) 159.95)(PRE "avec" NIL  NIL  ((NIL )) 7378.48)))
    and the second then:
  • The cut command returns: 'avec'
• scp
  • used for securely copying folders from one computer to another
  • scp -r folderName login@sub.domain.edu:~/
    
  • this copies the entire folder
• xargs - apply a command on every single file in folder and its subfolders
  • find folder/ -maxdepth 2 -type f | grep -E "this|or|that|in|filename" | xargs -l ./getFeat other args
    
• -exec if you want to fork the piped program for each input, this is especially helpful, if some of the inputs make the program fail. Then xargs would stop completely, but -exec would continue:
  • time find positives/new* -maxdepth 2 -type f -exec ./getFeat 2 1 {} \; >> goKart.txt
    
• screen http://www.linux.com/articles/56443
  • Allows you to run commands in the background, which continue even after you log-out of that machine.
  • screen # to open a new session
    Ctrl-a d # to detach, now you can log out of the machine and then log back into this session by:
    screen -r # to reattach
    screen # open another session
    screen -list # list al open sessions
    screen -r xxx # reattach screen with the number xxx
    screen -S name # opens a scrren with the name for easier reference later
    screen -r name # re-attach to that session
  • If 'Ctrl-a k' does not quit a screen, just run kill -9 id_of_screen_process.
  • If you want to kill all screen sessions, see perl one-liners below.
  • If you want to rename an already running screen session:
    Ctrl+A :sessionname name
• nohup http://en.wikipedia.org/wiki/Nohup - simple alternative to screen for some cases
• tail - print the end of the file and updates it, whenever the file changes, nice to watch log files
  • tail -f filename
    
• du -h /home/richard - this returns the size of the directory in a format readable by humans
• du -sh * - size of all files and folders in current directory
• df -h . - hard drive info of current drive
• split a file every 100 lines, give the resulting files digit identifications of length 10 and prefix batchWin
  split -l 100 -d -a 10 windows.txt batchWin
  
• tar and untar
  • This code packs all files in a directory into a g-zipped tar package, splits this tar package into multiple files of size 500MB and names them accordingly
    tar cz hugeDirectory/ | split -b500m - TaredAndZippedDirectory.tgz.split.
    
  • untar a .tar (leave out option z) or .tar.gz file into a directory
    tar -C newFolder -zxvf file.tar
    
• unzip a file into a directory
  • gunzip file.gz
    
  • if that doesn't work, try:
    gzip -d file.gz
    
• http://www.apmaths.uwo.ca/~xli/vim/quickstart.html for the **** vim
• http://www.unixgeeks.org/security/newbie/unix/cron-1.html for a good cron job tutorial

Bash Scripts

For loop over a bunch of directories and run some command in each:

• submitFolder=../gw/submit/
  resultsFile=allSentencesTeams.txt
  rm $resultsFile
  for dir in $submitFolder*/
  do
  dir=${dir%*/}
  java -jar pcfg.jar generate -n 20 $submitFolder${dir##*/}/S1.gr $submitFolder${dir##*/}/Vocab.gr >> $resultsFile
  done
  

• Rename all files ending in *final to *final.mat
  • for i in *final; do mv "$i" "$i.mat"; done

Find a string in the last n (9) updated files in the current directory (and print out only that string)

• ls -rt | tail -n9 | xargs -l grep results $1 | grep  -Eo " 91..."

Good Stuff to have in your .bashrc

• case $(id -u) in
      0)
          STARTCOLOUR='\[\e[31m\]';
          ;;
      *)
          STARTCOLOUR='\[\e[36m\]';
          ;;
  esac
  ENDCOLOR="\[\e[0m\]"
  PS1="$STARTCOLOUR\u@\h:\w$ $ENDCOLOR";
  
  export LS_OPTIONS='--color=auto -h'
  alias ls='ls $LS_OPTIONS'
  
  export FIGNORE=.svn
  
  alias sls='screen -ls'
  alias sn='screen -S'
  alias sr='screen -r'
  
  # make sure bash updates it's (echo $COLUMNS) when the size changes
  # so there are no ugly overlay effects of long lines
  shopt -s checkwinsize
   

Delete all files in a directory, except the first 100 files

• #!/bin/bash
  total=`ls $1 | wc -l`
  deleteTail=`expr $total - 100`
  rm -v `find $1 | tail -n $deleteTail`
• call it after copying it to a file keepOnlyFirst100.sh by: ./keepOnly100.sh directoryName

Call a script with all directories in your current directory

• find . -maxdepth 1 -type d -exec ./keepOnlyFirst100.sh '{}' \;

Apply a script which has multiple inputs to all txt files in a folder

• The 3 inputs here are a number and the input file and output file:
• find *.txt -maxdepth 1 -exec ./avgimg '400'  '{}'  '{}.jpg' \;

Delete all directories that are older than a certain time

• Here, we delete all folders that are older than 19 minutes and include the string _qsub
• find . -type d -mmin +19 | grep _qsub | xargs rm -rf

Append a line to all files in a folder

• for FILE in *; do cat $FILE | echo -e "added line for all file\n" >> $FILE; done

Bash One-Liner For Loop to delete qsub'ed jobs:

• for i in {13657..13683};do qdel $i;done
• Fancier: get job list, select only those with my username, select first row (job id), then run qdel with each jobid
• for jobid in $(qstat | grep rsocher | cut -d . -f 1); do qdel $jobid; don

Password-less ssh key

• Create a password-less ssh key to login into other machines using the same filesystem (i.e. be able to do ssh otherMachine without a password)
  • in your home directory:
  • ssh-keygen -t dsa
    # press enter 3 times
    cd .ssh
    cp id_dsa.pub authorized_keys2
    

Quick guide for setting up an svn

• cd /.../SVNDatabaseFolder/
  svnadmin create .
  cd /.../whereYouWantTheCheckedOutSVN/
  
  svn checkout file:///.../SVNDatabaseFolder
  # or if you're on a different machine
  svn checkout svn+ssh://machineName/SVNDatabaseFolder/
  # Checked out revision 0.
  # then move stuff into that folder and add them to the svn, make sure to only add text/code files, NO binary files
  svn add *.m
  # Then commit you text files to the svn:
  svn ci -m "initial commit"
  

Perl

• See Perl Scripts And One Liners

Comments

Add Comment
Sign as Author
Enter code: