Basic UNIX / Linux
If you would like to look at a book I suggest:
www.tldp.org/LDP/Bash-Beginners-Guide/Bash-Beginners-Guide.pdf
HOME
By default your HOME directory is your working directory when you first
log in. The pathname of this directory is stored in the HOME environment (see further down)
variable:
$ echo $HOME
In general there is not much space in your HOME but therefore, at UQAM,
a backup of everything you keep under your HOME is done every night and
kept for a week. The backup of Friday nights is even kept for a month.
You if you ever remove something by accident or if the machine/disk
crashes, there will still be a copy of everything you had in your HOME.
Therefore it is strongly suggested that you keep everything
which is small and importent in your HOME! Like programs,
scripts, configuration files.
If ever you need to get something back from the backup, talk to Nadjet
Labassi.
If you need to store larger amounts of data, talk to Katja Winger.
Shell
A UNIX system command processor. There are two major shell (command interpreter) families:
Bourne, Korn, Bash Shell
C Shell
The syntax of UNIX commands can vary from one shell to another.
At UQAM we use the Bourne Again Shell (bash)
for the environment but often Korn Shell (ksh) for scripting. Therefore
all commands below are bash commands.
Anatomy of a Unix Command
command-name -option(s) filename(s) or
arguments
Example: wc -l sample
The first word of the command line is usually the command name. This
is followed by the options/keys, if any, then the filenames, directory
name/names, or other arguments, if any, and then a RETURN. Options are
usually preceded by a dash and you may use more than one option per
command.
You can also most of the time combine options:
$ ls -l -t -r
is the same as
$ ls -lrt
In case you have a file name or an argument starting with a '-' you can use '--'. The command will NOT interpret anything following a '--' as an option/key.
For example if you are using 'grep' to look for '-a' in a file you could type:
$ grep -iw -- -a filename
For more detailed explainations about Program Argument Syntax Conventions, please click on the here.
The examples on this reference card use bold case for
command names and options and italics for arguments and
filenames.
whatis command -
display a one-line summary about command
man command
- display on-line manual pages about command
man can be use on almost all
of the UNIX commands listed below.
Important Note about UNIX/Linux Commands
UNIX/Linux is case sensitive.
Type
commands
exactly
as
shown;
most
UNIX
commands
are
lower
case.
File
and
directory
names
can be lower, upper, or mixed case but must be typed
exactly as listed.
User Information and Helpful Commands
| quota -v |
display your disk quota and usage.
There are quotas on several of our file systems. You might want to
check them once in a while.
|
| du -s |
display your total disk usage |
du -sc *
|
display the size of all files
and folders in current directory.
Very useful if you need to do a clean up because your quota is exceeded!
|
df -h
|
check if the file systems are
full - Do NOT do this on guillimin!
|
| echo $PATH |
inspect your search path |
List Files and Directories
pwd :
display the name of present working directory
true_path
directory-name : display the
"true" name of directory-name,
not the links. This command is a local add-on.
ls directory-name : list contents of
directory
options:
|
-a |
list all files including invisible files (starting with '.') |
|
-l |
long list - shows ownership, permission, and links |
|
-t |
list files chronologically |
|
-r |
reverse order of listing |
|
-d
|
do not display directory
contents but only the directory itself
|
|
-1
|
display only one file/directory
per line
|
Example:
$ ls -ld
/home/winger
drwxr-xr-x 20 winger
users 4096 May 24 08:56 /home/winger
Column 1 : a set of ten permission flags (see below)
Column 2 : link count (don't worry about this)
Column 3 : owner of the file
Column 4 : associated group for the file
Column 5 : size in bytes
Column
6-8: date of last modification (format varies, but always 3 fields)
Column 9 : name of file/directory/link (possibly with
path, depending on how ls was called)
The permission flags are read as follows (left to right)
| position |
Meaning |
| 1 |
directory flag, 'd' if a directory, '-' if a normal file,
something else occasionally may appear here for special devices. |
| 2,3,4 |
read, write, execute permission for User (Owner) of file |
| 5,6,7 |
read, write, execute permission for Group |
| 8,9,10 |
read, write, execute permission for Other |
| value |
Meaning |
| - |
in any position means that flag is not set |
| r |
file is readable by owner, group or other |
| w |
file is writeable. On a directory, write access means you can
add or delete files |
| x |
file is executable (only for programs and shell scripts - not
useful for data files).
Execute permission on a directory means you can list the files in that
directory |
| s |
in the place where 'x' would normally go is called the
set-UID or set-groupID flag. |
See 'chmod' below about how to change the permissions.
Change Working Directory
cd : to
change to your home directory
cd directory-name : to change to
another directory
cd - : to change back to the
previous directory
Directory Abbreviations
~ : home
directory (tilde)
~username :
another user's home directory
. :
current or working directory
.. :
parent of working directory
Make (or Create) Directory
mkdir directory-name : create a directory
called directory-name
options:
-p parents:
no error if existing, make parent directories as needed
Move (or Rename) Files and Directories
mv
present-filename new-filename
: to rename a file
mv
source-filename destination-directory
: to move a file into another
directory
options: -i interactive mode.
Must confirm file overwrites.
Copy Files
cp
source-filename destination-filename
: to copy a file into another
filename
cp
source-filename(s)
destination-directory
: to copy one or more files into another
directory
options:
-i interactive mode. Must
confirm
overwrites.
-r recursive. Copies the directory and all its files and
subdirectories and their files etc.
Link Files
ln -s target link_name :
symbolically links link_name to target (can be file or directory)
Remove (or Delete) File
rm filename :
remove a file
rm -r directory-name
: remove a directory
rmdir directory-name
: to remove an empty
directory
options: -i interactive mode. Prompt for confirmation.
Look at a File
| cat
filename |
display the whole file
contents one screen |
| more
filename |
display the file
contents one screen at a time |
| less filename |
program for browsing or
paging through files or other
output.
Can
use
arrow
keys for scrolling forward or backward. |
| head
filename |
display first 10 lines
of a file. Option -n : display
first n lines |
| tail
filename |
display last 10 lines of
a file. Option -n : display
last n lines |
Wild cards
A number of characters are interpreted by the Unix shell before
any other action takes place. These characters are known as wildcard
characters. Usually these characters are used in place of filenames
or directory names.
| *
|
An
asterisk matches any number of characters in a filename, including none. |
| ?
|
The
question mark matches any single character. |
| [ ]
|
Brackets enclose a set of characters, any one of which may
match a single character at that position. |
| -
|
A
hyphen used within [ ] denotes a range of characters. |
| ~ |
A
tilde at the beginning of a word expands to the name of your home
directory. If you append another user's login name to the character, it
refers to that user's home directory. |
Here are some examples:
- cat c* : displays any
file whose name begins with c
including the file c, if it exists.
- ls *.c : lists all
files that have a .c
extension.
- cp ../rmt? . : copies
every file in the parent
directory that is four characters long and begins with rmt to
the working directory. (The names will remain the same.)
- ls rmt[34567] : lists
every file that begins with rmt
and has a 3, 4, 5, 6, or 7
at the end.
- ls rmt[3-7] : does
exactly the same thing as the
previous example.
- ls ~ : lists your
home directory.
- ls ~hessen : lists
the home directory of the guy with
the user id hessen.
Command I/O
>
:
command output
redirection (create new)
>>
: command output redirection
(append)
<
:
command input
redirection (from file)
<<
: command input (from script or
standard input)
Example:
$ cat > text_file << EOF
> Hello
> How are you?
> EOF
$ cat text_file
Hello
How are you?
null device
/dev/null or the null device is a special file that discards all data written to it but reports that the write operation succeeded.
Change File Access Permissions
chmod [ who option permission ] filename
who can be any combination of:
u (user)
g (group)
o (other)
a (all) (i.e. ugo)
option adds or takes away
permission, and can be:
+ (add permission),
- (remove permission), or
= (set to exactly this permission).
permission can be any
combination of
r (read)
w (write)
x (execute)
Example: chmod a+x filename - makes filename executable by everyone.
Display a message
echo message : diaplays/prints 'message'.
Examples:
$ echo Hello
Hello
$ echo
"Hello there"
Hello there
Variables
You can create variables and assign values to them.
There are two kinds of variables: local variables and environment variables.
The difference between the two is that environment variables are passed down to child processes (sub shell, scripts, programs)
There are already several environment variables set which you can see
with:
env
A variable name always needs to start with a letter.
To get the content of a variable the variable needs to be preceded with
a dollar sign ($)
Examples:
$ person=Alex Never put a
space before or after the equal sign!
$ echo person
person
$ echo $person
Alex
$ echo $personis
$ echo ${person}is
Alexis
$ echo "$person"
Alex => double quotes evaluate
$ echo '$person'
$person
=> single quotes: No evaluation done !!!
Export
If you just set 'variable=word' the variable will only be
known in the current shell/script in which it is set.
If you "export" the variable becomes an environment variable and will also be know in all scripts which get
executed in this shell/script.
For example ifyou export a variable in your file '~/.profile_usr' resp.
'~/.profile.d/.interactive_profile' it will be available
in all sub shells.
export variable=word
The value of the variable is
set to word.
Example:
$ export
CMCLNG=english
or
$ CMCLNG=english
$ export CMCLNG
Escape character
\ : back slash, escape character
Unfortunately '\' has different meanings depending on where/how it gets used.
Examples:
$ person=Alex
$ echo $person
Alex
$ echo \$person
$person
=> No evaluation done !!!
$ echo "\$person"
$person
=> No evaluation done !!!
$ echo '\$person'
Alex
=> Evaluation done !!!
When used as the very last character in a line it will surpress the "new line".
$ cat > text_file << EOF
> Hello
> How are you?
> EOF
$ cat text_file
Hello
How are you?
$ cat > text_file << EOF
> Hello, \
> how are you?
> EOF
$ cat text_file
Hello, how are you?
Alias
alias alias-string='command-string'
Alias abbreviates a command string with an alias string. For
multi-command strings, enclose commands in quotes.
Examples:
$ alias 3331='.
r.sm.dot gemclim 3.3.3.1'
$ alias vo='voir
-iment'
$ alias rd=r.diag
$ alias rg='r.diag
ggstat'
If you put aliases in your file '~/.profile_usr' resp.
'~/.profile.d/.interactive_profile' they will be available
in all windows.
Date
date :
display time and date
Check 'man date' for different format options.
Evaluate a command
You can
evaluate a command and store the answer in a variable.
For example if you want to store the current date in a variable you can use:
$ current_date=`date`
or
$ current_date=$(date)
$ echo $current_date
Arithmetics
$ year=2013
$ next_year=$((
year + 1 ))
or
$ (( next_year =
year + 1 ))
$ echo $next_year
2014
% : modulo / remainder
$ echo $(( 10 % 3
))
1
Just be careful and never try arethmetic operations on numbers starting with a zero!!!
Formatted print
printf : produces output according to a format.
Example:
$ printf '%3.3d\n' $(( 1 + 15 ))
016
'\n' give you a new line.
For more information check: man 3 printf
History
history
: lists the most recent commands
Exit/Close a window or exit a script
To close a window do not just click on the "x" but type 'exit' and then
press <Enter>.
For each window some temporary files and directories are created with
will only get cleaned up when using 'exit'.
File Operations
Search for Patterns in Files
grep search-string filename [filename...] : to find and type out
lines containing the string in a file
Options:
-i : case insensitive search
-w : whole words only
-v : type out lines that do NOT contain the string
(invert the search)
-c : count the number of lines that contain a match
Search for files
find pathname -name search-string
Useful for finding particular files. find descends the directory tree
beginning at each pathname and locates files that meet the specified
conditions. Wild cards can be used in the search string.
find is a very powerful and useful command. Use 'man find' to find out
more.
It can for example also be used in combination with 'grep':
$ find . -name
'*.scr' -exec grep case {} \;
The above command will print all lines containing the word 'case' of
files with end on '.scr'.
Counting words/lines/characters in a file
wc
filename(s) : counts the
number of words, lines, and characters in a file
Options:
-l : count lines
-c : count characters
-w : count words
Pipe
| :
(pipe) redirect standard output of first command to standard input
of second command
Example:
$ ls -1 | wc -l : count the
number of files in a directory
Compare files
diff filename1 filename2 : compares
contents of
two files on a line-by-line basis
xxdiff filename1 filename2 [filename3] : graphically compares
contents of
two or three files on a line-by-line basis
xxdiff directory1 directory2 : compares
contents of two directories
xxdiff can also be used to edit files!
tkdiff filename1 filename2 : graphically
compares contents of
two files on a line-by-line basis. Use on guillimin onstead of 'xxdiff'.
File Transfer
rsync :
a fast, versatile, remote (and local) file/directory-copying tool
Usage:
rsync origin destination
rsync user@origin_machine:origin
destination
rsync origin user@destination_machine:destination
origin and destination can be file(s) and/or
directories.
Only the origin or the destination machine can be a nother machine, not
both.
Options:
|
-v |
verbose |
|
-r |
Recursively copy entire directories |
|
-u |
update, forces rsync to skip any files which exist on the
destination and have a modified time that is newer than the
source file. |
|
-l |
links, when symlinks are encountered, recreate the symlink
on the destination |
|
-L |
copy-links, When symlinks are encountered, the item that
they point to (the referent) is copied, rather than the symlink. |
|
-t |
preserves time |
|
-p
|
preserves permissions
|
scp : secure copy (remote file copy program)
Usage: similar to rsync above
Options:
|
-r |
Recursively copy entire directories. Note that scp follows
symbolic links encountered in the tree traversal. |
Also see:
sftp : secure file
transfer
program
ftp :
Internet file transfer
program
wget : web downloader
GNU Wget is a free utility for non-interactive download of files from the Web.
Example:
$ wget www.tldp.org/LDP/Bash-Beginners-Guide/Bash-Beginners-Guide.pdf
File archiving and compressions
tar :
tape archive, can create, add to, list, and retrieve files form an
archive file (a collection of files archived together as one file).
Options:
|
-v |
verbose |
|
-c |
create archive
|
|
-t |
table of contents
|
|
-x |
extract files
|
|
-f |
followed
by name of
tar-file
|
|
-t |
preserves time |
Examples:
Create tar-file: tar -cvf
tar-filename.tar filenames
Unarchive tar-file:
tar -xvf tar-filename.tar
gzip, gunzip
: compress resp. expand files
gzip filename :
compresses the file filename.
Whenever possible, each file is replaced(!)
by one with the extension .gz
gunzip filename.gz : uncompresses the gz-file again.
Good to save space!
zip, unzip
: package and compress resp. unarchive and uncompress (archive) files
zip zip-file.zip file_list : archives and compresses all
files in file_list into zip-file.zip. Leaves original files
untouched.
unzip zip-file.zip : unarchives and uncompresses all
files in zip-file.zip.
unzip zip-file.zip aaa : unarchives and uncompresses
only file 'aaa fromzip-file.zip.
Cursor movements in command line
emacs mode:
<Esc> . : repeat last word of previous command
Ctrl-a : jump to beginning of line
Ctrl-e : jump to end of line
Ctrl-r : look backwards into history
Ctrl-c : interrupt current process
Control stuctures
if then else
if expression_1 ; then
Statement(s) to be executed if expression 1 is true
elif expression_2 ; then
Statement(s) to be executed if expression 2 is true
elif expression_3 ; then
Statement(s) to be executed if expression 3 is true
else
Statement(s) to be executed if no expression is true
fi
'elif expression ; then' and 'else' are optional!!!
expressions can look like this:
Arithmetic expression examples ( a=2 ):
(( a == 3 ))
(( b != 3 ))
(( $a >= 3 ))
In Arithmetic expressions '(( ... ))'
it is not necessary to put the '$' infront of variables since strings
are not allowed so all strings are assumed to be variables anyways.
String expressions examples ( abc="Bonjour" ):
equal:
[[ "$abc" == "Hello" ]]
not equal:
[[ "$abc" != "Hello" ]]
The more common but less recomended way is:
[ $a -le 3 ]
resp.
[ "$abc" -nq "Hello" ]
Note: It is mandatory to keep a 'space' just before and after the double and single square brackets!!!
and: &&
or: ||
If you want to know more about the comparison operators try:
$ man test
or
$ man bash -> CONDITIONAL EXPRESSIONS
case-statement
case test_string in
pattern_1) statement(s) ;;
pattern_2) statement(s) ;;
*) default statement(s) ;;
esac
Examples:
mon="Jan"
case "$mon" in
"Jan") echo January
month=January ;;
"Feb") echo February
month=February ;;
*) echo "month not known" ;;
esac
# Determine number of days per month
case ${month} in
04|06|09|11) days=30 ;;
02)
if (( year % 4 == 0 )) && \
( (( year % 100 != 0 )) || (( year % 400 == 0 )) ) ; then
days=29
else
days=28
fi ;;
*) days=31 ;;
esac
for loop
for var in word1 word2 ... wordN ; do
Statement(s) to be executed for every word.
done
Examples:
for hour in 00 06 12 18 ; do
echo $hour
done
touch dm_01 dm_02
for fine in dm* ; do
echo $file
done
while loop
while expression ; do
Statement(s) to be executed while expression is true
done
Example:
a=1
while (( a <= 10 )) ; do
echo $a
a=$(( a + 1 )) # increase the loop parameter
done
You can also use a while-loop to read lines from an ASCII file.
Example:
$ cat > text_file << EOF
> Hello
> How are you?
> EOF
$ while read line ; do echo $line ; done < text_file
Hello
How are you?
break and continue
To interrupt a loop you can use 'break' and 'continue'.
break interrupts the loop
continue interrupts the current cycle of the loop
Manipulating Variables
Example string:
$ string="ABC_abc_123"
String length:
$ echo ${#string}
# removes minimal matching prefixes
## removes maximal matching prefixes
% removes minimal matching suffixes
%% removes maximal matching suffixes
:s:n get 'n' caracters starting from position 's' (first position is 0)
Look at these examples and you will understand the meaning of the above:
$ string="ABC_abc_123"
$ echo ${string#*_}
abc_123
=> removes everything before first occurance of '_'
$ echo ${string##*_}
123
=> removes everything before last occurance of '_'
$ echo ${string%_*}
ABC_abc
=> removes everything after last occurance of '_'
$ echo ${string%%_*}
ABC
=> removes everything after first occurance of '_'
$ echo ${string:s:n}
abc
=> gets n characters, starting from position s
Cut out part of the content of a variable:
cut prints selected parts of a string
Options:
|
|
-c |
select only these characters
|
|
-d |
delimiter=DELIM; use DELIM instead of TAB for field delimiter
|
|
-f |
select only these fields
|
Examples:
$ abc="I am hungry"
$ echo $abc | cut -c 6-9
hung
=> returns characters 6-9
$ echo $abc | cut -c 6-
hungry
=> returns characters 6 to end
$ echo $abc | cut -d' ' -f 3
hungry
=> returns 3rd element with ' ' used as seperator
$ abc="I am hungry. Are you?"
$ echo $abc | cut -d. -f 2
Are
you?
=> returns 2nd element with '.' used as seperator
set
When you call set without
options but with one or more arguments, it sets the values of the
command line argument variables ($1-$9) to its arguments.
Examples:
$ abc="I am hungry"
$ set $abc
$ echo $3
hungry
When options are specified, they set or unset shell attributes.
Options:
|
|
-x |
expands each simple command
This is very useful for debugging scripts!!!
|
|
+x
|
no expanding of commands anymore
|
For more options check man page: man set
Examples:
$ abc="I am hungry"
$ set -x
$ abc="I am hungry"
+ abc='I am hungry'
& -> Send job in background
You can send a job in the background by adding a '&' at the end of the command line.
This is useful to get the prompt back - beeing able to continue using
the window - when calling a program whic opens a nother window, like
for example Matlab, emacs, xrec.
Example:
$ matlab &
In case you forgot to add the '&' you can still send an already
launched job into the background with Ctrl-z followed by 'bg' (for background):
Example:
$ emacs
Ctrl-z
[1]+
Stopped
emacs
$ bg
[1]+ emacs &
Check and kill running processes
ps reports a snapshot of the current processes
Options:
|
|
-e |
select all processes
|
|
-f
|
do full-format listing
|
|
-u userlist
|
this selects the processes whose effective user name or ID is in userlist |
Try for example:
$ ps -fu $USER
If ever you need to kill one of these jobs you can use 'kill' followed by the 'PID'.
If a normal 'kill' does not work try 'kill -9' followed by the 'PID'.
Exercises
Click on the following links for Unix / Linux / Shell exercises and their solutions.
Author: Katja Winger
Last update: May 2013