WCC logo

CIS120Linux Fundementals

Chapter 1

Introduction to Linux

Unix and the Origins of Linux

Unix is a powerful, multiuser, multitasking operating system that originated in the late 1960s and early 1970s at AT&T's Bell Labs. It was initially developed by Ken Thompson, Dennis Ritchie, and others. Unix quickly became the foundation for many subsequent operating systems due to its robustness, flexibility, and portability. Its design philosophy emphasized simplicity and the use of small, modular utilities that could be combined to perform complex tasks. This modularity and efficiency made Unix highly influential in the computing world.

Linux, a Unix-like operating system, was created by Linus Torvalds in 1991 as a free and open-source alternative to the proprietary Unix. Torvalds, a Finnish software engineer, started the Linux project as a personal hobby while he was a student at the University of Helsinki. In 1991, he announced the project on the MINIX newsgroup, inviting others to contribute to the development. Torvalds made the source code available under the GNU General Public License (GPL), allowing anyone to use, modify, and distribute it. This collaborative approach marked the beginning of Linux's journey and its rapid growth.

GNU/Linux

While the term "Linux" is commonly used to refer to the entire operating system, it is technically only the kernel. The complete operating system is often referred to as GNU/Linux, acknowledging the critical role of the GNU Project in its development. The GNU Project, started by Richard Stallman in 1983, aimed to create a free Unix-like operating system. By the early 1990s, most components of the GNU system were ready, except for the kernel. Linus Torvalds' Linux kernel filled this gap, and together with the GNU components, formed a complete free operating system.

Open Source and Its Importance

Open Source Software (OSS) is characterized by its availability for anyone to use, modify, and distribute freely. The source code of OSS is openly shared, which encourages collaboration and community development. This model fosters innovation and rapid development, as developers from around the world can contribute to the project. Key features of open source include transparency, collaboration, freedom to modify the software, and strong community support. Linux, released under the GPL, exemplifies these principles, allowing it to grow and improve through global contributions.

The open-source movement has had a profound impact on the software industry, promoting the development of high-quality, reliable software. Major companies and organizations, including Google, IBM, and NASA, use and contribute to open-source projects. The open-source model also supports the idea of software freedom, where users have control over their software and can tailor it to meet their specific needs.

Linux Architecture

The architecture of Linux is composed of several layers, each serving a specific function within the operating system. At the base is the hardware layer, which includes all the physical devices such as the CPU, memory, and disk drives. Above this is the kernel layer, the core of the operating system that interacts directly with the hardware. The kernel manages critical tasks like memory management, process management, device management, and system calls.

System libraries, which provide functions and system calls for applications to interact with the kernel, sit above the kernel layer. These libraries enable developers to write applications without needing to manage hardware directly. System utilities are specialized programs that perform individual management tasks, such as disk management and network configuration. These utilities offer essential tools for system administrators and users to manage the operating system.

At the top of the architecture are user applications, which are the software programs used by end-users, such as browsers, text editors, and media players. These applications interact with the system libraries and utilities to perform various tasks, providing a user-friendly interface for interacting with the operating system.

Understanding the Kernel

The kernel is the heart of the Linux operating system, managing hardware resources and facilitating communication between hardware and software. It handles essential functions like memory management, process management, device management, and system calls. The kernel ensures that multiple processes can run simultaneously without interfering with each other, providing a stable and efficient computing environment.

There are different types of kernels, including monolithic kernels and microkernels. Monolithic kernels, like the one used in Linux, include all essential services and device drivers in one large block of code running in a single address space. This design can offer performance advantages but may become complex as more features are added. Microkernels, on the other hand, have only the most basic functions, with additional services running in user space. This design can improve stability and security but may introduce performance overhead.

The Linux kernel is known for its modularity, allowing users to load and unload modules dynamically. This feature enables customization and optimization of the operating system for different hardware and use cases. The Linux kernel also supports a wide range of hardware platforms, from embedded systems to supercomputers, making it highly versatile and adaptable.

In summary, Linux is a Unix-like, open-source operating system created by Linus Torvalds. Its development and evolution have been driven by the open-source community, making it a powerful and flexible operating system. Understanding the basics of Linux, including its architecture and the role of the kernel, provides a solid foundation for further exploration of this widely-used and influential software. As an open-source project, Linux continues to evolve, driven by contributions from developers worldwide, ensuring its relevance and innovation in the ever-changing technology landscape.

Understanding the Linux File System

The Linux file system is structured quite differently from Windows. It uses a hierarchical directory structure where everything starts from the root directory, denoted by a single slash (/), and branches out into various other directories each with its specific purpose. In this discussion, we'll cover the main directories you'll encounter in a Linux system and what they're typically used for.

Basic Structure of the Linux File System

Examples of Navigating and Understanding the Linux File System

1. Listing configuration files

You might want to see what configuration files are present for system-wide settings:

ls /etc

2. Checking mounted devices

To see all currently mounted devices and their partitions, you can look at:

cat /proc/mounts

3. Viewing user-specific binaries

If you want to check what binaries are available for all users:

ls /usr/bin

4. Exploring your home directory

To navigate to your home directory and see its contents:

cd ~  # ~ is a shortcut for /home/yourusername
ls

5. Viewing system logs

To check system logs, such as messages related to system functions:

sudo less /var/log/syslog

Conclusion

The Linux file system's hierarchical structure is designed to segregate and organize files in a logical manner, enhancing security, scalability, and manageability. By understanding the role of each directory within this structure, users and administrators can manage their systems more effectively. This model also supports permissions and ownership, making it highly versatile for multi-user and multi-process operations.

pwd, cd and tree Commands

The cd, pwd, and tree commands are essential tools for navigating the Linux file system efficiently. The cd (change directory) command allows users to move between directories, making it possible to access files and organize the system effectively. The pwd (print working directory) command helps users confirm their current location within the file system, which is particularly useful when working with deeply nested directories. The tree command provides a structured view of the directory hierarchy, displaying files and subdirectories in an easy-to-read format. Together, these commands enable users to move through the file system, verify their position, and visualize the directory structure, improving workflow and organization.

pwd Command

The pwd command in Linux stands for "print working directory." This command is used to display the current directory path you are working in. It is especially useful for verifying your location within the file system, which can help avoid confusion when navigating complex directory structures.

The command below outputs the absolute path of the current working directory. For example, if you are in /home/user/Documents, this command will output /home/user/Documents.

pwd

Outputs:

$ pwd
/home/user/Documents

cd Command

The cd command in Linux is used to change the current working directory. This command is fundamental for navigating the file system. Understanding how to use cd effectively will help you move between directories quickly and efficiently.

What the cd Command Does

The basic usage of the cd command changes the current working directory to the specified directory. It allows you to move to different directories, access files, and organize your work environment.

NOTE: In Linux directory structures, the single dot (".") represents the current directory, while the double dot ("..") refers to the parent directory, which is one level above the current directory. The dot (".") is often used to indicate files or commands within the current directory, such as "./script.sh" to run a script located in the directory you are currently in. On the other hand, the double dot ("..") is used to navigate up the directory tree. For example, using "cd .." will move you to the parent directory. These shortcuts provide an efficient way to reference and navigate directories in Linux.

Common Path Shortcuts for the cd Command

Here is a table of some of the most common path shortcuts you can use with the cd command and their descriptions:

Shortcut Description
.. or ../ Move up one directory level (parent directory)
~ Change to the home directory
- Change to the previous directory
/ Change to the root directory
. or ./ Stay in the current directory

Relative and Absolute Paths

Understanding the difference between relative and absolute paths is crucial for effective navigation in the Linux file system.

Absolute Path

An absolute path is the complete path from the root directory to the desired directory or file. It always starts with a forward slash /.

cd /home/user/Documents

This command uses an absolute path to change the current directory to /home/user/Documents.

Relative Path

A relative path specifies a location relative to the current directory. It does not start with a forward slash/.

cd Documents

If you are currently in /home/user, this command uses a relative path to change the directory to /home/user/Documents.

Examples:

Changing to a Specific Directory:

This command changes the current working directory to /home/user/Documents.

cd /home/user/Documents

Moving Up One Directory Level (parent directory):

This command moves the current working directory up one level. For example, if you are in /home/user/Documents, this command will take you to /home/user.

cd ..

Changing to the Home Directory:

This command changes the current working directory to the home directory of the user. For example, if your home directory is /home/user, this command will take you there from any location.

cd ~

Changing to the Root Directory:

This command changes the current working directory to the root directory of the file system.

cd /

Returning to the Previous Directory:

This command changes the current working directory to the previous directory you were in. It is useful for toggling between two directories.

cd -

Staying in the Current Directory:

This command keeps you in the current directory. It is often used in scripts to explicitly state that the current directory should remain unchanged.

cd .

tree Command

The tree command in Linux is used to display the directory structure in a hierarchical format, making it easier to visualize the organization of files and subdirectories. By default, it starts from the current directory and recursively lists all files and directories in a tree-like structure. Users can modify its behavior with options such as -L to limit the depth of recursion or -d to show only directories.

The tree command supports several options to customize its output, as shown in the table below:

Option Description
-L N Limits the depth of directory traversal to N levels. For example, tree -L 2 would just go two levels deep.
-d Displays only directories, excluding files.
-a Shows all files, including hidden ones.
-f Prints the full path for each file and directory.
-h Displays file sizes in a human-readable format.
--du Shows the cumulative disk usage of each directory.
--dirsfirst Lists directories before files in the output.

Below is a sample output of the tree command. In this example, the user is in a directory that contains three subdirectories: Documents, Pictures, and Scripts. The tree command displays the hierarchical structure of these folders and their contents. The Documents folder contains two files, report.docx and notes.txt. The Pictures folder has a subdirectory named vacation, which contains two image files, beach.jpg and sunset.jpg. The Scripts folder includes two script files, script.sh and backup.py. This output provides a clear visual representation of how files and directories are organized within the current working directory.

.
├── Documents
│   ├── report.docx
│   ├── notes.txt
├── Pictures
│   ├── vacation
│   │   ├── beach.jpg
│   │   ├── sunset.jpg
├── Scripts
│   ├── script.sh
│   ├── backup.py

ls Command

The ls command in Linux is used to list directory contents. It is one of the most frequently used commands and provides various options to customize the output to meet different needs. The command displays information about files and directories, including their names, sizes, permissions, and more.

What the ls Command Does

The basic usage of the ls command without any options lists the files and directories in the current directory. By adding various options to the command, you can control the amount and type of information displayed, as well as the format of the output.

Common Options for the ls Command

Here is a table of some of the most popular options for the ls command and their descriptions:

Option Description
-l Use a long listing format
-a Include directory entries whose names begin with a dot (.)
-h With -l, print sizes in human-readable format (e.g., 1K, 234M, 2G)
-R List subdirectories recursively
-t Sort by modification time, newest first
-r Reverse order while sorting
-S Sort by file size, largest first
-1 List one file per line
-d List directories themselves, not their contents
-i Print the index (inode) number of each file. An inode number in Linux is a unique identifier for a file or directory in a file system. Inodes are data structures that store metadata about files, such as their permissions, size, and owner.
-F Append indicator (one of */=>@)
-G Inhibit display of group information
--color Colorize the output (when output is to a terminal)
-p Append a slash (/) to directory names
-Q Enclose entry names in double quotes
--full-time Use a full time format for listings
-v Natural sort of (version) numbers within text

Examples

Basic ls command:

Displays a list of files and directories in the current location.

ls

Output:

file1.txt  file2.txt  dir1  dir2

Long listing format:

Provides detailed information such as file permissions, owner, and size.

ls -l

Output:

total 16
-rw-r--r-- 1 user group 1234 Jun  4 12:34 file1.txt
-rw-r--r-- 1 user group 5678 Jun  4 12:34 file2.txt
drwxr-xr-x 2 user group 4096 Jun  4 12:34 dir1
drwxr-xr-x 2 user group 4096 Jun  4 12:34 dir2

Including hidden files:

Shows all files, including those that start with a dot (.), which are hidden by default.

ls -a

Output:

.  ..  .hiddenfile  file1.txt  file2.txt  dir1  dir2

Human-readable file sizes:

Displays file sizes in KB, MB, or GB instead of bytes.

ls -lh

Output:

total 16K
-rw-r--r-- 1 user group 1.2K Jun  4 12:34 file1.txt
-rw-r--r-- 1 user group 5.6K Jun  4 12:34 file2.txt
drwxr-xr-x 2 user group 4.0K Jun  4 12:34 dir1
drwxr-xr-x 2 user group 4.0K Jun  4 12:34 dir2

Recursive listing:

Lists the contents of all subdirectories as well.

ls -R

Output:

.:
file1.txt  file2.txt  dir1  dir2

./dir1:
file3.txt

./dir2:
file4.txt

Sorting by modification time:

Displays files in order of their last modification, newest first.

ls -lt

Output:

total 16
-rw-r--r-- 1 user group 5678 Jun  4 12:34 file2.txt
-rw-r--r-- 1 user group 1234 Jun  4 12:34 file1.txt
drwxr-xr-x 2 user group 4096 Jun  4 12:34 dir1
drwxr-xr-x 2 user group 4096 Jun  4 12:34 dir2

Reverse order sorting:

Lists files in reverse order, with the oldest files displayed first.

ls -lr

Output:

total 16
drwxr-xr-x 2 user group 4096 Jun  4 12:34 dir2
drwxr-xr-x 2 user group 4096 Jun  4 12:34 dir1
-rw-r--r-- 1 user group 5678 Jun  4 12:34 file2.txt
-rw-r--r-- 1 user group 1234 Jun  4 12:34 file1.txt

Conclusion

The ls command is a powerful tool for navigating and managing files and directories in Linux. By understanding and using the various options available, you can customize the output to suit your needs and efficiently work with your filesystem. Experiment with the different options and combinations to get comfortable with this essential command.

The Less Command

Lesson on the less Command in Linux

Introduction to less

In Linux, less is a command-line utility that allows users to view the contents of a file interactively. It is an improved version of the older more command and provides features such as scrolling both forwards and backwards through text files, searching within files, and navigating large files efficiently.

Basic Usage

To use less, simply type less followed by the name of the file you want to view:

less filename.txt

This opens filename.txt in the less viewer, where you can navigate using the following keys:

Here are the navigation keys you can use while viewing a file with less:

Key Combination Description
Up/Down Arrows or j/k Scroll up and down one line at a time.
Page Up/Page Down or [SPACE]/[b] Scroll up and down one screen at a time.
G Go to the end of the file.
1G or gg Go to the beginning of the file.
/pattern Search forward for a pattern (n for next occurrence).
?pattern Search backward for a pattern (N for previous occurrence).

Examples

  1. Viewing a File:

    less example.log

  2. Navigating:

    • Scroll down using j or the down arrow key.
    • Scroll up using k or the up arrow key.
    • Jump to the end of the file with G.

  3. Searching:

    • Search for the word "error" in the file:
      /error

Common Options

Here are some common options you can use with less:

Option Description
-N Display line numbers.
-i Ignore case in searches.
-S Truncate long lines (useful for wide text files).
-F Quit automatically if entire file can be displayed.
+<line_number> Start viewing the file from a specific line number.

file Command

The file command in Linux is used to determine the type of a file. This command helps users identify files without relying solely on file extensions. The file command examines the content of a file and outputs a human-readable description of the file type.

Common Options for the file Command

The file command comes with various options to refine its behavior. Below is a table of some of the most common options and their descriptions:

Option Description
-b Brief mode: do not prepend filenames to output lines
-i Output MIME type strings rather than the traditional human-readable ones
-f <name> Read the filenames to be examined from name instead of from the command line
-z Try to look inside compressed files
-L Follow symbolic links
-s Read block or character special files
-r Raw mode: do not translate unprintable characters to \ooo notation

Examples

Here are some examples of how to use the file command with different options, along with the expected output:

Determining the File Type

This command checks the type of the file example.txt.

file example.txt

Output

example.txt: ASCII text

Using the -b Option (Brief Mode)

This command outputs only the file type without the filename.

file -b example.txt

Output

ASCII text

Using the -i Option (MIME Type)

This command outputs the MIME type of the file example.txt.

file -i example.txt

Output

example.txt: text/plain; charset=us-ascii

Checking Multiple Files

This command checks the types of multiple files.

file file1.txt file2.jpg file3.zip

Output

file1.txt: ASCII text
file2.jpg: JPEG image data, JFIF standard 1.01
file3.zip: Zip archive data, at least v2.0 to extract

Reading File Names from a File

Each line in filenames.txt should contain the name of a file that actually exists in the directory where you run the command. For example:

file -f filenames.txt

Below is the contents of filenames.txt

document.pdf
image.png
script.sh
binaryfile
textfile.txt
directory/

Output

document.pdf: PDF document, version 1.4
image.png: PNG image data, 800 x 600, 8-bit/color RGB, non-interlaced
script.sh: Bourne-Again shell script, ASCII text executable
binaryfile: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux)
textfile.txt: ASCII text
directory/: directory

Explanation

Looking Inside Compressed Files

This command attempts to look inside the compressed file compressed.tar.gz and identify its contents.

file -z compressed.tar.gz

Output

compressed.tar.gz: gzip compressed data, was "compressed.tar", last modified: ...

This command follows the symbolic link symlink and checks the type of the file it points to. The output depends on the file type of the target. NOTE: We will be looking at symbolic links in Chapter 2

file -L symlink

Output

symlink: ASCII text

Conclusion

The file command is a powerful tool for identifying the types of files in Linux. By understanding and utilizing its various options, you can efficiently determine file types and gain insights into the nature of the files you are working with. Practice using these options to become comfortable with the file command and enhance your file management skills in Linux.

Chapter 2

mkdir Command

The mkdir command in Linux is used to create directories. It is a straightforward command but comes with various options that allow for more advanced directory creation scenarios. Understanding how to use mkdir effectively can help you organize your file system efficiently.

What the mkdir Command Does

The basic usage of the mkdir command creates a single directory with the specified name. You can also create multiple directories at once or nested directories in one command.

Common Options for the mkdir Command

Here is a table of some of the most common options for the mkdir command and their descriptions:

Option Description
-p Create parent directories as needed
-v Print a message for each created directory
-m Set file mode (permissions) for the new directories
--help Display help information about the mkdir command
--version Output version information

Examples

Creating a single directory:

This command creates a single directory named mydirectory.

mkdir mydirectory

Creating multiple directories:

This command creates three directories named dir1, dir2, and dir3.

mkdir dir1 dir2 dir3

Creating nested directories:

This command creates a nested directory structure. If the parent directories do not exist, the -p option ensures they are created.

mkdir -p parent/child/grandchild

Verbose output:

The -v option provides verbose output, confirming the creation of each directory.

mkdir -v mydirectory

Setting permissions while creating a directory:

This command creates a directory named mysecureddir with permissions set to 755 (rwxr-xr-x).

mkdir -m 755 mysecureddir

Using the --help option:

The mkdir --help command displays a help message detailing the usage, available options, and descriptions for the mkdir command. It provides a quick reference for users on how to create directories with different options.

mkdir --help

Output:

Usage: mkdir [OPTION]... DIRECTORY...
Create the DIRECTORY(ies), if they do not already exist.

Mandatory arguments to long options are mandatory for short options too.
  -m, --mode=MODE   set file mode (as in chmod), not a=rwx - umask
  -p, --parents     no error if existing, make parent directories as needed
  -v, --verbose     print a message for each created directory
      --help        display this help and exit
      --version     output version information and exit

Conclusion

The mkdir command is essential for creating directories in Linux. With its various options, you can create single or multiple directories, set specific permissions, and handle nested directories with ease. By practicing these commands and options, you will become proficient in managing directory structures in your Linux environment.

cp, mv and rm Commands

The cp (copy), mv (move), and rm (remove) commands are essential tools in Linux for file and directory management. These commands allow users to copy, move, and delete files and directories. Understanding their options and usage can greatly enhance your efficiency when working with the Linux command line.

The cp Command

The cp command is used to copy files and directories. It can copy a file to another file, multiple files to a directory, or entire directories to another location.

cp [options] source destination

Commonly Used cp Options:

Option Description
-a Archive mode; preserves attributes and copies recursively
-i Prompt before overwriting files
-r Recursively copy directories
-u Copy only when the source file is newer than the destination file or when the destination file is missing
-v Verbose mode; show files as they are copied
-p Preserve file attributes (e.g., timestamps, ownership)
--backup Make a backup of each existing destination file

Examples:

To copy a file to another file:

This copies the contents of file1.txt to file2.txt. If file2.txt does not exist, it is created.

cp file1.txt file2.txt

To copy a file to a directory:

This copies file1.txt to the /home/user/documents/ directory.

cp file1.txt /home/user/documents/

To copy multiple files to a directory:

This copies file1.txt and file2.txt to the /home/user/documents/ directory.

cp file1.txt file2.txt /home/user/documents/

To copy a directory and its contents recursively:

This copies the documents directory and all its contents to the backup directory.

cp -r /home/user/documents /home/user/backup/

To copy files and preserve their attributes:

This copies file1.txt to the documents directory while preserving its attributes.

cp -p file1.txt /home/user/documents/

To copy files with verbose output:

This shows the files being copied in the terminal.

cp -v file1.txt /home/user/documents/

The mv Command

The mv command is used to move files and directories. It can also be used to rename files and directories. When a file or directory is moved to a new location, the original is deleted.

Basic usage of mv:

mv [options] source destination

Commonly Used mv Options:

Option Description
-i Prompt before overwriting files
-u Move only when the source file is newer than the destination file or when the destination file is missing
-v Verbose mode; show files as they are moved
-f Force move by overwriting destination files without prompt
--backup Make a backup of each existing destination file

Examples:

To move a file to another location:

This moves file1.txt to the documents directory.

mv file1.txt /home/user/documents/

To rename a file:

This renames file1.txt to file2.txt.

mv file1.txt file2.txt

To move a directory to another location:

This moves the documents directory to the backup directory.

mv /home/user/documents /home/user/backup/

To move files with verbose output:

This shows the files being moved in the terminal.

mv -v file1.txt /home/user/documents/

To move files and prompt before overwriting:

This prompts before overwriting file1.txt in the documents directory.

mv -i file1.txt /home/user/documents/

To force move files without prompting:

This moves file1.txt to the documents directory without prompting, even if it overwrites an existing file.

mv -f file1.txt /home/user/documents/

The rm Command

The rm command is used to remove files and directories. It is a powerful command that deletes files and directories permanently, so it must be used with caution.

Basic usage of rm:

rm [options] file

Commonly Used rm Options:

Option Description
-i Prompt before every removal
-f Force removal of files without prompting
-r Recursively remove directories and their contents
-v Verbose mode; show files as they are removed
--preserve-root Do not remove the root directory

Examples:

To remove a file:

This removes file1.txt.

rm file1.txt

To remove multiple files:

This removes file1.txt and file2.txt.

rm file1.txt file2.txt

To remove a directory and its contents recursively:

This removes the documents directory and all its contents.

rm -r /home/user/documents/

To remove files with verbose output:

This shows the files being removed in the terminal.

rm -v file1.txt

To force remove files without prompting:

This removes file1.txt without prompting, even if it is write-protected.

rm -f file1.txt

To prompt before every removal:

This prompts for confirmation before removing file1.txt.

rm -i file1.txt

Summary

The cp, mv, and rm commands are powerful tools for managing files and directories in Linux. The cp command allows for copying files and directories with various options to preserve attributes, prompt before overwriting, and provide verbose output. The mv command enables moving and renaming files and directories, with options to prompt before overwriting, force move, and provide verbose output. The rm command allows for removing files and directories, with options to force removal, prompt before deletion, and provide verbose output. By mastering these commands and their options, you can efficiently manage your files and directories in the Linux environment.

Creating Symbolic and Hard Links

Understanding the ln Command and Paths in Linux

The ln command in Linux is used to create links between files. Links are pointers that allow multiple filenames to refer to the same file content. There are two types of links: hard links and symbolic (or soft) links. Understanding the difference between these types and how to use the ln command can greatly enhance your file management capabilities in Linux. Additionally, knowing how to use relative and absolute paths is crucial when creating symbolic links.

A hard link is a direct reference to the physical data on the disk. When you create a hard link, it points to the same inode (index node) as the original file. This means that both the original file and the hard link are indistinguishable from each other. Any changes made to one will reflect in the other because they both point to the same data blocks. However, deleting one will not affect the other; the data remains accessible as long as there is at least one link pointing to it.

To create a hard link:

ln file1.txt file1_hardlink.txt

This command creates a hard link named file1_hardlink.txt pointing to file1.txt. Both filenames now refer to the same data.

A symbolic link, or soft link, is a reference that points to another file by its pathname. Unlike hard links, symbolic links are independent of the physical data. They act as shortcuts or aliases to the original file. If the original file is moved or deleted, the symbolic link becomes a broken link and no longer functions. Symbolic links can span different filesystems and can link to directories, whereas hard links cannot.

To create a symbolic link:

ln -s file1.txt file1_symlink.txt

This command creates a symbolic link named file1_symlink.txt pointing to file1.txt. If file1.txt is deleted or moved, file1_symlink.txt will become a broken link.

Examples and Differences

Creating Hard and Symbolic Links:

ln file1.txt file1_hardlink.txt
ln -s file1.txt file1_symlink.txt

These commands create a hard link and a symbolic link to file1.txt.

Modifying Content: If you modify the contents of file1.txt, both file1_hardlink.txt and file1_symlink.txt will reflect those changes because they reference the same data (for the hard link) or the same file (for the symbolic link).

Deleting Files:

Deleting file1.txt:

rm file1.txt

The hard link file1_hardlink.txt still retains access to the data, whereas the symbolic link file1_symlink.txt becomes a broken link.

File System Boundaries: Hard links cannot be created across different filesystems. Attempting to do so results in an error. Symbolic links, however, can point to files on different filesystems without any issues.

Linking to Directories: Hard links cannot be used to link directories due to the potential for filesystem inconsistencies. Symbolic links can link to directories without any problems.

ln -s /home/user/documents mydocuments_symlink

This command creates a symbolic link named mydocuments_symlink pointing to the /home/user/documents directory.

Relative Path Requirements for symbolic Links

Symbolic links require the path to be relative to their location. For example, if you have directory1 and directory2 in a directories folder and directory1 has file1.txt, to create a hard link in directory2 that points to file1.txt in directory1, you would use:

ln directory1/file1.txt directory2/hardln

For a symbolic link, the relative path matters:

ln -s ../directory1/file1.txt directory2/symbolicln

The ../ is necessary for the symbolic link because it needs the relative path based upon the locate of where the symbolic link is located. Since the symbolic link is in directory2 the path to the source file (file 1) must go up one level (out of directory2) and then into directory 1.

Understanding Relative and Absolute Paths

When creating links using the ln command, it's important to understand the concepts of relative and absolute paths. These concepts are crucial for correctly creating symbolic (symbolic) links, as the path you provide determines the link's functionality and reliability.

Absolute Paths: An absolute path specifies the complete directory location from the root directory (/). It provides the full path to a file or directory, regardless of the current working directory. Absolute paths are useful when you want to ensure that a link always points to the correct location, regardless of where the link is accessed from.

For example, if you want to create a symbolic link to a file located at /home/user/documents/file1.txt from any directory, you would use an absolute path:

ln -s /home/user/documents/file1.txt /home/user/links/file1_symlink.txt

This command creates a symbolic link named file1_symlink.txt in the /home/user/links directory, pointing to /home/user/documents/file1.txt.

Relative Paths: A relative path specifies the location of a file or directory in relation to the current working directory. It does not start with a slash (/) and is interpreted based on the current directory. Relative paths are useful when the structure of the directories is known and fixed, allowing for more flexible and portable links.

For example, consider the following directory structure:

/home/user/
    ├── documents/
    │   └── file1.txt
    └── links/

If you are in the /home/user/links directory and want to create a symbolic link to file1.txt located in documents, you would use a relative path:

ln -s ../documents/file1.txt file1_symlink.txt

This command creates a symbolic link named file1_symlink.txt in the current directory (/home/user/links), pointing to ../documents/file1.txt.

Summary

The ln command is a powerful tool for creating links between files in Linux. Hard links directly reference the physical data on the disk and can only be used within the same filesystem. They provide a robust way to access the same data through different filenames. symbolic links act as shortcuts to the original file or directory and can span different filesystems. They are more flexible but can become broken if the original file is deleted or moved. By mastering the use of hard and symbolic links, and understanding the differences between relative and absolute paths, you can efficiently manage and organize your files and directories in the Linux environment.

Chapter 3

Type Command

The type command in Linux is used to explain how a given command name will be interpreted when executed in the command line. It helps determine whether a command is a built-in shell command, an alias, a function, or an external executable program. Understanding this distinction is essential for effective command-line usage and troubleshooting.

Built-in Shell Command

A built-in shell command is a command that is executed directly by the shell itself, without calling any external program. These commands are integral to the shell and are typically used for managing the shell environment. For example, cd, which changes the current directory, is a built-in command.

Example:

type cd

Output:

cd is a shell builtin

Alias

An alias is a shortcut or a substitute for another command or a series of commands. It is defined by the user to simplify command execution. For example, the ls command might be aliased to ls --color=auto to always display directory listings with color.

Example:

type ls

Output:

ls is aliased to 'ls --color=auto'

Function

A function in the shell is a user-defined set of commands grouped together under a single name. Functions can include any number of commands and can be used to automate repetitive tasks. Functions are defined in the shell and executed directly by it.

Example:

type my_function

Output (assuming my_function is defined):

my_function is a function
my_function () 
{ 
    echo "This is a custom function"
}

External Executable Program

An external executable program is a standalone binary or script located in the filesystem, which the shell executes by spawning a new process. These programs reside in directories specified in the system's PATH environment variable. For example, grep is an external program used for pattern matching within text.

Example:

type grep

Output:

grep is /bin/grep

Summary

Using the type command allows users to discern how commands will be interpreted and executed, aiding in efficient command-line usage and troubleshooting.

Help and Man commands

In Linux, the help and man commands are essential tools for users to understand and utilize the vast array of commands and utilities available in the system. These commands provide detailed information and documentation, aiding both beginners and experienced users in mastering Linux commands.

The help Command

The help command is primarily used to display information about built-in shell commands. Since built-in commands are executed directly by the shell and not as separate programs, they often do not have separate manual pages. The help command provides a concise reference for these built-ins, explaining their usage and options.

To use the help command, simply type help followed by the name of the built-in command. For example:

help cd

This command will display information about the cd command, including its syntax and available options. The help command is specific to built-ins and will not work with external commands.

Common Options for the help Command

Option Description
-d Display a brief description of the command.
-m Display usage in a pseudo-manpage format.
-s Display a short usage synopsis for the command.
--help Display help information for the help command.

The man Command

The man (manual) command provides comprehensive documentation for a wide range of commands and programs in Linux. Each command or program typically has a manual page that describes its purpose, usage, options, and examples. The man pages are organized into sections, each covering a different category of commands or functions.

To use the man command, type man followed by the name of the command or program. For example:

man ls

This command will open the manual page for the ls command, which lists directory contents. The manual page includes a description of the command, its syntax, available options, and examples of how to use it.

Common Options for the man Command

Option Description
-k Search the short descriptions and manual page names for the keyword.
-f Display a one-line description of the command (equivalent to whatis).
-a Display all the manual pages for the command.
-w Print the location of the manual page files instead of displaying them.
--help Display a help message with available options.

Examples

Using help for a built-in command:

help echo

Output:

echo: echo [-neE] [arg ...]
    Write arguments to the standard output.
    
    Options:
      -n    do not append a newline
      -e    enable interpretation of backslash escapes
      -E    disable interpretation of backslash escapes (default)
    
    Exit Status:
    Returns success unless a write error occurs.

Using man for an external command:

man grep

Output:

GREP(1)                           User Commands                          GREP(1)

  NAME
         grep - print lines that match patterns
  
  SYNOPSIS
         grep [OPTION]... PATTERNS [FILE]...
         grep [OPTION]... -e PATTERNS ... [FILE]...
         grep [OPTION]... -f FILE ... [FILE]...
  
  DESCRIPTION
         grep searches for PATTERNS in each FILE.  A FILE of “-” stands for standard input.
         By default, grep prints the matching lines.
  
         In addition, three variant programs egrep, fgrep and rgrep are available:
         egrep is the same as ‘grep -E’.
         fgrep is the same as ‘grep -F’.
         rgrep is the same as ‘grep -r’.
  
  OPTIONS
         -i, --ignore-case
                Ignore case distinctions in patterns and input data.
  
         -v, --invert-match
                Select non-matching lines.
  
         -r, --recursive
                Read all files under each directory, recursively.
  
         -n, --line-number
                Prefix each line of output with the line number.
  
         -c, --count
                Suppress normal output; print a count of matching lines for each input file.
  
         -l, --files-with-matches
                Suppress normal output; print the name of each input file that contains a match.
  
         -w, --word-regexp
                Select only those lines containing matches that form whole words.
  
         -h, --no-filename
                Suppress the prefixing of file names on output.
  
         --help
                Display help and exit.
  
         --version
                Output version information and exit.
  
  EXAMPLES
         grep 'error' logfile
                Print all lines containing "error" in logfile.
  
         grep -i 'hello' file.txt
                Print lines containing "hello", case insensitive.
  
         grep -r 'TODO' ~/projects/
                Recursively search for "TODO" in all files under ~/projects/.
  
  SEE ALSO
         sed(1), awk(1), find(1), xargs(1)
  
  GNU Grep 3.7                        April 2023                            GREP(1)

Navigating the Man Pages

Within the man pages, you can navigate using the arrow keys to scroll up and down. Press q to quit and return to the command prompt.

Organization of man Pages

Man pages are organized into sections based on the type of command or documentation they contain. Here are the common sections:

Section Description
1 User commands (executable programs or shell commands).
2 System calls (functions provided by the kernel).
3 Library calls (functions within program libraries).
4 Special files (usually found in /dev).
5 File formats and conventions.
6 Games and screensavers.
7 Miscellaneous (macro packages, conventions, etc.).
8 System administration commands (usually only for root).

Accessing Specific Sections

If a command has entries in multiple sections, you can specify the section number. For example, to view the manual page for the passwd command related to user account management (section 1), use:

man 1 passwd

Apropos, Info and Whatis commands

In Linux, the apropos, info, and whatis commands are essential tools for finding and understanding command-line utilities and their functions. These commands help users navigate the vast amount of documentation available in Linux systems, making it easier to find relevant information about commands and programs.

The apropos Command

The apropos command searches the manual page names and descriptions for a keyword or phrase, returning a list of commands and their brief descriptions that match the search criteria. This is particularly useful when you know what a command should do but do not remember the exact command name. The apropos command is often used to find commands related to a specific topic.

Example Usage:

apropos copy

Output:

bcopy (3)             - copy byte sequence
    cp (1)                - copy files and directories
    copy_file_range (2)   - copy a range of data from one file to another
    dd (1)                - convert and copy a file
    memccpy (3)           - copy memory area
    memcpy (3)            - copy memory area
    memmove (3)           - copy memory area
    scopy (3)             - copy vectors
    strcpy (3)            - copy a string
    strlcpy (3)           - copy a string with size limit
    strncpy (3)           - copy a fixed number of characters from a string

Common Options for the apropos Command

Option Description
-a Combine multiple search keywords with a logical AND.
-e Use exact matching for search keywords.
-l Display output in a long listing format.
-s Search only within the specified manual section(s).
-w Search for whole words only.

The info Command

The info command provides access to the GNU info system, which contains detailed documentation for many commands and programs. Unlike the man pages, which provide a single, often concise entry, info pages can contain more comprehensive and structured documentation. The info system uses a tree structure and hyperlinks to navigate from one section to another, making it easier to explore detailed documentation.

The tree structure of info documentation allows users to start from a top-level menu and navigate through various nodes (sections), similar to browsing a website. Each node can contain links to other nodes, enabling users to jump directly to related topics or subtopics.

Example Usage:

info bash

This command will open the info documentation for the Bash shell, providing detailed information on its features and usage. The top-level menu will look something like this:

File: bash.info,  Node: Top,  Next: Introduction,  Up: (dir)

Bash (GNU Bourne-Again SHell)
*****************************

This manual is for Bash, the GNU command-line interpreter. This edition
describes Bash version 5.1.

* Menu:

* Introduction::                An introduction to the shell.
* Basic Shell Features::        The shell's basic features.
* Shell Builtin Commands::      Commands that are built into the shell.
* Shell Variables::             Variables used by the shell.
* Shell Arithmetic::            Shell arithmetic.
* Job Control::                 Job control features.
* Shell Commands::              Commands defined internally by the shell.
* Shell Functions::             Shell functions.
* Shell Scripts::               Shell scripts.
* Shell Grammar::               Shell grammar rules.
* Command Line Editing::        Command line editing features.
* Installing Bash::             How to build and install Bash.
* Reporting Bugs::              How to report bugs in Bash.
* GNU Free Documentation License::  Your rights under this license.
* Index::                       Index of concepts and functions.

To navigate through the documentation, you can use the arrow keys to move between nodes or press Enter to follow a hyperlink. For example, selecting "Basic Shell Features" and pressing Enter will take you to a detailed section on basic shell features.

Common Options for the info Command

Option Description
--all Display all available info documentation.
--apropos Search all indexes for the specified string.
--directory Add the specified directory to the list of info directories.
--output Specify a file to which to write the output.
--help Display a help message with available options.
Key Function
h Open the help menu
? Show help summary
q Quit the info viewer
Space Scroll forward one page
Backspace Scroll backward one page
n Move to the next node
p Move to the previous node
u Move up one level in the hierarchy
l Move back to the last visited node
t Move to the top (root) of the documentation
m Open a menu (then type the menu item name)
Enter Follow a hyperlink (when the cursor is on it)
Tab Jump to the next hyperlink
Shift + Tab Jump to the previous hyperlink
[ Move to the last node visited
] Move to the next node in the sequence

The whatis Command

The whatis command displays a one-line description of a specified command. It is used to quickly get an overview of what a command does, without delving into the full documentation provided by the man or info commands. The whatis command searches the manual page database for the given command name and displays the summary line from the relevant manual page.

Example Usage:

whatis ls

Output:

ls (1)               - list directory contents

Common Options for the whatis Command

Option Description
-l Display output in a long listing format.
-w Search for whole words only.
-r Use a regular expression for the search pattern.
-s Search only within the specified manual section(s).
--help Display a help message with available options.

The cat Command

The cat (short for "concatenate") command is one of the most frequently used commands in Linux, providing versatile functionalities such as reading and concatenating files, creating new files, and appending data to existing ones. Mastering the cat command is essential for efficient file management in a Linux environment.

Syntax

cat [OPTION]... [FILE]...

Common Options

Option Description
-A Show all, equivalent to -vET
-b Number non-blank output lines
-e Equivalent to -vE
-E Display $ at the end of each line
-n Number all output lines
-s Suppress repeated empty output lines
-T Display TAB characters as ^I
-v Use ^ and M- notation, except for LFD and TAB

Examples

To display the contents of a file, use:

cat filename.txt

This command outputs the contents of filename.txt to the terminal.

If you want to concatenate multiple files, you can use:

cat file1.txt file2.txt

This merges the contents of file1.txt and file2.txt and displays the combined content.

Creating a new file can be done with:

cat > newfile.txt
    This is a new file created using the cat command.
    Press Enter then Ctrl+D to save and exit.

Output:

cat newfile.txt
    This is a new file created using the cat command.

This command creates newfile.txt with the provided content.

To append data to an existing file, use:

cat >> existingfile.txt
    Appending this text to the existing file.
    Press Enter then Ctrl+D to save and exit.

Output:

cat existingfile.txt
    Appending this text to the existing file.

This appends the provided content to existingfile.txt.

For displaying line numbers, the command is:

cat -n filename.txt

This displays the contents of filename.txt with line numbers.

If you want to suppress repeated empty lines, use:

cat -s filename.txt

This command displays the contents of filename.txt but suppresses repeated empty lines.

To display $ at the end of each line, use:

cat -E filename.txt

This command displays the contents of filename.txt and adds a $ at the end of each line.

To show all characters, including non-printing characters, use:

cat -A filename.txt

Suppose file.txt contains:

John Doe
    Jane Smith
    
    Mike Johnson

Output:

John Doe$
    Jane Smith$
    $
    Mike Johnson$

This command displays the contents of filename.txt with non-printing characters made visible.

You can combine multiple options as well:

cat -nE filename.txt

This command displays the contents of filename.txt with line numbers and $ at the end of each line.

Summary

The cat command is a powerful tool in Linux for displaying, creating, and concatenating files. Understanding its options and applications can greatly enhance your efficiency when working with text files in the terminal.

The sort and uniq Commands

The sort and uniq commands in Linux are essential for organizing and processing text files. They are often used together to sort and remove duplicates from lists of data. Understanding these commands and their options can significantly improve your efficiency in managing text data.

The sort Command

The sort command sorts lines of text files. It arranges the lines in a specified order, either alphabetically or numerically. The basic syntax for the sort command is:

sort [OPTION]... [FILE]...

Common Options for sort

Option Description
-b Ignore leading blanks
-d Consider only blanks and alphanumeric characters
-f Fold lowercase to uppercase characters
-g General numeric sort
-i Consider only printable characters
-M Sort by month name
-n Numeric sort
-r Reverse the result of comparisons
-k Sort via a key

To sort the contents of a file alphabetically, use:

sort filename.txt

This command sorts the lines in filename.txt alphabetically. For a numeric sort, where lines are sorted based on numerical values, use:

sort -n filename.txt

If you want to sort the lines in reverse order, the -r option can be used:

sort -r filename.txt

Combining options allows for more specific sorting, such as ignoring leading blanks and sorting numerically:

sort -bn filename.txt

The uniq Command

The uniq command filters out repeated lines in a file, displaying only unique lines. It is often used after the sort command because uniq works only on adjacent duplicate lines. The basic syntax for the uniq command is:

uniq [OPTION]... [FILE]...

Common Options for uniq

Option Description
-c Prefix lines by the number of occurrences
-d Only print duplicate lines
-u Only print unique lines
-i Ignore differences in case when comparing
-f Skip fields before comparing
-s Skip characters before comparing

To remove duplicate lines from a file, first sort the file and then use uniq:

sort filename.txt | uniq

This command sorts the lines in filename.txt and then filters out duplicate lines. To count the occurrences of each line, use the -c option:

sort filename.txt | uniq -c

This will prefix each line with the number of times it appears in the file. If you want to display only the duplicate lines, use the -d option:

sort filename.txt | uniq -d

For displaying only unique lines, use the -u option:

sort filename.txt | uniq -u

Examples

Sorting a file alphabetically:

sort fruits.txt

Sorting a file numerically:

sort -n numbers.txt

Sorting a file and ignoring leading blanks:

sort -b names.txt

Sorting a file in reverse order:

sort -r items.txt

Removing duplicate lines from a sorted file:

sort animals.txt | uniq

Counting the occurrences of each line:

sort animals.txt | uniq -c

Displaying only duplicate lines:

sort colors.txt | uniq -d

Displaying only unique lines:

sort colors.txt | uniq -u

Summary

The sort and uniq commands are powerful tools for organizing and processing text data in Linux. sort arranges lines in a specified order, while uniq filters out repeated lines. Mastering these commands and their options will enable you to efficiently manage and manipulate text files in your Linux environment.

The head, tail and wc Commands

The head, tail, and wc commands in Linux are fundamental tools for handling text files and streams. Each command serves a specific purpose, allowing users to view and analyze file content efficiently.

The head Command

The head command is used to display the beginning of a file. By default, it shows the first ten lines, but you can specify the number of lines to display. The basic syntax is:

head [OPTION]... [FILE]...

Common Options for head

Option Description
-n Print the first NUM lines
-c Print the first NUM bytes
-q Never print headers
-v Always print headers

For example, to display the first 5 lines of a file, use:

head -n 5 filename.txt

To display the first 20 bytes of a file, use:

head -c 20 filename.txt

The tail Command

The tail command is used to display the end of a file. Like head, it shows the last ten lines by default but can be customized to show a specific number of lines or bytes. The basic syntax is:

tail [OPTION]... [FILE]...

Common Options for tail

Option Description
-n Print the last NUM lines
-c Print the last NUM bytes
-f Output appended data as the file grows (useful for logs)
-q Never print headers
-v Always print headers

For example, to display the last 10 lines of a file, use:

tail -n 10 filename.txt

To follow a file and display new lines as they are added (useful for monitoring log files), use:

tail -f filename.txt

The wc Command

The wc (word count) command is used to count the number of lines, words, and bytes in a file. It provides a quick way to get an overview of a file's size. The basic syntax is:

wc [OPTION]... [FILE]...

Common Options for wc

Option Description
-l Print the newline counts
-w Print the word counts
-c Print the byte counts
-m Print the character counts
-L Print the length of the longest line

For example, to count the number of lines in a file, use:

wc -l filename.txt

To count the number of words in a file, use:

wc -w filename.txt

To count the number of bytes in a file, use:

wc -c filename.txt

Examples

Displaying the first 10 lines of a file:

head filename.txt

Displaying the last 15 lines of a file:

tail -n 15 filename.txt

Counting the number of lines, words, and bytes in a file:

wc filename.txt

Counting the number of characters in a file:

wc -m filename.txt

Summary

The head, tail, and wc commands are essential for viewing and analyzing file contents in Linux. The head command allows you to view the beginning of a file, while the tail command lets you see the end. The wc command provides counts of lines, words, characters, and bytes, offering a quick way to understand the size and structure of a file. Mastering these commands will enhance your ability to manage and analyze text files efficiently in a Linux environment.

Redirection

Redirection in Linux is a fundamental concept that allows users to control the input and output of commands. By default, commands take input from the keyboard and send output to the terminal. Redirection modifies this behavior to send output to files, read input from files, and handle errors effectively.

Standard Output and Input Redirection

The most basic form of redirection involves redirecting the standard output (stdout) of a command to a file. This is done using the > operator. For example:

ls > filelist.txt

This command lists the directory contents and writes the output to filelist.txt, overwriting the file if it exists. To append the output to an existing file instead of overwriting it, the >> operator is used:

echo "Additional line" >> filelist.txt

This command adds "Additional line" to filelist.txt, preserving its existing contents.

Standard Error Redirection

In addition to standard output, Linux commands also produce standard error (stderr) messages. These messages can be redirected to a file using the 2> operator. For example:

ls non_existent_file 2> errorlog.txt

This command attempts to list a non-existent file, redirecting the resulting error message to errorlog.txt. If you want to append error messages to an existing file instead of overwriting it, use the 2>> operator:

ls another_non_existent_file 2>> errorlog.txt

Combining Standard Output and Error Redirection

Often, it's useful to redirect both stdout and stderr to the same file. This can be done using the &> operator:

ls existing_file non_existent_file &> combinedlog.txt

This command lists the contents of existing_file and attempts to list a non-existent file, redirecting both the output and error messages to combinedlog.txt. To append both stdout and stderr to an existing file, use the &>> operator:

ls another_existing_file another_non_existent_file &>> combinedlog.txt

Redirecting Input

Redirection can also be used to take input from a file instead of the keyboard. This is done using the < operator. For example:

sort < unsorted_list.txt

This command sorts the contents of unsorted_list.txt and displays the sorted list on the terminal.

The grep Command

The grep command in Linux is a powerful tool used for searching and filtering text. It searches through files or input for lines that match a given pattern and prints the matching lines. The name "grep" stands for "global regular expression print," reflecting its ability to search globally for lines matching a pattern and print them. This command is especially useful for sifting through large files or streams of data to find specific information quickly and efficiently. The basic syntax for the grep command is:

grep [OPTION]... PATTERN [FILE]...

Common Options for grep

Option Description
-i Ignore case distinctions
-v Invert the match, displaying lines that do not match
-r Recursively search directories
-l Print the names of files with matching lines
-n Prefix each line of output with the line number
-c Print only a count of matching lines per file
-w Match whole words only
-e Specify multiple patterns to search for
-A Print lines after the matching line
-B Print lines before the matching line
-C Print lines around the matching line

Examples

To search for a specific pattern in a file, you can use:

grep "pattern" filename.txt

This command searches for "pattern" in filename.txt and prints all lines containing that pattern. If you want to ignore case distinctions, use the -i option:

grep -i "pattern" filename.txt

This command searches for "pattern" in a case-insensitive manner. To invert the match, showing lines that do not contain the pattern, use the -v option:

grep -v "pattern" filename.txt

To search recursively through directories, you can use the -r option:

grep -r "pattern" /path/to/directory

This command searches for "pattern" in all files within /path/to/directory and its subdirectories. If you want to print the names of files containing the matching lines, use the -l option:

grep -l "pattern" *.txt

For including line numbers with the matched lines, use the -n option:

grep -n "pattern" filename.txt

To count the number of matching lines per file, use the -c option:

grep -c "pattern" filename.txt

To match whole words only, preventing partial matches, use the -w option:

grep -w "pattern" filename.txt

If you need to specify multiple patterns, you can use the -e option:

grep -e "pattern1" -e "pattern2" filename.txt

To print lines after the matching line, use the -A option followed by the number of lines to display:

grep -A 3 "pattern" filename.txt

To print lines before the matching line, use the -B option:

grep -B 3 "pattern" filename.txt

For printing lines around the matching line, use the -C option:

grep -C 3 "pattern" filename.txt

Summary

The grep command is a versatile and powerful tool for searching and filtering text in Linux. By understanding and utilizing its various options, you can efficiently locate and manage data within files and directories. Whether you need to search for a specific pattern, count occurrences, or view surrounding lines, grep provides the functionality needed to handle these tasks effectively.

Linux Pipelines

Pipelines in Linux are a powerful feature that allows users to connect multiple commands together, passing the output of one command as the input to the next. This enables complex operations to be performed in a streamlined and efficient manner, utilizing the strengths of different commands to process data in a sequential flow. The pipe operator (|) is used to create a pipeline, facilitating this flow of data between commands.

Using pipelines, you can combine simple commands to perform complex tasks without the need for intermediate files. For example, to find all files in a directory that contain a specific string, you could use a combination of grep and ls. By using a pipeline, the output of ls (a list of files) is passed directly to grep to search for the string:

ls | grep "pattern"

This command lists all files in the current directory and filters those that contain "pattern" in their names. Another common use of pipelines is to filter and sort data. For instance, you can use cat to display a file's contents, grep to filter lines containing a specific string, and sort to sort the filtered lines:

cat filename.txt | grep "pattern" | sort

This command displays the contents of filename.txt, filters lines containing "pattern", and sorts the resulting lines. Pipelines can also be used to process numerical data. For example, you can use ps to list running processes, grep to filter for specific processes, and wc to count the number of matching processes:

ps aux | grep "process_name" | wc -l

This command lists all running processes, filters for those containing "process_name", and counts the number of matching processes. Additionally, pipelines can be used for text manipulation and analysis. For instance, you can use cat to display a file's contents, tr to translate or delete characters, and cut to extract specific fields:

cat data.txt | tr '[:lower:]' '[:upper:]' | cut -d',' -f1

This command converts the contents of data.txt to uppercase and extracts the first field (assuming the fields are comma-separated). Pipelines are also useful for monitoring system performance. You can use top to display system information, head to display the top lines, and tee to save the output to a file while also displaying it:

top -b -n 1 | head -n 10 | tee top_output.txt

This command runs top in batch mode, displays the first 10 lines, and saves the output to top_output.txt.

Summary

Pipelines in Linux provide a flexible and efficient way to connect commands and process data sequentially. By passing the output of one command as the input to another, pipelines enable the creation of powerful command chains that can perform complex tasks with minimal effort. Understanding and utilizing pipelines can greatly enhance your productivity and capability in the Linux environment.

The tee Command

The tee command in Linux is a versatile tool used for reading from standard input and writing to both standard output and one or more files simultaneously. This command is particularly useful when you need to view the output of a command on the screen while also saving it to a file. The name "tee" is derived from the T-splitter used in plumbing, symbolizing its ability to split the input data stream into two directions. The basic syntax for the tee command is:

tee [OPTION]... [FILE]...

Common Options for tee

Option Description
-a Append the output to the file instead of overwriting it
-i Ignore interrupt signals
-p Diagnose errors writing to non-pipes

For example, to write the output of a command to a file and display it on the terminal simultaneously, you can use:

ls -l | tee output.txt

This command lists the directory contents in long format, displays it on the terminal, and writes it to output.txt. If you want to append the output to an existing file instead of overwriting it, use the -a option:

ls -l | tee -a output.txt

This command appends the directory listing to output.txt without overwriting the existing content. The -i option is used to ignore interrupt signals, ensuring that the tee command continues to run even if you accidentally press Ctrl+C:

ls -l | tee -i output.txt

The tee command can also be used to write output to multiple files. For instance:

ls -l | tee output1.txt output2.txt

This command writes the output of ls -l to both output1.txt and output2.txt, while still displaying it on the terminal.

Examples

Writing output to a file and terminal:

echo "Hello, World!" | tee hello.txt

Appending output to an existing file:

echo "Additional line" | tee -a hello.txt

Ignoring interrupt signals:

ls -l | tee -i filelist.txt

Writing output to multiple files:

echo "Multi-file output" | tee file1.txt file2.txt

Listing all files in a directory, saving to getall.txt, and filtering lines containing "hello" to hello.txt:

ls -l | tee getall.txt | grep "hello" > hello.txt

Summary

The tee command is a powerful and flexible utility for duplicating the output of a command, allowing it to be both displayed on the terminal and saved to one or more files. This dual functionality makes it an invaluable tool for tasks such as logging command output or capturing real-time data for later analysis. Understanding and utilizing the tee command's options can greatly enhance your ability to manage and redirect command output effectively in a Linux environment.

Chapter 4

Linux Expansion

In Linux, expansion is the process where the shell takes certain special characters or patterns in commands and translates them into a more concrete form before executing the command. This allows you to use shortcuts or patterns to save time and effort when interacting with the system. There are several types of expansions in Linux that you’ll use often, including pathname expansion, tilde expansion, arithmetic expansion, brace expansion, parameter expansion, and command substitution. Let's explore these one by one.

Pathname Expansion (Wildcard Expansion)

Pathname expansion, or wildcard expansion, is when the shell interprets special characters like * and ? to match files and directories.

Examples:

This will list all files and directories in the current working directory.

echo *

This will list all files in the current directory with a .txt extension.

echo *.txt

This will match files like file1.txt, fileA.txt, but not file10.txt (because ? only matches one character).

ls file?.txt

This will match all .jpg files whose names start with any letter between a and c.

ls [a-c]*.jpg

Tilde Expansion

Tilde expansion uses the ~ symbol to represent the current user’s home directory, or another user’s home directory.

Examples:

Changes to your home directory.

cd ~

Lists the contents of your Documents folder within your home directory.

ls ~/Documents

Expands to the home directory of another user named username. This will just print the expanded path to the terminal

echo ~username

Copies a file from your Downloads directory to your Documents directory.

cp ~/Downloads/file.txt ~/Documents/

Arithmetic Expansion

Arithmetic expansion lets you perform mathematical operations directly in your commands. It uses the format $((expression)), where expression is a mathematical formula.

Examples:

Outputs 8 by adding 5 and 3.

echo $((5 + 3))

Outputs 7 by subtracting 2 from 9.

echo $((9 - 2))

Outputs 12 by multiplying 2 and 6.

echo $((2 * 6))

Outputs 5 by dividing 10 by 2.

echo $((10 / 2))

Follows the correct order of operations and outputs 13 (5 * 2 = 10, then 3 + 10 = 13).

echo $((3 + 5 * 2))

The parentheses ensure that 5 + 3 is evaluated first (resulting in 8), then multiplied by 2 to get 16.

echo $(( (5 + 3) * 2 ))

Brace Expansion

Brace expansion generates arbitrary strings by expanding text enclosed in braces {}. This is very useful for creating sequences of similar file or directory names.

Examples:

Expands to fileA.txt, fileB.txt, fileC.txt.

echo file{A,B,C}.txt

Creates directories folder_1, folder_2, ..., folder_5.

mkdir folder_{1..5}

Creates files image_001.png, image_002.png, ..., image_005.png.

touch image_{001..005}.png

Expands to project_alpha_test, project_beta_test, and project_gamma_test.

echo project_{alpha,beta,gamma}_test

Parameter Expansion

Parameter expansion retrieves the value of variables and allows for operations on those variables. Variables in Linux can store data like strings or numbers and are often used in scripts.

Examples:

Displays the value of the HOME variable, which is your home directory path.

echo $HOME

Displays the value of the USER variable, which is your current username.

echo $USER

Displays the list of directories where the system searches for executable files.

echo ${PATH}

You can also modify variables using parameter expansion. For instance, setting a default value if a variable isn’t set:

echo ${UNSET_VARIABLE:-default_value}

This outputs default_value if UNSET_VARIABLE is not defined.

Command Substitution

Command substitution allows the output of one command to be used as an argument in another command. The syntax is either $(command) or `command`.

Examples:

This will print the current date by substituting the output of the date command.

echo "The current date is: $(date)"

This prints a list of files in the current directory.

echo "Files in the directory: $(ls)"

This counts and prints the number of files in the current directory by combining ls and wc -l (which counts lines).

echo "Number of files: $(ls | wc -l)"

More Examples of Combining Expansions

You can also combine different types of expansions to perform more complex tasks. Here are some useful Examples:

Combining Brace Expansion and Tilde Expansion:

mkdir ~/projects/{project1,project2,project3}

This creates three directories (project1, project2, and project3) inside the projects folder in your home directory.

Combining Arithmetic Expansion and Command Substitution:

echo "The result of 5 + 10 is $((5 + 10))"

Outputs The result of 5 + 10 is 15.

Combining Command Substitution with Pathname Expansion:

echo "Today's log file: log_$(date +%Y%m%d).txt"

Creates a dynamic filename for the log based on today’s date, for example, log_20240925.txt.

Conclusion

Linux expansions allow you to write more powerful and efficient commands by automating common tasks like listing files, performing calculations, creating multiple directories or files, and substituting commands within other commands. As you become familiar with expansions like pathname, tilde, arithmetic, brace, parameter expansion, and command substitution, you will find yourself saving time and effort when working on the command line. This flexibility is one of the reasons why Linux is such a powerful system for both casual users and professionals.

Single and Double Quotes

In Linux, quotes play a crucial role in how the shell interprets and processes text. Single quotes (' ') and double quotes (" ") are used to define strings and manage the interpretation of special characters within those strings. Understanding the differences between single and double quotes is essential for effective command-line usage.

Single Quotes

Single quotes (' ') are used to preserve the literal value of each character within the quotes. This means that any special characters, such as $, *, or \, are treated as regular characters and not as special shell characters.

Example:

[me@linuxbox ~]$ echo 'text ~/*.txt {a,b} $(echo foo) $((2+2)) $USER'
text ~/*.txt {a,b} $(echo foo) $((2+2)) $USER

In this example, everything inside the single quotes is treated literally. The shell does not perform any expansion or interpretation of special characters. Thus, the output is exactly what was input: text ~/*.txt {a,b} $(echo foo) $((2+2)) $USER.

Double Quotes

Double quotes (" ") preserve the literal value of most characters within the quotes. However, they allow the interpretation of certain special characters, such as $, \, !, and backticks `.

Example:

[me@linuxbox ~]$ echo "text ~/*.txt {a,b} $(echo foo) $((2+2)) $USER"
text ~/*.txt {a,b} foo 4 me

In this example, the double quotes allow for variable expansion and command substitution. Here's the breakdown of what happens inside the double quotes:

The tilde (~) and brace {} expansions are not processed within double quotes.

No Quotes

When no quotes are used, the shell processes all forms of expansion: pathname expansion, brace expansion, command substitution, arithmetic expansion, and variable expansion.

Example:

[me@linuxbox ~]$ echo text ~/*.txt {a,b} $(echo foo) $((2+2)) $USER
text /home/me/ls-output.txt a b foo 4 me

Here’s what happens step-by-step:

The result is text /home/me/ls-output.txt a b foo 4 me.

Creating Files and Folders with Spaces

When creating files or folders with spaces in their names, quotes are essential to ensure the shell correctly interprets the spaces as part of the name rather than as separators between different arguments.

Examples:

This command creates a directory named My Folder.

mkdir "My Folder"

This command creates a file named New Document.txt.

touch "New Document.txt"

This command renames a file from Old Name.txt to New Name.txt.

mv "Old Name.txt" "New Name.txt"

Summary

Single and double quotes in Linux are powerful tools for controlling how the shell interprets text and special characters. Single quotes preserve the literal value of all characters within the quotes, making them ideal for strings that contain special characters that should not be interpreted by the shell. Double quotes allow for variable expansion, command substitution, and arithmetic expansion, providing flexibility when working with dynamic content. Understanding the differences between single and double quotes, as well as how unquoted text is processed, is crucial for writing effective and accurate shell commands.

Keyboard Tricks

Command line users don't enjoy excessive typing, which is why many commands have short names like cp, ls, mv, and rm. One of the cherished goals of using the command line is to achieve the most with the fewest keystrokes, minimizing the need to use a mouse. This chapter explores bash features that enhance keyboard efficiency.

Command Line Editing

The bash shell uses a library called Readline to implement command line editing, offering features beyond basic arrow key navigation. These tools, while numerous, can be selectively learned to increase productivity. Note that some key sequences, especially those involving the Alt key, may be intercepted by the GUI but will work correctly in a virtual console.

Cursor Movement Commands:

Key Action
Ctrl-a Move cursor to the beginning of the line.
Ctrl-e Move cursor to the end of the line.
Ctrl-f Move cursor forward one character (same as right arrow key).
Ctrl-b Move cursor backward one character (same as left arrow key).
Alt-f Move cursor forward one word.
Alt-b Move cursor backward one word.
Ctrl-l Clear the screen and move the cursor to the top-left corner.

Modifying Text Commands:

Key Action
Ctrl-d Delete the character at the cursor location.
Ctrl-t Transpose the character at the cursor with the preceding one.
Alt-t Transpose the word at the cursor with the preceding word.
Alt-l Convert characters from the cursor to the end of the word to lowercase.
Alt-u Convert characters from the cursor to the end of the word to uppercase.

Cutting and Pasting (Killing and Yanking) Text:

Readline documentation refers to cutting and pasting as killing and yanking. Cut items are stored in a temporary buffer called the kill-ring.

Key Action
Ctrl-k Kill text from the cursor to the end of the line.
Ctrl-u Kill text from the cursor to the beginning of the line.
Alt-d Kill text from the cursor to the end of the current word.
Alt-Backspace Kill text from the cursor to the beginning of the current word (or previous word if at the start of a word).
Ctrl-y Yank text from the kill-ring and insert it at the cursor location.

The Meta Key

The term meta key, often used in Readline documentation, maps to the Alt key on modern keyboards. Historically, before the widespread use of PCs, terminals connected to larger computers used a key designated as meta. On modern systems, pressing the Esc key can substitute for holding the Alt key.

Completion

The shell provides a completion mechanism using the Tab key. Completion simplifies the process of typing commands by automatically filling in pathnames, variables, usernames, commands, and hostnames.

Examples:

For completion to be successful, the input must be unambiguous. For example let's say you have mutiple folders in the same directory that start with dir (dir1, dir2 and dir3). If you enter the line below and press the tab key once you will hear a chime and on the second press it will give you all the folders (or files) that start with dir

First tab press

[me@linuxbox ~]$ ls dir

Second tab press

[me@linuxbox ~]$ ls dir
dir1/ dir2/ dir3

If you have 100 or more folders or files that start with 0 (000 - 099) and you enter 0 and press tab. You will hear a chime and then the following will be displayed

Display all 100 possibilites? (y or n)

This is the behavior on the cidermill server.

Completion commands:

Key Action
Alt-? Display a list of possible completions.
Alt-* Insert all possible completions.

Using History

bash maintains a history of entered commands, stored in the .bash_history file in the home directory. This feature, combined with command line editing, reduces typing effort.

To view the history list:

[me@linuxbox ~]$ history | less

To search the history list for commands:

[me@linuxbox ~]$ history | grep /usr/bin

Using history expansion:

[me@linuxbox ~]$ !88

History Expansion Commands:

Sequence Action
!! Repeat the last command.
!number Repeat history list item number.
!string Repeat last history list item starting with string.
!?string Repeat last history list item containing string.

script Command

The script command in Linux is used to record a terminal session, capturing all the input and output during the session into a file. It is particularly useful for creating logs of interactive shell sessions, debugging, or documenting command-line activities.

Syntax

script [options] [file]

Key Features

  1. Records Terminal Sessions
    Captures everything displayed in the terminal, including user input and command output.

  2. Default Output File
    If no file is specified, the session is saved in a file named typescript.

  3. Interactive Use
    The command runs interactively and exits when the user types exit or presses Ctrl+D.

Common Options

Option Description
-a Append output to the specified file instead of overwriting it.
-c <command> Run a specific command and log its output. The session ends after completion.
-f Flush output to the file as it is written, useful for real-time monitoring.
-q Run in quiet mode, suppressing the start and end messages.
--timing=<file> Record timing information for playback with the scriptreplay command.

Examples

Basic Use

script session.log

This starts recording the session into session.log.

Appending to a File

script -a session.log

Appends the current session to an existing log file.

Running a Command

script -c "ls -l" output.log

Logs the output of ls -l into output.log.

Quiet Mode

script -q log.txt

Runs the session without displaying the starting and ending messages.

Using Timing

script --timing=timing.log session.log

Creates a timing file timing.log to replay the session later.

Replay the Session

The scriptreplay command can be used to replay a session recorded with timing information:

scriptreplay timing.log session.log

Notes

Summing Up

In this chapter, we've explored various keyboard tricks and features in bash that can significantly reduce typing effort. These tools are optional but can be very helpful as you become more familiar with the command line.

Chapter 5

Linux Permissions

Lesson: Understanding Linux Permissions

Linux permissions are a fundamental aspect of the operating system's security model, controlling the access rights of users to files and directories. Each file and directory in Linux is associated with an owner (author), a group, and permission settings for three categories of users: the owner, the group, and everyone else.

Owner (Author), Group, and Everyone

Owner (Author): The owner is typically the user who created the file or directory. By default, the owner has full control over the file or directory, meaning they can read, write, and execute it. The owner can also change the permissions for the group and others.

Group: Each file and directory is associated with a group. Users in this group have specific permissions that can be different from those of the owner. This is useful in collaborative environments where a set of users need to share access to files and directories.

Everyone (Others): This category includes all other users who are not the owner and do not belong to the group associated with the file or directory. Permissions for others are generally more restrictive to maintain security.

Types of Permissions

There are three types of permissions in Linux:

  1. Read (r): Permission to read the contents of the file or directory.
  2. Write (w): Permission to modify the contents of the file or directory.
  3. Execute (x): Permission to execute the file, if it is a script or program. For directories, execute permission allows users to enter the directory and access its contents.

Viewing Permissions

You can view the permissions of files and directories using the ls -l command. The output provides detailed information, including the permission settings.

Example:

$ ls -l
-rwxr-xr-- 1 user group  4096 Jul 10 14:55 file.txt

The output can be broken down as follows:

The permission string -rwxr-xr-- can be interpreted as:

id, chmod and umask

In Linux, managing user identities and permissions is crucial for system security and functionality. Commands like id, chmod, and umask play a significant role in this management. Understanding these commands helps in effectively setting and modifying access controls on files and directories.

The id Command

The id command is used to display the user ID (UID) and group ID (GID) of the current user or a specified user. It provides detailed information about the user, including their primary group and any supplementary groups they belong to.

Example:

$ id
uid=1000(user) gid=1000(user) groups=1000(user),27(sudo),1001(developers)

In this example, uid=1000(user) indicates that the user's ID is 1000. The primary group ID is also 1000, and the user belongs to the sudo and developers groups as well.

You can also specify a username to get the ID details for that particular user:

$ id alice
uid=1001(alice) gid=1001(alice) groups=1001(alice),1002(projects)

The chmod Command

The chmod command is used to change the permissions of files and directories. Permissions determine who can read, write, or execute a file. There are three types of permissions: read (r), write (w), and execute (x). Each permission has an associated numerical value:

These values are added together to set permissions. For instance, a permission setting of rwx (read, write, and execute) adds up to 7 (4+2+1).

Permissions are set for three categories of users: the owner, the group, and others. The chmod command can use either symbolic or numeric modes to change permissions.

Symbolic Mode

In Linux, symbolic mode is used to set file permissions by representing the user (or class of users) and the permissions themselves symbolically, using characters. This is an alternative to the octal (numeric) mode for controlling access to files and directories. Let me break it down step by step:

User Classes

In symbolic mode, permissions are assigned to three classes of users:

Permissions

There are three types of permissions that can be granted or denied to each user class:

Operators

You use the following operators to set permissions:

Examples

Combining Changes

You can make changes to multiple classes at once:

Symbolic mode is helpful because it's human-readable and allows setting permissions selectively for different user classes in a clear way.

Numeric Mode

In numeric mode, you use a three-digit octal number to set permissions. Each digit represents the permissions for the owner, group, and others, respectively.

Example:

chmod 755 file.txt

This sets the permissions to:

Another example:

chmod 644 file.txt

This sets the permissions to:

The umask Command

The umask command sets the default permissions for newly created files and directories. The umask value determines which permission bits will be turned off by default. It is specified as a three-digit octal number.

Example:

umask 022

A umask value of 022 means that new files will be created with permissions 644 (666 - 022) and new directories with permissions 755 (777 - 022). The value 022 masks off the write permission for the group and others.

To view the current umask setting, simply run:

umask

To change the umask value temporarily for the current session, you can use:

umask 027

This sets the default permissions so that new files are created with permissions 640 and directories with permissions 750.

Unusual umask Values

A umask value of 222 is technically valid but results in very restrictive permissions for new files and directories.

Example:

umask 222

With a umask of 222:

Such a setting might be useful in very specific scenarios where files and directories should not be modified by anyone once created.

Examples

  1. Viewing user ID and group information:
id

Output:

uid=1000(user) gid=1000(user) groups=1000(user),27(sudo),1001(developers)
  1. Changing file permissions using symbolic mode:
chmod g+w file.txt

This adds write permission for the group.

  1. Changing file permissions using numeric mode:
chmod 755 script.sh

This sets the permissions to rwxr-xr-x.

  1. Setting the umask value:
umask 027

This sets the default permissions so that new files are created with rw-r----- and directories with rwxr-x---.

  1. Setting an unusual umask value:
umask 222

This results in new files with r--r--r-- and directories with r-xr-xr-x.

Summary

Understanding and effectively using id, chmod, and umask is essential for managing user identities and permissions in Linux. The id command provides information about user and group IDs, chmod allows for precise control over file and directory permissions, and umask sets default permission settings for new files and directories. Mastering these commands ensures better security and proper access control in a Linux environment.

sudo and su Commands

In Linux, su and sudo are powerful commands used to perform administrative tasks and execute commands with elevated privileges. Both commands are essential for system administration and security, enabling users to manage the system effectively while maintaining proper access control.

NOTE: You do not have permissions to use the su or sudo commands on the Cidermill server.

The su Command

The su command stands for "substitute user" or "switch user." It allows a user to switch to another user account in the system, including the superuser (root). When executed without any arguments, su defaults to switching to the root user. This command opens a new shell session as the specified user, maintaining their environment and permissions.

Basic usage of su:

su [username]

Switching to the root user:

su

This command prompts for the root user's password and, upon successful authentication, grants the user root privileges. The shell prompt changes to indicate the new user context, typically # for the root user.

Switching to a different user:

su alice

This command switches to the user alice, prompting for alice's password. The user now operates with alice's permissions and environment.

Starting a login shell with -:

su - alice

This command switches to alice and simulates a full login, loading alice's environment variables and shell configuration files, such as .bashrc and .profile.

The sudo Command

The sudo command stands for "superuser do." It allows a permitted user to execute a command as the superuser or another user, as specified by the security policy in the /etc/sudoers file. Unlike su, sudo does not switch the user context but runs a single command with elevated privileges. This approach enhances security by limiting the scope of administrative actions and reducing the risk of accidental system changes.

Basic usage of sudo:

sudo [command]

Executing a command as root:

sudo apt-get update

This command updates the package lists for upgrades and new packages. The user must authenticate with their own password, not the root password.

Running a command as a different user with -u:

sudo -u alice ls /home/alice

This command lists the contents of alice's home directory while running as alice.

Editing the sudoers file:

sudo visudo

This command opens the sudoers file in a safe editor, allowing administrators to configure permissions and security policies for sudo usage.

Differences Between su and sudo

Examples

Using su to become root:

su
Password:
[root@linuxbox ~]#

The user switches to the root account and gains full administrative privileges.

Using su to switch to another user:

su alice
Password:
[alice@linuxbox ~]$

The user switches to alice, gaining alice's environment and permissions.

Using sudo to run a command as root:

sudo systemctl restart apache2
[sudo] password for user:

The user restarts the Apache web server with root privileges after authenticating with their password.

Using sudo to run a command as another user:

sudo -u alice ls /home/alice

The user lists the contents of alice's home directory, executing the command as alice.

Configuring sudo permissions:

sudo visudo

The administrator edits the sudoers file to grant specific permissions to users or groups, ensuring secure and controlled use of sudo.

Summary

The su and sudo commands are essential tools for managing user permissions and performing administrative tasks in Linux. su allows switching to another user account, opening a new shell session with that user's permissions and environment. sudo provides a more secure and flexible approach by enabling the execution of individual commands with elevated privileges, controlled by the sudoers configuration. Mastering these commands ensures effective and secure system administration.

chown, chgrp and passwd Commands

In Linux, managing file ownership and user passwords is crucial for system security and user management. The chown, chgrp, and passwd commands are essential tools for these tasks. Understanding these commands allows administrators to effectively control file access and maintain secure user accounts.

NOTE: You will not have permissions to do these commands on the Cidermill server.

The chown Command

The chown command stands for "change owner." It is used to change the ownership of files and directories. In Linux, each file and directory is associated with an owner and a group. The chown command allows the superuser (root) or a user with the necessary permissions to change the owner of a file or directory.

Basic usage of chown:

chown [owner] [file]

To change the owner of a file:

sudo chown alice file.txt

This command changes the owner of file.txt to alice. The sudo command is used because changing ownership typically requires superuser privileges.

To change the owner and group of a file:

sudo chown alice:developers project/

This command changes the owner of the project directory to alice and the group to developers.

To recursively change ownership:

sudo chown -R alice:developers /home/alice

This command recursively changes the owner and group of all files and directories within /home/alice to alice and developers.

The chgrp Command

The chgrp command stands for "change group." It is used to change the group ownership of a file or directory. This command is useful when you need to change the group associated with a file or directory without modifying the owner.

Basic usage of chgrp:

chgrp [group] [file]

To change the group of a file:

sudo chgrp developers file.txt

This command changes the group of file.txt to developers.

To recursively change the group of a directory:

sudo chgrp -R developers /project

This command recursively changes the group of all files and directories within /project to developers.

The passwd Command

The passwd command is used to change a user's password. This command can be used by any user to change their own password or by the superuser to change the password of another user. Changing passwords regularly is an essential practice for maintaining system security.

Basic usage of passwd:

passwd [username]

To change your own password:

passwd

This command prompts the user to enter their current password, followed by the new password. The password must meet the system's complexity requirements.

To change another user's password:

sudo passwd alice

This command allows the superuser to change the password for the user alice. The superuser is prompted to enter the new password for alice.

To lock a user's password:

sudo passwd -l alice

This command locks alice's password, preventing the user from logging in.

To unlock a user's password:

sudo passwd -u alice

This command unlocks alice's password, allowing the user to log in again.

Examples

To change the owner of a file:

sudo chown bob report.txt

This changes the owner of report.txt to bob.

To change the group of a directory:

sudo chgrp managers /reports

This changes the group of the /reports directory to managers.

To change your own password:

passwd

The user is prompted to enter their current password and then the new password.

To change another user's password:

sudo passwd john

The superuser is prompted to enter a new password for the user john.

To recursively change the owner and group:

sudo chown -R user1:team1 /home/user1

This changes the owner to user1 and the group to team1 for all files and directories within /home/user1.

Summary

The chown, chgrp, and passwd commands are essential for managing file ownership and user passwords in Linux. chown changes the owner (and optionally the group) of files and directories, chgrp changes the group ownership, and passwd changes user passwords. These commands help maintain proper access controls and ensure the security of the system. Understanding how to use these commands effectively is a crucial skill for Linux administrators.

Chapter 6

ps and top Commands

Lesson: Understanding the ps and top Commands in Linux

In Linux, monitoring system processes is crucial for managing system performance and troubleshooting issues. The ps and top commands are essential tools for this task, providing detailed information about running processes. Understanding these commands and their options allows administrators to efficiently manage system resources.

The ps Command

The ps (process status) command is used to display information about active processes. It provides a snapshot of the current processes, allowing users to see details such as process IDs (PIDs), CPU and memory usage, and the commands that started the processes.

Basic usage of ps:

ps

This command displays information about the processes running in the current shell session.

Common options for ps:

Option Description
-e Show information about all processes.
-f Display full format listing (detailed).
-u Display processes for a specific user.
-l Display long format listing.
-aux Show all processes with detailed information.
--sort Sort processes based on a specified field.

Examples of ps usage:

To display all processes:

ps -e

Output:

  PID TTY          TIME CMD
    1 ?        00:00:02 systemd
    2 ?        00:00:00 kthreadd
    3 ?        00:00:00 rcu_gp
    ...

To display detailed information about all processes:

ps -ef

Output:

UID          PID    PPID  C STIME TTY          TIME CMD
root           1       0  0 14:19 ?        00:00:02 /sbin/init
root           2       0  0 14:19 ?        00:00:00 [kthreadd]
root           3       2  0 14:19 ?        00:00:00 [rcu_gp]
...

To display processes for a specific user:

ps -u alice

Output:

  PID TTY          TIME CMD
 1234 ?        00:00:00 bash
 2345 ?        00:00:01 firefox
 ...

To display processes in long format:

ps -l

Output:

F S UID        PID  PPID  C PRI  NI ADDR SZ WCHAN  TTY          TIME CMD
4 S root         1     0  0  80   0 -  5466 -      ?        00:00:02 systemd
1 S root         2     0  0  80   0 -     0 -      ?        00:00:00 kthreadd
1 S root         3     2  0  80   0 -     0 -      ?        00:00:00 rcu_gp
...

To show all processes with detailed information:

ps aux

Output:

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.1 169936  1232 ?        Ss   14:19   0:02 /sbin/init
root         2  0.0  0.0      0     0 ?        S    14:19   0:00 [kthreadd]
root         3  0.0  0.0      0     0 ?        I<   14:19   0:00 [rcu_gp]
...

To sort processes by CPU usage:

ps -eo pid,ppid,cmd,%mem,%cpu --sort=-%cpu

Output:

  PID  PPID CMD                         %MEM %CPU
 2345     1 /usr/bin/firefox             2.3 10.5
 1234     1 /usr/bin/bash                0.1  1.0
 ...

Columns in ps Command Output

Column Description
PID Process ID, a unique identifier for each process.
TTY Terminal type associated with the process.
TIME Cumulative CPU time used by the process.
CMD Command that initiated the process.
UID User ID of the process owner.
PPID Parent process ID.
%CPU Percentage of CPU time used by the process.
%MEM Percentage of physical memory used by the process.
C CPU utilization of the process.
STIME Start time of the process.
VSZ Virtual memory size of the process.
RSS Resident Set Size, the non-swappable physical memory used by the process.
STAT Process state code.

Here is a list of the common Process state codes ps:

Code Description
R Running or runnable (on run queue)
S Sleeping (waiting for an event to complete)
D Uninterruptible sleep (usually waiting for I/O)
Z Zombie (terminated but not reaped by parent)
T Stopped, either by a job control signal or because it is being traced
t Stopped by debugger during the tracing
X Dead (should never be seen)
K Wakekill (woken up by a fatal signal while in an uninterruptible sleep)
W Paging (not valid since Linux 2.6.x)
P Parked (used by real-time threads that are parked)
I Idle kernel thread
L Locked (process is waiting for CPU lock or I/O operation to finish)
< High-priority process (has a real-time priority or is in high-priority scheduling class)
N Low-priority process (has a lower priority than normal)
s Session leader (a process that is a session leader)
l Multi-threaded (using CLONE_THREAD, so that it shares the thread group)
+ Process is in the foreground process group

This format makes the process state codes stand out for easier reference.

The top Command

The top command provides a real-time, dynamic view of the system's running processes. It continuously updates the display, showing which processes are using the most system resources. top is highly configurable and allows users to sort and filter processes based on various criteria.

Basic usage of top:

top

This command opens the top interface, displaying information about the most resource-intensive processes.

Common options for top:

Option Description
-d Set the delay between updates (in seconds).
-u Show processes for a specific user.
-p Monitor specific PIDs.
-n Number of iterations before top exits.
-b Run top in batch mode (useful for logging).
-o Sort processes based on a specified field.

Examples of top usage:

To run top with the default settings:

top

Output:

top - 14:29:03 up  1:10,  2 users,  load average: 0.48, 0.34, 0.23
Tasks: 193 total,   1 running, 192 sleeping,   0 stopped,   0 zombie
%Cpu(s):  1.0 us,  0.5 sy,  0.0 ni, 98.0 id,  0.0 wa,  0.0 hi,  0.5 si,  0.0 st
KiB Mem :  8000224 total,  3145752 free,  2392308 used,  2462164 buff/cache
KiB Swap:  8191996 total,  8191996 free,        0 used.  5062348 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 2345 alice     20   0 2674292 473240 100768 S  10.5  5.9   1:23.45 firefox
 1234 bob       20   0   43456   3824   3236 S   1.0  0.1   0:00.75 bash
...

To set the delay between updates to 5 seconds:

top -d 5

To show processes for a specific user:

top -u alice

Output:

top - 14:31:12 up  1:12,  2 users,  load average: 0.42, 0.32, 0.22
Tasks:  13 total,   1 running,  12 sleeping,   0 stopped,   0 zombie
%Cpu(s):  1.2 us,  0.3 sy,  0.0 ni, 98.0 id,  0.0 wa,  0.0 hi,  0.5 si,  0.0 st
KiB Mem :  8000224 total,  3145752 free,  2392308 used,  2462164 buff/cache
KiB Swap:  8191996 total,  8191996 free,        0 used.  5062348 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 2345 alice     20   0 2674292 473240 100768 S  10.5  5.9   1:23.45 firefox
...

To monitor specific PIDs:

top -p 1234,5678

Output:

top - 14:32:45 up  1:14,  2 users,  load average: 0.35, 0.31, 0.21
Tasks:   2 total

,   1 running,   1 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.7 us,  0.3 sy,  0.0 ni, 99.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem :  8000224 total,  3145752 free,  2392308 used,  2462164 buff/cache
KiB Swap:  8191996 total,  8191996 free,        0 used.  5062348 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 1234 bob       20   0   43456   3824   3236 S   1.0  0.1   0:00.75 bash
 5678 charlie   20   0  123456  5678    678 S   0.0  0.1   0:00.05 someprocess

To run top for a specific number of iterations:

top -n 10

To run top in batch mode for logging:

top -b -n 1 > top_output.txt

To sort processes by memory usage:

top -o %MEM

Output:

top - 14:34:19 up  1:16,  2 users,  load average: 0.30, 0.28, 0.20
Tasks: 193 total,   1 running, 192 sleeping,   0 stopped,   0 zombie
%Cpu(s):  1.0 us,  0.5 sy,  0.0 ni, 98.0 id,  0.0 wa,  0.0 hi,  0.5 si,  0.0 st
KiB Mem :  8000224 total,  3145752 free,  2392308 used,  2462164 buff/cache
KiB Swap:  8191996 total,  8191996 free,        0 used.  5062348 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 2345 alice     20   0 2674292 473240 100768 S  10.5  5.9   1:23.45 firefox
 6789 bob       20   0 1234567 234567  34567 S   1.0  3.0   0:12.34 someotherprocess
...

Columns in top Command Output

Column Description
PID Process ID, a unique identifier for each process.
USER User who owns the process.
PR Priority of the process.
NI Nice value, which affects the process priority.
VIRT Virtual memory size used by the process.
RES Resident Set Size, the non-swappable physical memory used by the process.
SHR Shared memory size used by the process.
S Process state (e.g., running, sleeping).
%CPU Percentage of CPU time used by the process.
%MEM Percentage of physical memory used by the process.
TIME+ Total CPU time used by the process.
COMMAND Command that started the process.

Explanation of Fields

Header Information

Field Description
top - 14:34:19 Current time when top was run.
up 1:16 System uptime, showing how long the system has been running.
2 users Number of users currently logged into the system.
load average: 0.30, 0.28, 0.20 System load averages for the last 1, 5, and 15 minutes.

Task Information

Field Description
Tasks: 193 total Total number of tasks (processes) currently managed by the system.
1 running Number of tasks currently running.
192 sleeping Number of tasks currently sleeping (inactive but not stopped).
0 stopped Number of tasks currently stopped (halted).
0 zombie Number of zombie tasks (terminated but not reaped by parent).

CPU Usage Information

Field Description
%Cpu(s): Overview of CPU usage breakdown.
1.0 us Percentage of CPU time spent in user space.
0.5 sy Percentage of CPU time spent in kernel space (system).
0.0 ni Percentage of CPU time spent on low-priority processes.
98.0 id Percentage of CPU time spent idle.
0.0 wa Percentage of CPU time spent waiting for I/O operations to complete.
0.0 hi Percentage of CPU time spent handling hardware interrupts.
0.5 si Percentage of CPU time spent handling software interrupts.
0.0 st Percentage of CPU time stolen from a virtual machine.

Memory Usage Information

Field Description
KiB Mem : Overview of physical memory usage.
8000224 total Total amount of physical memory.
3145752 free Amount of free memory.
2392308 used Amount of used memory.
2462164 buff/cache Amount of memory used for buffers and cache.

Swap Usage Information

Field Description
KiB Swap: Overview of swap memory usage.
8191996 total Total amount of swap space.
8191996 free Amount of free swap space.
0 used Amount of used swap space.
5062348 avail Mem Amount of memory available for starting new applications, without swapping.

Summary

The ps and top commands are powerful tools for monitoring and managing system processes in Linux. ps provides a static snapshot of current processes, offering various options for detailed views and sorting. top offers a dynamic, real-time view, allowing users to interactively monitor and manage processes. By mastering these commands and their options, administrators can effectively manage system resources and ensure optimal performance.

jobs, fg and bg Commands

In Linux, managing background and foreground processes is essential for multitasking and efficient workflow. The jobs, fg, and bg commands allow users to control and manipulate these processes. Additionally, keyboard shortcuts like Ctrl-C and Ctrl-Z provide quick ways to interact with running processes. Understanding these commands and shortcuts can greatly enhance your command-line productivity.

Creating and Running a Background Script

Consider the following script that repeatedly outputs a message every 15 seconds:

echo 'echo "I am running"; while sleep 15; do echo "I am running"; done' > myscript.sh
chmod +x myscript.sh

This script prints "I am running" immediately and then every 15 seconds. The chmod +x myscript.sh command makes the script executable. NOTE: You just do the chmod +x myscript.sh or chmod 700 myscript.sh

To run this script in the background, use the following command:

./myscript.sh &

This command runs the myscript.sh script in the background, allowing you to continue using the terminal. Alternatively, you can run the script using:

bash myscript.sh &

The jobs Command

The jobs command lists all background jobs started in the current shell session. Each job is assigned a job ID, which can be used to manipulate the job with other commands.

Example:

jobs

Output:

[1]+  Running ./myscript.sh &

In this example, job 1 is running in the background.

The fg Command

The fg (foreground) command is used to bring a background job to the foreground. You can specify the job ID to bring a specific job to the foreground.

Example:

fg %1

This command brings job 1 (the myscript.sh script) to the foreground.

If you omit the job ID, fg brings the most recently stopped or background job to the foreground:

fg

The bg Command

The bg (background) command resumes a stopped job in the background. Like fg, you can specify the job ID to resume a specific job.

Example:

bg %1

This command resumes job 1 (the myscript.sh script) in the background.

If you omit the job ID, bg resumes the most recently stopped job:

bg

Keyboard Shortcuts: Ctrl-C and Ctrl-Z

Ctrl-C: The Ctrl-C keyboard shortcut sends the SIGINT (interrupt) signal to the foreground process, terminating it immediately. This is useful for stopping processes that are taking too long or behaving unexpectedly.

Example:

./myscript.sh

Press Ctrl-C to stop the myscript.sh script.

Ctrl-Z: The Ctrl-Z keyboard shortcut sends the SIGTSTP (terminal stop) signal to the foreground process, suspending it and putting it in the background as a stopped job. This allows you to temporarily halt a process and resume it later with fg or bg.

Example:

./myscript.sh

Press Ctrl-Z to suspend the myscript.sh script.

Examples Using the Script

To run the script in the background:

./myscript.sh &

This runs the myscript.sh script in the background.

To list all background jobs:

jobs

Output:

[1]+  Running                 ./myscript.sh &

To bring the background job to the foreground:

fg %1

This brings job 1 to the foreground.

To suspend the foreground script:

./myscript.sh

Press Ctrl-Z to suspend the myscript.sh script.

To resume the stopped job in the background:

bg %1

This resumes job 1 in the background.

To terminate the foreground script:

./myscript.sh

Press Ctrl-C to stop the myscript.sh script.

Summary

The jobs, fg, and bg commands, along with keyboard shortcuts Ctrl-C and Ctrl-Z, provide powerful tools for managing processes in Linux. These commands allow you to run, suspend, resume, and terminate processes, enhancing your ability to multitask and control your workflow efficiently. By mastering these commands and shortcuts, you can become more productive and effective in the Linux command-line environment.

kill, killall and shutdown Commands

In Linux, controlling processes and managing system shutdowns are crucial for maintaining system stability and performance. The kill, killall, and shutdown commands are essential tools for these tasks. Understanding these commands and their options allows administrators to effectively manage processes and system power states.

The kill Command

The kill command is used to send signals to processes, typically to terminate them. Each process in Linux has a unique process ID (PID), and the kill command uses this PID to target specific processes. While kill can send various signals, it is most commonly used to terminate processes.

Basic usage of kill:

kill [options] <PID>

Commonly Used kill Signals:

Signal Description Example Command
1 SIGHUP - Hangup kill -1 <PID>
2 SIGINT - Interrupt from keyboard kill -2 <PID>
9 SIGKILL - Kill signal kill -9 <PID>
15 SIGTERM - Termination signal kill -15 <PID>
18 SIGCONT - Continue if stopped kill -18 <PID>
19 SIGSTOP - Stop process kill -19 <PID>

Examples:

To gracefully terminate a process with PID 1234:

kill -15 1234

To forcefully terminate a process with PID 1234:

kill -9 1234

To stop a process with PID 1234 without terminating it:

kill -19 1234

To continue a stopped process with PID 1234:

kill -18 1234

The killall Command

The killall command is used to send signals to multiple processes by name rather than by PID. This command is useful when you need to terminate all instances of a specific process.

Basic usage of killall:

killall [options] <process_name>

Examples:

To gracefully terminate all instances of the nano editor:

killall -15 nano

To forcefully terminate all instances of the firefox browser:

killall -9 firefox

The shutdown Command

The shutdown command is used to bring the system down in a safe way. It allows administrators to power off, reboot, or halt the system. The shutdown command can schedule a shutdown for a specified time or immediately.

NOTE: You will not have permissions to shutdown the Cidermill server.

Basic usage of shutdown:

shutdown [options] [time] [message]

Common shutdown Options:

Option Description Example Command
-h Halt the system shutdown -h now
-r Reboot the system shutdown -r now
-c Cancel a scheduled shutdown shutdown -c
+m Schedule shutdown in m minutes shutdown -h +10 "System update"

Examples:

To shut down the system immediately:

shutdown -h now

To reboot the system immediately:

shutdown -r now

To schedule a shutdown in 15 minutes with a message:

shutdown -h +15 "System will shut down in 15 minutes"

To cancel a scheduled shutdown:

shutdown -c

You can also reboot the systerm using reboot

reboot

Summary

The kill, killall, and shutdown commands are essential tools for managing processes and system power states in Linux. The kill command allows for sending various signals to processes using their PIDs, while killall targets processes by name. The shutdown command provides options for halting, rebooting, or scheduling system shutdowns. By mastering these commands and their options, administrators can effectively control processes and manage system shutdowns, ensuring system stability and performance.

Understanding the Environment in Linux

The shell maintains a body of information during a session called the environment. Programs use data stored in the environment to determine facts about the system's configuration. While most programs use configuration files to store settings, some also look for values in the environment to adjust their behavior. Knowing this, we can customize our shell experience using the environment.

In this lesson, we will work with the following commands:

What is Stored in the Environment?

The shell stores two basic types of data in the environment: environment variables and shell variables. Shell variables are bits of data placed there by bash, while environment variables are everything else. In addition to variables, the shell stores some programmatic data, namely aliases and shell functions.

Examining The Environment

To see what is stored in the environment, you can use either the set builtin in bash or the printenv program. The set command will show both shell and environment variables, while printenv will only display the latter. Since the list of environment contents can be long, it is best to pipe the output into less.

Example:

printenv | less

Output:

USER=me
PAGER=less
LSCOLORS=Gxfxcxdxbxegedabagacad
XDG_CONFIG_DIRS=/etc/xdg/xdg-ubuntu:/usr/share/upstart/xdg:/etc/xdg
PATH=/home/me/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
DESKTOP_SESSION=ubuntu
QT_IM_MODULE=ibus
QT_QPA_PLATFORMTHEME=appmenu-qt5
JOB=dbus
PWD=/home/me
XMODIFIERS=@im=ibus
GNOME_KEYRING_PID=1850
LANG=en_US.UTF-8
GDM_LANG=en_US
MANDATORY_PATH=/usr/share/gconf/ubuntu.mandatory.path
MASTER_HOST=linuxbox
IM_CONFIG_PHASE=1
COMPIZ_CONFIG_PROFILE=ubuntu
GDMSESSION=ubuntu
SESSIONTYPE=gnome-session
XDG_SEAT=seat0
HOME=/home/me
SHLVL=2
LANGUAGE=en_US
GNOME_DESKTOP_SESSION_ID=this-is-deprecated
LESS=-R
LOGNAME=me
COMPIZ_BIN_PATH=/usr/bin/
LC_CTYPE=en_US.UTF-8
XDG_DATA_DIRS=/usr/share/ubuntu:/usr/share/gnome:/usr/local/share:/usr/share
QT4_IM_MODULE=xim
DBUS_SESSION_BUS_ADDRESS=unix:abstract=/tmp/dbus-IwaesmWaT0
LESSOPEN=| /usr/bin/lesspipe %s
INSTANCE=

The output lists environment variables and their values. For example, the variable USER contains the value me. The printenv command can also list the value of a specific variable.

Example:

printenv USER

Output:

me

The set command, when used without options or arguments, will display both shell and environment variables, as well as any defined shell functions. Unlike printenv, its output is sorted in alphabetical order.

Example:

set | less

Commonly Used set Options:

Option Description
-o Set a shell option
+o Unset a shell option
-u Treat unset variables as an error
-x Print commands and their arguments as they are executed
-v Print shell input lines as they are read

Examples:

To set the shell option to treat unset variables as an error:

set -u

To print commands and their arguments as they are executed:

set -x

To unset these options:

set +u
set +x

To view shell options:

set -o

You can also view the contents of a variable using the echo command:

echo $HOME

Output:

/home/me

To see aliases, enter the alias command without arguments:

alias

Output:

alias l.='ls -d .* --color=tty'
alias ll='ls -l --color=tty'
alias ls='ls --color=tty'
alias vi='vim'
alias which='alias | /usr/bin/which --tty-only --read-alias --showdot --show-tilde'

To create an alias, use the alias command followed by the name you want to assign and the command it represents:

alias l='ls -l'
alias rm='rm -i'

Some Interesting Variables

The environment contains quite a few variables, and while they may differ from system to system, the following are commonly found:

Variable Contents
DISPLAY The name of the display if running a graphical environment, usually :0.
EDITOR The name of the program to be used for text editing.
SHELL The name of the user’s default shell program.
HOME The pathname of your home directory.
LANG Defines the character set and collation order of your language.
OLDPWD The previous working directory.
PAGER The name of the program to be used for paging output, often /usr/bin/less.
PATH A colon-separated list of directories that are searched when you enter the name of an executable program.
PS1 This stands for "prompt string 1" and defines the contents of the shell prompt.
PWD The current working directory.
TERM The name of your terminal type.
TZ Specifies your time zone.
USER Your username.

How Is The Environment Established?

When you log on to the system, bash starts and reads a series of configuration scripts called startup files, which define the default environment shared by all users. This is followed by more startup files in your home directory that define your personal environment. The exact sequence depends on the type of shell session being started: login or non-login.

Login Shell Session: A session where you are prompted for your username and password, such as when starting a virtual console session.

Non-Login Shell Session: Typically occurs when launching a terminal session in the GUI.

Startup Files for Login Shell Sessions:

File Contents
/etc/profile A global configuration script that applies to all users.
~/.bash_profile A user's personal startup file to extend or override settings in the global configuration script.
~/.bash_login If ~/.bash_profile is not found, bash attempts to read this script.
~/.profile If neither ~/.bash_profile nor ~/.bash_login is found, bash attempts to read this file. This is the default in Debian-based distributions.

Startup Files for Non-Login Shell Sessions:

File Contents
/etc/bash.bashrc A global configuration script that applies to all users.
~/.bashrc A user's personal startup file to extend or override settings in the global configuration script.

Non-login shells inherit the environment from their parent process, usually a login shell.

The ~/.bashrc file is important because it is almost always read. Non-login shells read it by default, and most startup files for login shells are written to read the ~/.bashrc file as well.

What's in a Startup File?

A typical .bash_profile might look like this:

# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi

# User specific environment and startup programs
PATH=$PATH:$HOME/bin

export PATH

Lines starting with # are comments. The fourth line contains an if compound command:

if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi

This means: If the file ~/.bashrc exists, then read it. This is how a login shell gets the contents of .bashrc.

The next part deals with the PATH variable:

PATH=$PATH:$HOME/bin

This modifies PATH to add $HOME/bin to the end of the list of directories searched when a command is entered. This allows you to create a bin directory in your home directory for storing private programs.

Finally:

export PATH

The export command makes the contents of PATH available to child processes of the shell.

Modifying the Environment

You can customize your environment by modifying the startup files.

Which Files Should You Modify?

Text Editors: To modify shell startup files and configuration files, use a text editor. Linux systems typically come with several text

editors, both graphical (like gedit for GNOME or kate for KDE) and text-based (like nano, vi/vim, and emacs).

For this class we will be using vi/vim.

Chapter 7

ping and tracroute Commands

The ping and traceroute commands are essential network diagnostic tools in Linux. These commands help users test connectivity and trace the path packets take to reach a destination, respectively. Understanding how to use these commands and interpret their outputs is crucial for troubleshooting network issues.

The ping Command

The ping command checks the connectivity between your computer and another host. It sends Internet Control Message Protocol (ICMP) Echo Request packets to the target host and waits for Echo Reply packets. This helps determine if the target host is reachable and measures the round-trip time for messages sent.

Basic usage of ping:

ping [options] destination

Commonly Used ping Options:

Option Description
-c Specify the number of packets to send
-i Specify the interval between sending each packet
-t Set the Time to Live (TTL) for packets
-q Quiet output, showing summary only at the end
-s Specify the number of data bytes to be sent
-w Specify a timeout, in seconds, before the command exits

Examples:

To ping a host:

ping google.com

Output:

PING google.com (172.217.16.206) 56(84) bytes of data.
64 bytes from dfw25s14-in-f14.1e100.net (172.217.16.206): icmp_seq=1 ttl=53 time=11.6 ms
64 bytes from dfw25s14-in-f14.1e100.net (172.217.16.206): icmp_seq=2 ttl=53 time=10.8 ms
64 bytes from dfw25s14-in-f14.1e100.net (172.217.16.206): icmp_seq=3 ttl=53 time=10.9 ms
--- google.com ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 10.805/11.113/11.590/0.347 ms

To ping a host with a specific number of packets:

ping -c 4 google.com

Output:

PING google.com (172.217.16.206) 56(84) bytes of data.
64 bytes from dfw25s14-in-f14.1e100.net (172.217.16.206): icmp_seq=1 ttl=53 time=11.6 ms
64 bytes from dfw25s14-in-f14.1e100.net (172.217.16.206): icmp_seq=2 ttl=53 time=10.8 ms
64 bytes from dfw25s14-in-f14.1e100.net (172.217.16.206): icmp_seq=3 ttl=53 time=10.9 ms
64 bytes from dfw25s14-in-f14.1e100.net (172.217.16.206): icmp_seq=4 ttl=53 time=10.7 ms
--- google.com ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3002ms
rtt min/avg/max/mdev = 10.708/11.005/11.586/0.321 ms

The traceroute Command

The traceroute command shows the path that packets take to reach a destination. It sends packets with incrementally increasing Time to Live (TTL) values, causing routers along the path to return ICMP Time Exceeded messages. This helps identify each hop along the route to the destination.

Basic usage of traceroute:

traceroute [options] destination

Commonly Used traceroute Options:

Option Description
-m Set the maximum TTL for packets
-p Set the destination port
-q Set the number of queries per hop
-w Set the time to wait for a response, in seconds
-n Print hop addresses numerically rather than resolving hostnames

Examples:

To trace the route to a host:

traceroute google.com

Output:

traceroute to google.com (172.217.16.206), 30 hops max, 60 byte packets
 1  192.168.1.1 (192.168.1.1)  2.232 ms  2.089 ms  2.055 ms
 2  * * *
 3  10.0.0.1 (10.0.0.1)  11.133 ms  11.052 ms  10.939 ms
 4  172.217.16.206 (172.217.16.206)  10.714 ms  10.544 ms  10.362 ms

The *** in the output indicates that the router at that hop did not respond to the traceroute request within the expected time frame. This can happen due to network congestion, firewalls, or routers configured not to send ICMP Time Exceeded messages.

To trace the route with a maximum of 10 hops:

traceroute -m 10 google.com

Output:

traceroute to google.com (172.217.16.206), 10 hops max, 60 byte packets
 1  192.168.1.1 (192.168.1.1)  2.232 ms  2.089 ms  2.055 ms
 2  * * *
 3  10.0.0.1 (10.0.0.1)  11.133 ms  11.052 ms  10.939 ms
 4  172.217.16.206 (172.217.16.206)  10.714 ms  10.544 ms  10.362 ms

Summary

The ping and traceroute commands are indispensable for network troubleshooting in Linux. The ping command checks connectivity and measures round-trip time between your computer and a host, while the traceroute command traces the path packets take to reach a destination, providing insights into the route and potential points of failure. The *** in the traceroute output indicates non-responsive routers, which can be due to various reasons such as network congestion or firewall settings. By mastering these commands and understanding their outputs, you can effectively diagnose and resolve network issues.

ip and netstat Commands

The ip and netstat commands are essential tools for managing and monitoring network configurations and connections in Linux. The ip command is used for network interface configuration, IP address manipulation, and routing, while netstat is used to display network connections, routing tables, interface statistics, masquerade connections, and multicast memberships. Understanding these commands, their outputs, and their options is crucial for effective network management.

The ip Command

The ip command is part of the iproute2 package and is used to configure network interfaces, IP addresses, and routing.

Basic usage of ip:

ip [OPTIONS] OBJECT { COMMAND | help }

Commonly Used ip Options:

Option Description
addr Display or manipulate IP addresses
link Display or manipulate network interfaces
route Display or manipulate routing tables
neigh Display or manipulate ARP cache
-s Display detailed statistics

Examples and Output Explanations:

To display all IP addresses:

ip addr

Output:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 08:00:27:53:8b:dc brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.10/24 brd 192.168.1.255 scope global dynamic enp0s3
       valid_lft 86352sec preferred_lft 86352sec
    inet6 fe80::a00:27ff:fe53:8bdc/64 scope link 
       valid_lft forever preferred_lft forever

To display the link layer information:

ip link

Output:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 08:00:27:53:8b:dc brd ff:ff:ff:ff:ff:ff

To display the routing table:

ip route

Output:

default via 192.168.1.1 dev enp0s3 proto dhcp metric 100 
192.168.1.0/24 dev enp0s3 proto kernel scope link src 192.168.1.10 metric 100

The netstat Command

The netstat command displays various network-related information such as network connections, routing tables, interface statistics, masquerade connections, and multicast memberships.

Basic usage of netstat:

netstat [OPTIONS]

Commonly Used netstat Options:

Option Description
-a Show all sockets (listening and non-listening)
-t Show TCP connections
-u Show UDP connections
-n Show numerical addresses instead of resolving hostnames
-r Display the routing table
-i Display network interface statistics
-s Display summary statistics for each protocol
-p Show process using the socket

Examples and Output Explanations:

To display all network connections:

netstat -a

Output:

Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State      
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN     
tcp        0      0 192.168.1.10:22         192.168.1.100:54678     ESTABLISHED
udp        0      0 0.0.0.0:68              0.0.0.0:*                          
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags       Type       State         I-Node   Path
unix  2      [ ]         DGRAM                    13383    /run/systemd/notify

To display the routing table:

netstat -r

Output:

Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
default         192.168.1.1     0.0.0.0         UG        0 0          0 enp0s3
192.168.1.0     *               255.255.255.0   U         0 0          0 enp0s3

To display network interface statistics:

netstat -i

Output:

Kernel Interface table
Iface      MTU    RX-OK  RX-ERR RX-DRP RX-OVR TX-OK  TX-ERR TX-DRP TX-OVR Flg
enp0s3     1500   4823   0      0      0      4325   0      0      0      BMRU
lo         65536  1522   0      0      0      1522   0      0      0      LRU

To display summary statistics for each protocol:

netstat -s

Output:

Ip:
    4325 total packets received
    0 forwarded
    0 incoming packets discarded
    4325 incoming packets delivered
    4325 requests sent out
Tcp:
    240 active connection openings
    6 passive connection openings
    0 failed connection attempts
    0 connection resets received
    2 connections established
    2103 segments received
    2200 segments sent out
    0 segments retransmitted
    0 bad segments received.
    0 resets sent
Udp:
    200 packets received
    0 packets to unknown port received.
    0 packet receive errors
    200 packets sent

Summary

The ip and netstat commands are powerful tools for network management and monitoring in Linux. Theipcommand allows for detailed configuration and display of network interfaces, IP addresses, and routing tables. Thenetstat` command provides extensive information on network connections, routing tables, and interface statistics. By mastering these commands and understanding their outputs, you can effectively manage and troubleshoot network issues.

wget, ftp and sftp Commands

The wget, ftp, and sftp commands are essential tools for transferring files between computers in Linux. wget is a non-interactive network downloader, while ftp and sftp are used for interactive file transfers, with sftp being the preferred and more secure option.

The wget Command

The wget command is used for downloading files from the web. It supports HTTP, HTTPS, and FTP protocols and can download files non-interactively, making it ideal for automated downloads.

Basic usage of wget:

wget [options] [URL]

Commonly Used wget Options:

Option Description
-O Write output to a file instead of standard output
-c Resume a partially downloaded file
-q Quiet mode (no output)
-r Recursive download
-l Set the maximum depth for recursive downloads
--limit-rate Limit the download speed

To download a file using wget:

wget http://example.com/file.txt

Output:

--2024-07-10 10:00:00--  http://example.com/file.txt
Resolving example.com (example.com)... 93.184.216.34
Connecting to example.com (example.com)|93.184.216.34|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1234 (1.2K) [text/plain]
Saving to: 'file.txt'

file.txt          100%[===================>]   1.21K  --.-KB/s    in 0s      

2024-07-10 10:00:01 (234 MB/s) - ‘file.txt’ saved [1234/1234]

The ftp Command

The ftp (File Transfer Protocol) command is used to transfer files between a client and a server. It is an older protocol that is less secure compared to sftp. ftp transfers data in plaintext, making it vulnerable to interception and attacks. Due to its security limitations, sftp is now the preferred method for file transfers.

The sftp Command

The sftp (Secure File Transfer Protocol) command is a secure version of ftp that uses SSH (Secure Shell) to encrypt the data being transferred. It is preferred over ftp due to its enhanced security features.

Basic usage of sftp:

sftp [user@]hostname

Commonly Used sftp Options:

Option Description
-b Batch mode reads a series of commands from a file
-C Enable compression
-P Specify an alternate port
-i Specify an identity file (private key)
-q Quiet mode
-v Verbose mode

Examples and Output Explanations

To connect to an SFTP server:

sftp user@ftp.example.com

Output:

Connecting to ftp.example.com...
user@ftp.example.com's password: 
sftp>

To list files on the remote server:

ls

Output:

files
file.txt

To list files on the local machine:

lls

Output:

Desktop  Documents  Downloads  file.txt  Pictures

To change directories on the remote server:

cd /path/to/remote/directory

Output:

sftp> cd /path/to/remote/directory

To change directories on the local machine:

lcd /path/to/local/directory

Output:

sftp> lcd /path/to/local/directory

To download a file from the remote server:

get file.txt

Output:

Fetching /home/user/file.txt to file.txt
/home/user/file.txt                             100% 1234     1.2KB/s   00:00

To upload a file to the remote server:

put localfile.txt remotefile.txt

Output:

Uploading localfile.txt to /home/user/remotefile.txt
localfile.txt                                   100% 1234     1.2KB/s   00:00

To download multiple files:

mget *.txt

To upload multiple files:

mput *.txt

Differences Between ftp and sftp

The key difference between ftp and sftp is security. ftp transfers data in plaintext, which makes it vulnerable to interception and attacks. In contrast, sftp uses SSH to encrypt the data being transferred, providing a secure channel that protects the data from being intercepted or tampered with.

Due to its security advantages, sftp is now preferred over ftp for file transfers.

Summary

The wget, ftp, and sftp commands are powerful tools for downloading and transferring files in Linux. wget is used for non-interactive downloads, while ftp and sftp are used for interactive file transfers, with sftp being the preferred option due to its enhanced security features. By mastering these commands and understanding their options, you can efficiently manage file transfers in the Linux environment.

Linux Archiving and Zipping

The gzip, zip, and tar commands are essential tools in Linux for compressing and archiving files. Each tool has unique features and is used for different purposes.

The gzip Command

gzip is a compression tool used to reduce the size of files. It does not create archives (collections of multiple files); instead, it compresses single files.

Common gzip Options:

Option Description
-d Decompress
-k Keep the original file
-l List compression information
-r Recursively compress directories
-v Verbose output
-1 to -9 Set compression level (1 = fastest, 9 = slowest/maximum)

Examples:

To compress a file:

gzip file.txt

To decompress a file:

gzip -d file.txt.gz

To keep the original file while compressing:

gzip -k file.txt

To recursively compress all files in a directory:

gzip -r directory/

The zip Command

zip is a tool used for both compressing and archiving files. It can create zip archives containing multiple files and directories.

Common zip Options:

Option Description
-r Recursively add directories
-d Delete files from the archive
-u Update files in the archive
-l List files in the archive
-v Verbose output
-e Encrypt the archive

Examples:

To create a zip archive:

zip archive.zip file1.txt file2.txt

To unzip an archive:

unzip archive.zip

To add files to an existing archive:

zip archive.zip file3.txt

To recursively add directories to an archive:

zip -r archive.zip directory/

The tar Command

tar is used to create archive files that can contain multiple files and directories. While tar itself does not compress files, it is often used in combination with gzip or bzip2 to create compressed archives (.tar.gz or .tar.bz2).

Common tar Options:

Option Description
-c Create a new archive
-x Extract files from an archive
-t List files in an archive
-v Verbose output
-f Specify the filename of the archive
-z Filter the archive through gzip
-j Filter the archive through bzip2

Examples:

To create a tar archive:

tar -cvf archive.tar file1.txt file2.txt

To extract a tar archive:

tar -xvf archive.tar

To create a compressed tar archive:

tar -czvf archive.tar.gz file1.txt file2.txt

To list the contents of a tar archive:

tar -tvf archive.tar

Differences Between gzip and zip

Differences Between gzip, zip, and tar

Viewing Contents of a Tar Archive Without Extracting

You can use the -t option with tar to list the contents of an archive without extracting it.

Example:

tar -tvf archive.tar

This command lists all files and directories in the archive.tar file.

Summary

The gzip, zip, and tar commands are essential for file compression and archiving in Linux. gzip is used for compressing single files, zip is used for creating compressed archives of multiple files and directories, and tar is used for archiving multiple files and directories without compression (but can be combined with gzip for compressed archives). Understanding the differences and common options for these commands allows you to efficiently manage and organize files in Linux.

Chapter 8

locate and find comands

The locate and find commands are powerful tools in Linux for searching files and directories. Each serves a different purpose and has distinct advantages and characteristics.

The locate Command

The locate command searches for files and directories by name using a prebuilt database. This database is created and updated regularly by the updatedb command, making locate extremely fast for searching. However, because it relies on this database, the results might not reflect recent changes to the filesystem unless the database has been updated.

Basic usage of locate:

locate [options] pattern

Commonly Used locate Options:

Option Description
-i Perform case-insensitive matching
-r Use regular expressions
-n Limit the number of results
-e Print only existing files
--existing Same as -e, print only existing files
--regex Same as -r, interpret the pattern as a regex

Examples:

To find files with "report" in their name:

locate report

To find files with "report" in their name, case-insensitively:

locate -i report

To find files using a regular expression:

locate -r 'report[0-9]*\.txt'

To update the locate database:

sudo updatedb

This command updates the database that locate uses, ensuring the search results are current.

The find Command

The find command is used to search for files and directories within the filesystem based on various criteria. Unlike locate, find searches the directory tree in real-time, making it more versatile but generally slower.

Basic usage of find:

find [path] [expression]
Tests

Tests are criteria used by find to match files and directories.

Test Description
-name Matches files with the specified name
-iname Matches files with the specified name, case-insensitive
-type Matches files of the specified type (f for file, d for directory)
-size Matches files of the specified size
-mtime Matches files modified n days ago
-user Matches files owned by the specified user
-cmin Matches files changed n minutes ago
-cnewer Matches files changed more recently than another file
-ctime Matches files changed n days ago
-empty Matches empty files and directories
Operators

Operators are used to combine multiple tests in find.

Operator Description
-and Logical AND
-or Logical OR
! Logical NOT
Predefined Actions

Actions are operations that find performs on the matched files and directories.

Action Description
-print Print the matching files
-delete Delete the matching files
-exec Execute a command on matching files

Examples of Using Operators and Predefined Actions:

To find files named "report.txt" and print their names:

find /path/to/search -name report.txt -print

To find files that are either named "report.txt" or "summary.txt":

find /path/to/search \( -name report.txt -or -name summary.txt \) -print

To find files that are not directories:

find /path/to/search ! -type d -print

To find files named "report.txt" and "summary.txt" and delete them:

find /path/to/search \( -name report.txt -or -name summary.txt \) -delete

To find files owned by "john" and modified in the last 7 days:

find /path/to/search -user john -and -mtime -7 -print
Size Options

Size options specify the size criteria for matching files.

Option Description
+n Greater than n units
-n Less than n units
n Exactly n units
c Bytes (default)
k Kilobytes (1024 bytes)
M Megabytes (1024 kilobytes)
G Gigabytes (1024 megabytes)

Examples of Size Options:

To find files larger than 1MB:

find /path/to/search -size +1M

To find files between 1MB and 10MB:

find /path/to/search -size +1M -size -10M

To find files smaller than 1MB:

find /path/to/search -size -1M

Using the -exec Option

The -exec option in the find command allows you to execute a command on each file that matches the search criteria. The {} placeholder is used to represent the current file, and \; is used to terminate the command.

Examples with -exec:

To delete all .tmp files:

find /path/to/search -name "*.tmp" -exec rm {} \;

To change permissions of all .sh files to executable:

find /path/to/search -name "*.sh" -exec chmod +x {} \;

To move all .log files to another directory:

find /path/to/search -name "*.log" -exec mv {} /path/to/destination/ \;

To copy all .conf files to a backup directory:

find /path/to/search -name "*.conf" -exec cp {} /path/to/backup/ \;

To list all .txt files with detailed information:

find /path/to/search -name "*.txt" -exec ls -l {} \;

Understanding xargs

The xargs command is used to build and execute command lines from standard input. It is often used in conjunction with find to handle a large number of files.

Examples with xargs:

To delete all .tmp files using find and xargs:

find /path/to/search -name "*.tmp" -print | xargs rm

To change permissions of all .sh files to executable using find and xargs:

find /path/to/search -name "*.sh" -print | xargs chmod +x

To move all .log files to another directory using find and xargs:

find /path/to/search -name "*.log" -print | xargs -I {} mv {} /path/to/destination/

To copy all .conf files to a backup directory using find and xargs:

find /path/to/search -name "*.conf" -print | xargs -I {} cp {} /path/to/backup/

To list all .txt files with detailed information using find and xargs:

find /path/to/search -name "*.txt" -print | xargs ls -l

Summary

The locate and find commands are essential for searching files and directories in Linux. locate uses a prebuilt database for fast searches and is ideal for quick lookups. However, it requires the database to be updated regularly. find is more versatile, allowing for real-time searches based on various criteria, but can be slower. The -exec option in find allows executing commands on matched files, and xargs is a powerful tool to handle large numbers of files efficiently. By mastering these commands and understanding their options, you can efficiently locate and manage files in the Linux environment.

Regular Expressions

Understanding grep

The grep command, short for "global regular expression print," is a powerful utility used to search text files for lines that match a specified pattern. It reads the file line by line and prints any lines that contain a match. grep supports regular expressions, which allow for complex and flexible pattern matching. This makes grep an invaluable tool for searching and analyzing text data in Unix-like systems.

Basic usage of grep:

grep [options] pattern [file...]

Example: To search for lines containing the word "error" in a file named logfile.txt:

grep 'error' logfile.txt

What are Regular Expressions?

Regular expressions (regex) are symbolic notations used to identify patterns in text. They enable powerful and flexible text searches, matches, and manipulations. Regular expressions are supported by many command-line tools and programming languages, making them essential for effective text processing.

Basic Regular Expressions (BRE)

Basic Regular Expressions (BRE) are the simpler form of regular expressions, supported by utilities like grep. BRE uses a limited set of metacharacters and requires some characters to be escaped with a backslash ().

Common Metacharacters in BRE:

Metacharacter Description Example Explanation
^ Matches the start of a line ^abc Matches "abc" at the beginning of a line
$ Matches the end of a line abc$ Matches "abc" at the end of a line
. Matches any single character a.c Matches "abc", "a c", "a-c", etc.
[] Matches any single character within the brackets [abc] Matches "a", "b", or "c"
* Matches zero or more occurrences of the previous character a* Matches "", "a", "aa", "aaa", etc.
\{n\} Matches exactly n occurrences of the previous character a\{3\} Matches "aaa"
\{n,m\} Matches between n and m occurrences of the previous character a\{2,4\} Matches "aa", "aaa", or "aaaa"
\? Matches zero or one occurrence of the previous character a\? Matches "a" or ""
\+ Matches one or more occurrences of the previous character a\+ Matches "a", "aa", "aaa", etc.

Examples of BRE:

^abc matches "abc" at the beginning of a line:

echo "abcdef" | grep '^abc'

Output:

abcdef

abc$ matches "abc" at the end of a line:

echo "123abc" | grep 'abc$'

Output:

123abc

a.c matches any character between "a" and "c":

echo "abc" | grep 'a.c'

Output:

abc

[abc] matches any single character "a", "b", or "c":

echo "a" | grep '[abc]'
echo "b" | grep '[abc]'
echo "c" | grep '[abc]'

Output:

a
b
c

a* matches zero or more occurrences of "a":

echo "aaab" | grep 'a*'

Output:

aaab

a\{3\} matches exactly three occurrences of "a":

echo "aaa" | grep 'a\{3\}'

Output:

aaa

a\{2,4\} matches between two and four occurrences of "a":

echo "aaa" | grep 'a\{2,4\}'

Output:

aaa

a\? matches zero or one occurrence of "a":

echo "a" | grep 'a\?'

Output:

a

a\+ matches one or more occurrences of "a":

echo "aaa" | grep 'a\+'

Output:

aaa

Extended Regular Expressions (ERE)

Extended Regular Expressions (ERE) are a more powerful and flexible version of regular expressions. They were developed to address the limitations of BRE by introducing additional metacharacters and operators. ERE does not require escaping for certain characters, making the expressions more readable and easier to write. ERE is supported by utilities like egrep or grep -E.

Common Metacharacters in ERE:

Metacharacter Description Example Explanation
^ Matches the start of a line ^abc Matches "abc" at the beginning of a line
$ Matches the end of a line abc$ Matches "abc" at the end of a line
. Matches any single character a.c Matches "abc", "a c", "a-c", etc.
[] Matches any single character within the brackets [abc] Matches "a", "b", or "c"
* Matches zero or more occurrences of the previous character a* Matches "", "a", "aa", "aaa", etc.
{n} Matches exactly n occurrences of the previous character a{3} Matches "aaa"
{n,m} Matches between n and m occurrences of the previous character a{2,4} Matches "aa", "aaa", or "aaaa"
? Matches zero or one occurrence of the previous character a? Matches "a" or ""
+ Matches one or more occurrences of the previous character a+ Matches "a", "aa", "aaa", etc.
() Groups expressions `(abc def)`

Examples of ERE:

^abc matches "abc" at the beginning of a line:

echo "abcdef" | grep -E '^abc'

Output:

abcdef

abc$ matches "abc" at the end of a line:

echo "123abc" | grep -E 'abc$'

Output:

123abc

a.c matches any character between "a" and "c":

echo "abc" | grep -E 'a.c'

Output:

abc

[abc] matches any single character "a", "b", or "c":

echo "a" | grep -E '[abc]'
echo "b" | grep -E '[abc]'
echo "c" | grep -E '[abc]'

Output:

a
b
c

a* matches zero or more occurrences of "a":

echo "aaab" | grep -E 'a*'

Output:

aaab

a{3} matches exactly three occurrences of "a":

echo "aaa" | grep -E 'a{3}'

Output:

aaa

a{2,4} matches between two and four occurrences of "a":

echo "aaa" | grep -E 'a{2,4}'

Output:

aaa

a? matches zero or one occurrence of "a":

echo "a" | grep -E 'a?'

Output:

a

a+ matches one or more occurrences of "a":

echo "aaa" | grep -E 'a+'

Output:

aaa

a|b matches either "a" or "b":

echo "a" | grep -E 'a|b'
echo "b" | grep -E 'a|b'

Output:

a
b

(abc|def) matches "abc" or "def":

echo "abc" | grep -E '(abc|def)'
echo "def" | grep -E '(abc|def)'

Output:

abc
def

Character Classes

Character classes in regular expressions allow you to match specific sets of characters. These sets are predefined and make it easier to work with groups of characters.

Common Character Classes:

Character Class Description Example Explanation
[:blank:] Matches spaces and tabs grep '[[:blank:]]' Matches spaces and tabs in the text
[:upper:] Matches uppercase letters grep '[[:upper:]]' Matches any uppercase letter
[:lower:] Matches lowercase letters grep '[[:lower:]]' Matches any lowercase letter
[:digit:] Matches digits grep '[[:digit:]]' Matches any digit
[:alpha:] Matches alphabetic characters grep '[[:alpha:]]' Matches any alphabetic character
[:alnum:] Matches alphanumeric characters grep '[[:alnum:]]' Matches any alphanumeric character
[:punct:] Matches punctuation characters grep '[[:punct:]]' Matches any punctuation character
[:space:] Matches all whitespace characters grep '[[:space:]]' Matches spaces, tabs, and newlines

Examples of Character Classes:

To match spaces and tabs using [:blank:]:

echo "hello world" | grep '[[:blank:]]'

Output:

hello world

To match uppercase letters using [:upper:]:

echo "Hello World" | grep '[[:upper:]]'

Output:

Hello World

Summary

Regular expressions are a powerful tool for text manipulation in Linux. They allow you to identify patterns in text and perform complex searches and replacements. Basic Regular Expressions (BRE) provide a fundamental set of pattern matching capabilities, while Extended Regular Expressions (ERE) offer more advanced features and flexibility. Character classes, like [:blank:] and [:upper:], further enhance your ability to match specific sets of characters. By understanding and mastering both BRE and ERE, along with character classes, you can significantly enhance your ability to manipulate and analyze text in Unix-like systems.

Chapter 9

cut,paste and join Commands

The cut, paste, and join commands in Unix/Linux are powerful tools for text processing, allowing users to manipulate and merge data from text files efficiently. Each command has its specific functionality and options that make it versatile for various tasks.

cut Command

The cut command is used to extract sections from each line of input files. It operates by specifying a delimiter and selecting the desired fields or columns of data. The cut command can also work with fixed-length fields.

Common Options:

Option Description
-b Selects only the bytes specified.
-c Cuts based on character positions.
-d Specifies a delimiter (default is tab).
-f Specifies the fields to be extracted.
--complement Complements the selection.
--output-delimiter Sets the output delimiter for the cut fields.

Examples:

Suppose file.txt contains the following lines:

John,Doe,30,Engineer
Jane,Smith,25,Designer
Mike,Johnson,40,Manager

Extracting the first and third fields using a comma as a delimiter:

cut -d ',' -f 1,3 file.txt

Output:

John,30
Jane,25
Mike,40

Extracting the first 5 characters of each line:

cut -c 1-5 file.txt

Output:

John,
Jane,
Mike,

Extracting the first field using a space as a delimiter (assuming fields are space-separated):

cut -d ' ' -f 1 file.txt

Extracting bytes 2 to 6 of each line:

cut -b 2-6 file.txt

Output:

ohn,D
ane,S
ike,J

Extracting all fields except the second using a tab delimiter:

cut -d $'\t' --complement -f 2 file.txt

Note: This example assumes that fields in the file are tab-separated.

paste Command

The paste command merges lines of files horizontally by outputting lines from each file side by side, separated by a delimiter. This command is useful for combining files where each line corresponds to the same logical row of data.

Common Options:

Option Description
-d Specifies a delimiter to separate pasted lines.
-s Serially paste one file at a time.

Examples:

Suppose file1.txt contains:

John
Jane
Mike

And file2.txt contains:

Doe
Smith
Johnson

Pasting two files side by side with a tab delimiter:

paste file1.txt file2.txt

Output:

John    Doe
Jane    Smith
Mike    Johnson

Suppose file1.txt contains:

John
Jane
Mike

And file2.txt contains:

Doe
Smith
Johnson

Pasting files with a comma as the delimiter:

paste -d ',' file1.txt file2.txt

Output:

John,Doe
Jane,Smith
Mike,Johnson

Suppose file1.txt contains:

John
Jane
Mike

And file2.txt contains:

Doe
Smith
Johnson

Pasting files with a space as the delimiter:

paste -d ' ' file1.txt file2.txt

Output:

John Doe
Jane Smith
Mike Johnson

Suppose file.txt contains:

John
Doe
Jane
Smith
Mike
Johnson

Serially pasting lines from a single file:

paste -s file.txt

Output:

John    Doe
Jane    Smith
Mike    Johnson

Suppose file1.txt contains:

John
Jane
Mike

And file2.txt contains:

Doe
Smith
Johnson

Pasting files with a custom delimiter:

paste -d '|' file1.txt file2.txt

Output:

John|Doe
Jane|Smith
Mike|Johnson

join Command

The join command merges lines of two sorted files based on a common field. This command is similar to the SQL join operation, combining records with matching keys.

Common Options:

Option Description
-1 Specifies the field number to join on from the first file.
-2 Specifies the field number to join on from the second file.
-t Specifies a delimiter (default is space).
-o Formats the output.
-a Prints unpairable lines from the specified file.
-e Replaces empty output fields with the specified string.

Examples:

Suppose file1.txt contains:

A John
B Jane
C Mike

And file2.txt contains:

A Doe
B Smith
C Johnson

Joining these files on the first field (default delimiter is space):

join file1.txt file2.txt

Output:

A John Doe
B Jane Smith
C Mike Johnson

Suppose file1.txt contains:

Alpha,John
Beta,Jane
Gamma,Mike

And file2.txt contains:

Alpha,Doe
Beta,Smith
Gamma,Johnson

Joining these files on the first field using a comma as the delimiter:

join -t ',' file1.txt file2.txt

Output:

Alpha,John,Doe
Beta,Jane,Smith
Gamma,Mike,Johnson

Suppose file1.txt contains:

1A John
2B Jane
3C Mike

And file2.txt contains:

1A Doe
2B Smith
3C Johnson

Joining these files on the first field:

join file1.txt file2.txt

Output:

1A John Doe
2B Jane Smith
3C Mike Johnson

In summary, cut, paste, and join are essential commands for text manipulation in Unix/Linux. They offer a variety of options for extracting, merging, and combining data efficiently, making them invaluable for tasks involving text processing. Understanding these commands and their options enables users to perform complex data manipulations with ease.

comm, diff and patch Commands

comm Command

The comm command is used to compare two sorted files line by line. It outputs three columns:

  1. Lines unique to the first file.
  2. Lines unique to the second file.
  3. Lines common to both files.

Syntax:

comm [OPTION]... FILE1 FILE2

Common Options:

Option Description
-1 Suppress column 1 (lines unique to FILE1)
-2 Suppress column 2 (lines unique to FILE2)
-3 Suppress column 3 (lines common to both files)

Example:

file1.txt:

apple
banana
cherry

file2.txt:

banana
cherry
date

Command:

comm file1.txt file2.txt

Output:

apple
        banana
        cherry
                date

diff Command

The diff command is used to compare files line by line. It shows the differences between two files.

Syntax:

diff [OPTION]... FILES

Common Options:

Option Description
-u Output in unified format
-c Output in context format
-i Ignore case differences
-w Ignore all white space

Example with -c option:

file1.txt:

apple
banana
cherry

file2.txt:

banana
cherry
date

Command:

diff -c file1.txt file2.txt

Output:

*** file1.txt 2024-07-11 10:00:00.000000000 +0000
--- file2.txt 2024-07-11 10:00:00.000000000 +0000
***************
*** 1,3 ****
  apple
  banana
  cherry
--- 1,3 ----
  banana
  cherry
  date

Here's how to interpret it the results shown above:

Change Context

Content Changes

Key Observations

  1. Line Removed:
    The line apple from file1.txt is removed in file2.txt.

  2. Line Added:
    The line date is added in file2.txt.

Using diff to create a patch file

You can also create a patch file (to be used later with the patch command) using diff

diff file1 file2 > patchfile

patch Command

The patch command is used to apply changes to a file. It takes a diff file created by the diff command and applies those changes to the original file.

Syntax:

patch [OPTION]... [ORIGFILE [PATCHFILE]]

Common Options:

Option Description
-pNUM Strip NUM leading components from file names
-R Reverse the effect of the patch
-i Read patch from file
-o Output to specified file

Example using diff.patch:

file1.txt:

apple
banana
cherry

diff.patch:

1d0
< apple
3a3
> date

Command:

patch file1.txt diff.patch

Output (file1.txt after patch):

banana
cherry
date

These commands are very useful for file comparison and manipulation tasks in Linux environments, especially when dealing with text files and source code.

tr, sed and aspell Commands

tr Command

The tr command in Linux is used for translating or deleting characters from the input provided. It reads from standard input and writes to standard output.

Common Options for tr:

Option Description
-d Delete characters from the input
-s Squeeze repeated characters into a single character
-c Complement the set of characters

Examples:

Consider a file example.txt with the following content:

hello world
hello universe

To convert all lowercase characters to uppercase, you can use:

cat example.txt | tr 'a-z' 'A-Z'

Output:

HELLO WORLD
HELLO UNIVERSE

To delete all vowels from the text:

cat example.txt | tr -d 'aeiou'

Output:

hll wrld
hll nvrse

To change the blank space between two words to a hyphen:

cat example.txt | tr ' ' '-'

Output:

hello-world
hello-universe

To change all blank spaces (including tabs) to a hyphen using [[:blank:]]:

cat example.txt | tr '[[:blank:]]' '-'

Output:

hello-world
hello-universe

[[:blank:]] is a character class that matches all horizontal whitespace characters, which include spaces and tabs. This is useful when you want to ensure that all types of horizontal whitespace are replaced.

sed Command

The sed command, short for stream editor, is used to perform basic text transformations on an input stream (a file or input from a pipeline).

Common Options for sed:

Option Description
-e Add the script to the commands to be executed
-f Add the script file
-i Edit files in place
-n Suppress automatic printing of pattern space

Examples:

Consider a file example.txt with the following content:

hello world
hello hello universe
hello everyone hello

To replace the first occurrence of the word "hello" with "hi" on each line:

sed 's/hello/hi/' example.txt

Output:

hi world
hi hello universe
hi everyone hello

To replace all occurrences of the word "hello" with "hi" using the global /g option:

sed 's/hello/hi/g' example.txt

Output:

hi world
hi hi universe
hi everyone hi

To delete lines containing the word "everyone":

sed '/everyone/d' example.txt

Output:

hello world
hello hello universe

aspell Command

The aspell command is a spell-checking utility in Linux. It can be used interactively to check the spelling of a document.

Common Options for aspell:

Option Description
-c Check a file
-a Run in 'pipe mode'
-l List available dictionaries
-d Use a specific dictionary

Examples:

Consider a file example.txt with the following content:

helo wrld
hello everyone

To check the spelling of the file:

aspell check example.txt

Interactive mode will prompt corrections:

@(#) International Ispell Version 3.1.20 (but really Aspell 0.60.8)
 & helo 5 0: hello, he lo, halo, help, helot
 & wrld 8 0: world, word, weld, wild, wold, wrld's, WRLD, WLRD

To list available dictionaries:

aspell dicts

Output might include:

en
en_GB
en_US
fr
de
...

These commands are essential for text processing and transformation tasks in Linux, providing powerful tools for manipulating and validating text in various ways.

Chapter 10

A Gentle Guid to VIM (alias vi)

The vim editor is a powerful text editor available in most Unix-like operating systems. To launch vim, simply type vim followed by the filename you wish to edit, such as vim myfile.txt. When you open vim, it starts in Command mode, which is the default mode. vim operates in two primary modes: Command mode and Insert mode. Understanding how to switch between these modes is key to effectively using vim.

In Command mode, you can navigate through the file, delete text, and perform various editing tasks. To switch from Command mode to Insert mode, you can press i to insert text before the cursor, a to insert text after the cursor, o to open a new line below the current line, or O to open a new line above the current line. For example, if your cursor is on the letter 'h' in "hello" and you press i, you can start typing before 'h'. If you press a, you start typing after 'h'. If you press o, a new line opens below the current line and switches to Insert mode. If you press O, a new line opens above the current line and switches to Insert mode. Once in Insert mode, you can type text as you normally would. To switch back to Command mode, press the Esc key.

# Open a file
vim myfile.txt

# Insert text before the cursor
i

# Insert text after the cursor
a

# Open a new line below the current line
o

# Open a new line above the current line
O

# Switch back to Command mode from Insert mode
<Esc>

While in Command mode, basic navigation can be performed using the keys h, j, k, and l to move the cursor left, down, up, and right respectively. You can also use the arrow keys for navigation (this is much easier). For example, pressing 4j will move the cursor down four lines. If you want to delete a character under the cursor, press x. To delete an entire line, use the dd command. For example, if you want to delete three lines starting from the current line, you can type 3dd. Copying (or yanking) a line can be done with yy, and you can paste it below the current line by pressing p. For example, 3yy will yank (copy) three lines, and p will paste them below the cursor. If you make a mistake, you can undo your last change by pressing u. To save (write) your changes to the file, use :w, and to quit vim, use :q. If you want to save and quit simultaneously, you can use :wq or the shortcut ZZ.

# Move the cursor down four lines
4j

# Delete the character under the cursor
x

# Delete three lines starting from the current line
3dd

# Yank (copy) three lines
3yy

# Paste the copied lines below the cursor
p

# Undo the last change
u

To save your changes to the file, use the :w command. To save and quit vim simultaneously, you can use :wq or the shortcut ZZ. If you want to quit without saving, use :q!.

# Save the file
:w

# Save and quit vim
:wq

# Save and quit vim (alternative)
ZZ

# Quit without saving
:q!

For more advanced navigation, vim offers commands like gg to go to the beginning of the file and G to go to the end. You can go to a specific line by typing nG, where n is the line number. For example, 10G takes you to line 10. Moving to the beginning of the current line can be done with 0, and moving to the end of the line can be done with $.

# Go to the beginning of the file
gg

# Go to the end of the file
G

# Go to line 10
10G

# Move to the beginning of the current line
0

# Move to the end of the current line
$

Editing in vim can be very efficient with commands like r to replace a character, cw to change a word, and dw to delete a word. For instance, if the cursor is on the letter 'h' in "hello" and you press rj, the 'h' will be replaced with 'j', resulting in "jello". Using cw on "hello" allows you to change the entire word, for example to "world", by typing cw world. Using dw will delete from the cursor position to the end of the current word. You can also select text visually by pressing v and then moving the cursor to highlight the desired text. This selected text can be deleted, copied, or changed using d, y, or c followed by a movement command. For example, d$ will delete from the cursor to the end of the line, and y2w will yank (copy) two words.

# Replace the character under the cursor with 'j'
rj

# Change the word under the cursor to 'world'
cw world

# Delete from the cursor to the end of the word
dw

# Start visual mode to select text
v

# Delete from the cursor to the end of the line
d$

# Yank (copy) two words
y2w

Searching within vim is straightforward. To search forward for a pattern, use /pattern, and to search backward, use ?pattern. For example, /hello searches forward for the word "hello" and ?hello searches backward. After performing a search, pressing n will repeat the search in the same direction, while pressing N will repeat it in the opposite direction.

# Search forward for 'hello'
/hello

# Search backward for 'hello'
?hello

# Repeat the search in the same direction
n

# Repeat the search in the opposite direction
N

When you are ready to exit vim, you have a few options. To quit without saving, use :q!. To save your changes and quit, use :wq or the shortcut ZZ, which performs the same function.

# Quit without saving
:q!

# Save and quit vim
:wq

# Save and quit vim (alternative)
ZZ

Practicing these commands will help you become proficient with vim. Remember that pressing Esc always returns you to Command mode, which is a useful fallback if you ever feel lost. As you get more comfortable, you can explore vim's more advanced features, but mastering these basics will provide a solid foundation for effective text editing with vim.

Working With Multiple Files in Vim

When working with multiple files in Vim, you can use buffers to keep several files open at once, while also taking advantage of split screens to view and edit files side by side. Let’s walk through how these tools make it easier to manage multiple files.

To start, you can open multiple files in Vim by listing them when you launch it. For example, to open file1.txt and file2.txt, type this at the command line:

vim file1.txt file2.txt

This command opens both files, each in a separate buffer. A buffer is where Vim stores each open file in memory, allowing you to switch between files easily. To move between buffers, use the :buffer command followed by the buffer number. For example, if you’re viewing file1.txt and want to switch to file2.txt, type:

:buffer 2

You can find buffer numbers by typing:

:ls

The :ls command shows a list of all open buffers, making it easy to see which files are available and their corresponding buffer numbers.

To open a new file in a buffer while in Vim, use:

:e file3.txt

This loads file3.txt into a new buffer, which you can access with :buffer commands or open in a new split screen if you want to view it alongside other files.

When you’re done with a file, you can close its buffer with the command:

:bd

This closes the buffer of the file you’re currently viewing. You can also delete a specific buffer by its number, like this:

:bd 2

You can also move forward and backward between bufferes with the :bn (next buffer) and :bp (previous buffer) commands.

If you want to view files side by side, you can use split screens. To open another file in a split screen, use the following command:

:split file2.txt

This command splits the window horizontally, placing file2.txt above or below file1.txt. To split vertically, you can use:

:vsplit file2.txt

Now, file1.txt will be on the left and file2.txt on the right. You can navigate between these split windows by pressing Ctrl + w and then using the arrow keys. For example, Ctrl + w followed by the right arrow key will move your cursor to the window on the right. This way, you can view and edit two files at the same time.

To close the split window where your cursor is currently active, use:

:q

This will close the current split. If there are unsaved changes, Vim will prompt you to save or discard them.

If you want to keep only the current split open and close all other splits, use:

:only

This command closes all other splits except the one you are currently working in, bringing you back to a single window view.

To force-close a split without saving any changes, use:

:q!

This will close the split immediately, discarding any changes.

These commands let you easily manage split screens in Vim and return to a single screen setup.

Using buffers and split screens allows you to handle multiple files efficiently in Vim. You can open files, switch between them with buffers, and view them side by side with split screens, giving you all the flexibility you need for multitasking on your files.

Table of popular vim Commands

Command Description
vim myfile.txt Open a file
i Insert text before the cursor (Insert mode)
a Insert text after the cursor (Insert mode)
o Open a new line below the current line (Insert mode)
O Open a new line above the current line (Insert mode)
<Esc> Switch back to Command mode
h, j, k, l Move cursor left, down, up, right (also arrow keys)
4j Move cursor down four lines
x Delete character under the cursor
3dd Delete three lines starting from the current line
3yy Yank (copy) three lines
p Paste copied lines below the cursor
u Undo the last change
:w Save the file
:wq Save and quit vim
ZZ Save and quit vim (alternative)
:q! Quit without saving
gg Go to the beginning of the file
G Go to the end of the file
10G Go to line 10
0 Move to the beginning of the current line
$ Move to the end of the current line
rj Replace the character under the cursor with 'j'
cw world Change the word under the cursor to 'world'
dw Delete from the cursor to the end of the word
v Start visual mode to select text
d$ Delete from the cursor to the end of the line
y2w Yank (copy) two words
/hello Search forward for 'hello'
?hello Search backward for 'hello'
n Repeat the search in the same direction
N Repeat the search in the opposite direction

Other vim commands that help with how vim appears

Command Description
:set nu Set line numbers
:set nonu Remove line numbers
:colorscheme ctrl-d Colorscheme space then ctrl-d. This will provide a list of colorschemes
:colorscheme color Colorscheme space then theme color. This will switch the color scheme to the color you entered
:!bash % Run the current bash script in vim
:noh Clear highlighting

Creating a Bash Script, Comments, Variables and Read

A Bash script is a file containing a series of commands that are executed by the Bash shell, a popular command-line interpreter in Unix-like operating systems. These scripts are used to automate tasks, manage system operations, and perform complex sequences of commands with a single execution.

Writing a Simple Bash Script

Creating a Bash script involves writing commands in a plain text file and then executing the file. Here's an example of a simple Bash script that prints "Hello, World!" to the console:

  1. Open a text editor and write the following lines:

    #!/bin/bash
# This is a comment
echo "Hello, World!"
    • The #!/bin/bash at the top is called a shebang. It tells the system to use the Bash interpreter to execute the script.
    • The echo command prints text to the terminal.
    • The line starting with # is a comment, which is ignored by the shell and used to add explanations or notes.

  2. Save the file with a .sh extension, for example, hello_world.sh.

Changing Script Permissions

Before running the script, you need to change its permissions to make it executable. This can be done using the chmod command:

chmod 700 hello_world.sh

The 700 permission means that the file's owner has read, write, and execute permissions, while no one else has any permissions. This ensures that only the creator can run the script.

Running a Bash Script

To run the script, navigate to the directory where the script is saved and use one of the following methods:

  1. Using ./:

    ./hello_world.sh

    The ./ indicates that the script is located in the current directory.

  2. Using bash:

    bash hello_world.sh

    This explicitly uses the Bash interpreter to run the script.

  3. Using the full path:

    /path/to/your/script/hello_world.sh

    This method runs the script by specifying its full path.

Comments in Bash Scripts

Comments in Bash scripts are lines that begin with #. They are not executed by the shell and are used to add descriptions, explanations, or notes within the script. Comments are useful for:

For example:

#!/bin/bash
# This script prints a greeting message
echo "Hello, World!"  # This line prints "Hello, World!"

Variables in Bash Scripts

Variables in Bash scripts are used to store data that can be referenced and manipulated throughout the script. Variables make scripts more flexible and easier to manage. You can assign a value to a variable using the = operator and access its value by prefixing the variable name with a $ sign. The $ sign is necessary to differentiate the variable's value from the variable's name.

IMPORTANT NOTE: Variable names need to be repesentative of the data. For example, you would using the variable firstName to contain someones first name. Also variables in bash scripting (unless you use other libraries) contain strings which is any text within double quotes, like firstName="Scott". Or numbers which is just a whole number like num=43. Finally, variable names cannot contain spaces. For example, the variable first name must be written as first_name or using camel case like firstName where the first letter of each word is capitalized.

For example:

#!/bin/bash
# This script uses a variable to print a message
greeting="Hello, World!"
echo $greeting

In this script:

Without the $ sign, the shell would interpret greeting as a literal string rather than the variable's value.

Using read Command

The read command in Linux is a simple and powerful way to get input from a user in a script. When you use read, the command waits for the user to type something and press Enter, and then it stores that input in a variable you can use in the rest of your script. This is especially useful in interactive scripts where you need the user’s input to make decisions or perform actions.

Let’s look at a basic example of how read works. Suppose you want to ask the user for their name and then greet them:

echo "What is your name?"
    read name
    echo "Hello, $name!"

In this script:

  1. echo "What is your name?" displays the question to the user.
  2. read name waits for the user to type their name and press Enter. The input is stored in a variable called name.
  3. echo "Hello, $name!" prints a greeting that includes the user’s name.

When the script runs, the user will see “What is your name?” on the screen. They can type their name, and the script will respond with “Hello, [name]!” where [name] is whatever they typed.

The read command can also handle multiple pieces of input. For example, if you want to ask for both a first and last name, you can do this:

echo "Enter your first and last name:"
    read first last
    echo "Hello, $first $last!"

Here, read first last allows the user to type both their first and last name separated by a space. The first word they type goes into the first variable, and the second word goes into the last variable.

You can also use options with read to make it more useful. For example, the -p option lets you display a prompt on the same line as the read command, and the -s option hides the user’s input, which is helpful for passwords:

read -p "Enter your username: " username
    read -sp "Enter your password: " password
    echo
    echo "Username: $username, Password: [hidden]"

In this example:

  1. -p "Enter your username: " displays the prompt on the same line.
  2. -s hides the user’s input for password.
  3. The echo command after the read for password is there to move to a new line after the hidden input.

The read command makes scripts interactive and lets users provide the information you need, allowing for dynamic and responsive scripts. It’s an essential command for adding interactivity to your Linux scripts!

Understanding /bin and Adding to PATH

The /bin directory (short for "binary") is one of the standard directories in Unix-like systems where essential command binaries are stored. You should have a bin directory in your home directory, if not, you can create one. If you put your scripts in the bin directory you can call them but just entering the script by name (no need for ./ or bash or a full path). To run scripts or commands from any directory you can add their directory to the system's PATH environment variable. This allows the system to locate the executable files in the specified directories.

To add a directory to your PATH, you can modify your shell configuration file, such as .bashrc or .bash_profile. Here’s how you can add a custom directory (e.g., ~/scripts) to your PATH:

  1. Open your shell configuration file in a text editor:

    vim ~/.bashrc

  2. Add the following line at the end of the file:

    export PATH="$PATH:~/scripts"

  3. Save the file and reload the configuration:

    source ~/.bashrc

Now, any executable files in the ~/scripts directory can be run from any location in the terminal.

Summary

A Bash script is a powerful tool for automating tasks in Unix-like operating systems. Writing a script involves creating a text file with a series of commands, changing its permissions to make it executable, and using comments to document the script. Variables in Bash scripts allow you to store and manipulate data, making your scripts more flexible and maintainable. To run a script, you can use the ./ notation, the bash command, or the full path. Adding directories to the PATH environment variable enables you to run scripts and commands from any location without specifying their full path. By understanding these fundamental concepts, you can create efficient and effective Bash scripts to streamline your workflow.

Chapter 11

Bash Functions

Syntax Variations

Without parentheses:

function_name {
    # Commands
}

With parentheses:

function_name() {
    # Commands
}

Differences and Similarities

Function Keyword (Optional):

In some variations, you might see the function keyword used to define a function. This is optional and generally not necessary in modern Bash scripting. However, when using the function keyword, the parentheses are typically omitted:

function function_name {
    # Commands
}

Parentheses:

The use of parentheses () is optional in Bash. They are commonly used for clarity and readability to explicitly indicate a function definition. Function definitions with parentheses are more common and align with the style of many other programming languages, making the script easier to understand for people with experience in those languages.

POSIX Compliance:

The syntax function_name() { ... } is more aligned with POSIX standards and is preferred for writing scripts intended to be portable across different Unix-like systems. The function keyword is not POSIX-compliant and is specific to Bash and some other shells like Zsh and Ksh.

Variable Scope:

There is no difference in how variables are scoped within the function based on the syntax used to define the function. Both styles allow the use of local variables within the function using the local keyword.

Here's a simple example of a function that prints a greeting message:

#!/bin/bash

# Define a function
greet() {
    echo "Hello, World!"
}

# Call the function
greet

In this script, the greet function is defined to print "Hello, World!" and is then called to execute its commands. Note that functions in Bash can be defined with or without parentheses. The use of parentheses is a common convention to clearly indicate the function definition, although they are not strictly necessary if no parameters are being passed.

Returning Values from Functions

Bash functions can return status codes using the return command. The return value is an integer between 0 and 255, where 0 typically signifies success, and any non-zero value indicates an error. Bash functions cannot return string or numeric values directly using return. Instead, you can use echo or variable assignment to get these values out of a function.

Here’s an example of a function that returns a status code:

#!/bin/bash

# Define a function that checks if a number is positive
is_positive() {
    if [ $1 -gt 0 ]; then
        return 0  # Success
    else
        return 1  # Failure
    fi
}

# Call the function with a number
is_positive 5
status=$?

if [ $status -eq 0 ]; then
    echo "The number is positive."
else
    echo "The number is not positive."
fi

In this script, the is_positive function checks if a given number (passed as an argument) is positive. The function returns 0 if the number is positive and 1 otherwise. The return status is captured in the $? variable and used to print an appropriate message.

Scoping in Bash Functions

Variables in Bash have a global scope by default, meaning they are accessible throughout the script, including inside functions. To create local variables within a function, you can use the local keyword. Local variables are only accessible within the function where they are defined.

Here’s an example demonstrating variable scoping:

#!/bin/bash

# Global variable
message="Hello, World!"

# Define a function
print_message() {
    local message="Hello, Local World!"
    echo $message
}

# Call the function
print_message

# Print the global variable
echo $message

In this script:

Passing Arguments to Functions

Functions in Bash can accept arguments. These arguments are accessed using positional parameters $1, $2, and so on, within the function.

Here’s an example of a function that takes arguments:

#!/bin/bash

# Define a function that prints a greeting message
greet() {
    echo "Hello, $1!"
}

# Call the function with an argument
greet "Alice"

In this script, the greet function takes one argument and prints a greeting message using that argument. When calling the function with "Alice" as the argument, it prints "Hello, Alice!".

Here is another example that takes multiple arguments. The arguments are in order so $1 = 5, $2 = 2 and $3 = 6


add(){
    echo "The numbers added together are $(($1 + $2 + $3))"
}
    
add 5 2 6

If you don't have the correct number of arguments you will get an error. The code below would generate an error.


add(){
    echo "The numbers added together are $(($1 + $2 + $3))"
}
    
add 5 2

If you use a number character in string bash will infer that it is a number. The code below will generate 12 as the answer.


add(){
    echo "The numbers added together are $(($1 + $2 + $3))"
}
    
add 5 2 "5"

If you add a alpha string it will just omit it. Here it returns 7 because it does not know what to do with "scott"


add(){
    echo "The numbers added together are $(($1 + $2 + $3))"
}
    
add 5 2 "scott"

Summary

Functions in Bash scripts are powerful tools that help organize and reuse code. They can accept arguments, return status codes, and have local variable scopes. By using functions, you can create more modular and maintainable scripts. Understanding how to define, call, and manage functions is essential for efficient Bash scripting. Functions help break down complex tasks into simpler parts, making scripts easier to read, debug, and maintain.

if, elif Statements

In Bash scripting, if statements are used to test conditions and make decisions based on the results. They help control the flow of execution in a script.

Basic if Statement

A basic if statement in Bash follows this syntax:

if [ condition ]; then
    # Commands to execute if condition is true
fi

Example:

#!/bin/bash

number=5

if [ $number -gt 3 ]; then
    echo "The number is greater than 3"
fi

In this example, the script checks if the variable number is greater than 3 and prints a message if the condition is true.

if-else Statement

An if-else statement allows you to execute different commands based on whether the condition is true or false.

if [ condition ]; then
    # Commands to execute if condition is true
else
    # Commands to execute if condition is false
fi

Example:

#!/bin/bash

number=2

if [ $number -gt 3 ]; then
    echo "The number is greater than 3"
else
    echo "The number is not greater than 3"
fi

In this example, the script prints a different message based on whether the number is greater than 3.

Multiple if-else Statements (elif)

You can use multiple if-else statements to test several conditions.

if [ condition1 ]; then
    # Commands to execute if condition1 is true
elif [ condition2 ]; then
    # Commands to execute if condition2 is true
else
    # Commands to execute if neither condition1 nor condition2 is true
fi

Example:

#!/bin/bash

number=3

if [ $number -gt 3 ]; then
    echo "The number is greater than 3"
elif [ $number -eq 3 ]; then
    echo "The number is equal to 3"
else
    echo "The number is less than 3"
fi

In this example, the script checks multiple conditions and prints the appropriate message based on the value of number.

Nested if-else Statements

Nested if-else statements are if-else statements within another if-else statement.

Example:

#!/bin/bash

number=4

if [ $number -gt 0 ]; then
    if [ $number -lt 10 ]; then
        echo "The number is between 1 and 9"
    else
        echo "The number is 10 or greater"
    fi
else
    echo "The number is zero or negative"
fi

In this example, the script first checks if the number is greater than 0. If true, it then checks if the number is less than 10 and prints the appropriate message based on the nested conditions.

Comparisons in Bash

In Bash, you can use various comparison operators to test conditions.

Equality (==):

if [ "$str1" == "$str2" ]; then
    echo "Strings are equal"
fi

Regex Match (=~):

if [[ "$str" =~ ^[0-9]+$ ]]; then
    echo "String contains only digits"
fi

Inequality (!=):

if [ "$str1" != "$str2" ]; then
    echo "Strings are not equal"
fi

Numeric Comparisons:

Greater than (-gt):

if [ $num1 -gt $num2 ]; then
    echo "$num1 is greater than $num2"
fi

Less than (-lt):

if [ $num1 -lt $num2 ]; then
    echo "$num1 is less than $num2"
fi

Greater than or equal to (-ge):

if [ $num1 -ge $num2 ]; then
    echo "$num1 is greater than or equal to $num2"
fi

Less than or equal to (-le):

if [ $num1 -le $num2 ]; then
    echo "$num1 is less than or equal to $num2"
fi

Equal to (-eq):

if [ $num1 -eq $num2 ]; then
    echo "$num1 is equal to $num2"
fi

Not equal to (-ne):

if [ $num1 -ne $num2 ]; then
    echo "$num1 is not equal to $num2"
fi

Checks if a file exits using (-f). If so then outputs the contents, if not then says "No file found".

if [ -f file.txt ]; then
    echo "File contents:"
    cat file.txt
else
    echo "No file found."
fi

Logical Operators

You can also use logical operators to combine multiple conditions.

AND (&&):

if [ $num -gt 0 ] && [ $num -lt 10 ]; then
    echo "The number is between 1 and 9"
fi

OR (||):

if [ $num -eq 0 ] || [ $num -eq 10 ]; then
    echo "The number is 0 or 10"
fi

String Comparisons

String comparisons in Bash can also be performed using various operators:

Check if a string is empty (-z):

if [ -z "$str" ]; then
    echo "String is empty"
fi

Check if a string is not empty (-n):

if [ -n "$str" ]; then
    echo "String is not empty"
fi

Check if a string starts with a specific substring:

if [[ "$str" == prefix* ]]; then
    echo "String starts with 'prefix'"
fi

Comparison Operators Table

Comparison Operator Example Description
Equality == [ "$str1" == "$str2" ] Strings are equal
Regex Match =~ [[ "$str" =~ ^[0-9]+$ ]] String matches regex
Inequality != [ "$str1" != "$str2" ] Strings are not equal
Numeric Greater -gt [ $num1 -gt $num2 ] num1 is greater than num2
Numeric Less -lt [ $num1 -lt $num2 ] num1 is less than num2
Numeric Greater or Equal -ge [ $num1 -ge $num2 ] num1 is greater than or equal to num2
Numeric Less or Equal -le [ $num1 -le $num2 ] num1 is less than or equal to num2
Numeric Equal -eq [ $num1 -eq $num2 ] num1 is equal to num2
Numeric Not Equal -ne [ $num1 -ne $num2 ] num1 is not equal to num2
String Empty -z [ -z "$str" ] String is empty
String Not Empty -n [ -n "$str" ] String is not empty
String Starts With == prefix* [[ "$str" == prefix* ]] String starts with 'prefix'
Checks if file exits -f [[ -f file.txt ]] Checks if file.txt exits
Logical AND && [ $num -gt 0 ] && [ $num -lt 10 ] Both conditions are true
Logical OR || [ $num -gt 10 ] || [ $num -lt 5 ] Only one condition needs to be true for this condtion to evaluate as true

Summary

Bash provides powerful constructs to handle conditional execution using if, if-else, and nested if-else statements. Various comparison operators like ==, =~, !=, -gt, -lt, and logical operators && and || help in forming complex conditions. Understanding these constructs and operators allows you to write more efficient and readable Bash scripts.

Chapter 12

Loops

In Bash scripting, loops are used to repeat a series of commands until a specific condition is met. There are several types of loops available in Bash: for, while, and until loops. These loops can also contain if-else statements to add conditional logic within the loop.

for Loop

A for loop iterates over a list of items and executes a block of code for each item.

Syntax:

for var in list; do
    # Commands
done

Example:

#!/bin/bash

for num in 1 2 3 4 5; do
    echo "Number: $num"
done

This script iterates over the numbers 1 to 5 and prints each number.

while Loop

A while loop executes a block of code as long as the specified condition is true.

Syntax:

while [ condition ]; do
    # Commands
done

Example:

#!/bin/bash

count=1

while [ $count -le 5 ]; do
    echo "Count: $count"
    count=$((count + 1))
done

This script prints the count from 1 to 5.

until Loop

An until loop is similar to a while loop but executes the block of code as long as the specified condition is false.

Syntax:

until [ condition ]; do
    # Commands
done

Example:

#!/bin/bash

count=1

until [ $count -gt 5 ]; do
    echo "Count: $count"
    count=$((count + 1))
done

This script prints the count from 1 to 5, stopping when the count is greater than 5.

Nesting if-else Statements in Loops

You can nest if-else statements within loops to add conditional logic to your loops.

Example with for Loop:

#!/bin/bash

for num in 1 2 3 4 5; do
    if [ $num -eq 3 ]; then
        echo "Found three!"
    else
        echo "Number: $num"
    fi
done

In this script, the for loop iterates over the numbers 1 to 5. When the number is equal to 3, it prints "Found three!" Otherwise, it prints the number.

Example with while Loop:

#!/bin/bash

count=1

while [ $count -le 5 ]; do
    if [ $count -eq 3 ]; then
        echo "Found three!"
    else
        echo "Count: $count"
    fi
    count=$((count + 1))
done

This script prints the count from 1 to 5. When the count is equal to 3, it prints "Found three!" Otherwise, it prints the count.

Example with until Loop:

#!/bin/bash

count=1

until [ $count -gt 5 ]; do
    if [ $count -eq 3 ]; then
        echo "Found three!"
    else
        echo "Count: $count"
    fi
    count=$((count + 1))
done

This script prints the count from 1 to 5, stopping when the count is greater than 5. When the count is equal to 3, it prints "Found three!" Otherwise, it prints the count.

Nested Loops

You can also nest loops within other loops to perform more complex iterations.

Example:

#!/bin/bash

for i in 1 2 3; do
    for j in a b c; do
        echo "Combination: $i$j"
    done
done

This script prints all combinations of numbers 1 to 3 with letters a to c.

Summary

Bash provides various looping constructs like for, while, and until loops to perform repetitive tasks efficiently. These loops can be enhanced with nested if-else statements to introduce conditional logic within the loops. Understanding how to use these loops and conditional statements allows you to write more complex and dynamic Bash scripts.

Arrays

Arrays in Bash scripting are a powerful way to store and manage multiple pieces of related data. Think of an array as a collection of items, like a list of favorite songs or the names of your friends. Each item in the array has a specific position or "index" that lets you access it quickly. In Bash, arrays are easy to create and work with, and they can store strings, numbers, or a combination of both.

To create an array in Bash, you simply assign values using parentheses. For example, let’s create an array of fruits:

fruits=("apple" "banana" "cherry" "date")

In this example, the array fruits contains four items. The first item is "apple," the second is "banana," and so on. In programming, arrays are zero-indexed, which means the first item is at position 0, the second at position 1, and so forth. So, if we want to access "banana," we would use index 1, like this:

echo ${fruits[1]}

This command will output "banana." You can access any element in an array by using the array name followed by the index in square brackets, with a dollar sign before it.

You can also change an item in the array by specifying its index and giving it a new value. Suppose we want to change "date" to "dragonfruit." We’d do it like this:

fruits[3]="dragonfruit"

Now, if we print the entire array, "date" is replaced with "dragonfruit." To print all the items in an array at once, use the following command:

echo ${fruits[@]}

This will print out all the items: "apple banana cherry dragonfruit." The [@] symbol lets Bash know we want to access all elements.

Sometimes, you’ll need to know the total number of items in an array. Bash provides an easy way to do this by using ${#arrayname[@]}. For example:

echo ${#fruits[@]}

This command will output 4, showing there are four items in the fruits array.

Finally, you might want to loop through each item in an array to process them one by one. You can do this with a for loop like so:

for fruit in "${fruits[@]}"
do
  echo "I love $fruit"
done

This loop goes through each item in the fruits array and prints out "I love apple," "I love banana," and so on. Loops like this are handy when you want to perform the same action on each item in an array.

Arrays in Bash make it easy to handle lists of related data, and as you practice, you’ll find they can help simplify and organize your scripts.

Chapter 13

Strings and Numbers

Computer programs are all about working with data. In past chapters, we have focused on processing data at the file level. However, many programming problems need to be solved using smaller units of data such as strings and numbers. In this chapter, we will look at several shell features used to manipulate strings and numbers. The shell provides a variety of parameter expansions that perform string operations.

Parameter Expansion

Though parameter expansion came up earlier in this course, we did not cover it in detail because most parameter expansions are used in scripts rather than on the command line. We have already worked with some forms of parameter expansion, such as shell variables. The shell provides many more. It's always good practice to enclose parameter expansions in double quotes to prevent unwanted word splitting, unless there is a specific reason not to. This is especially true when dealing with filenames, as they can often include embedded spaces and other assorted nastiness.

Basic Parameters

The simplest form of parameter expansion is reflected in the ordinary use of variables. For example, $a becomes whatever the variable a contains when expanded. Simple parameters may also be surrounded by braces, as in ${a}. This has no effect on the expansion but is required if the variable is adjacent to other text, which may confuse the shell. For example, attempting to create a filename by appending the string _file to the contents of the variable a will result in nothing because the shell will try to expand a variable named a_file rather than a. This problem can be solved by adding braces around the “real” variable name, as in ${a}_file.

Example:

a="foo"
echo "$a_file"      # Outputs nothing because it looks for a variable named a_file
echo "${a}_file"    # Outputs foo_file

Expansions to Manage Empty Variables

Several parameter expansions are intended to deal with nonexistent and empty variables. These expansions are handy for handling missing positional parameters and assigning default values to parameters.

Example:

foo=
echo ${foo:-"substitute value if unset"}  # Outputs "substitute value if unset"
foo=bar
echo ${foo:-"substitute value if unset"}  # Outputs "bar"

Example:

foo=
echo ${foo:="default value if unset"}  # Outputs "default value if unset" and assigns it to foo
echo $foo                             # Outputs "default value if unset"
foo=bar
echo ${foo:="default value if unset"}  # Outputs "bar"

Example:

foo=
echo ${foo:?"parameter is empty"}  # Outputs an error message and exits
foo=bar
echo ${foo:?"parameter is empty"}  # Outputs "bar"

Example:

foo=
echo ${foo:+"substitute value if set"}  # Outputs nothing
foo=bar
echo ${foo:+"substitute value if set"}  # Outputs "substitute value if set"

Expansions That Return Variable Names

The shell can return the names of variables in some situations. For example, ${!prefix*} and ${!prefix@} return the names of existing variables with names beginning with prefix. Both forms perform identically.

Example:

echo ${!BASH*}  # Outputs names of all variables starting with "BASH"

String Operations

There is a large set of expansions that can be used to operate on strings. Many of these expansions are particularly well suited for operations on pathnames.

Example:

foo="This string is long."
echo "'$foo' is ${#foo} characters long."  # Outputs "'This string is long.' is 20 characters long."

Example:

foo="This string is long."
echo ${foo:5}        # Outputs "string is long."
echo ${foo:5:6}      # Outputs "string"

Example:

foo=file.txt.zip
echo ${foo#*.}       # Outputs "txt.zip"
echo ${foo##*.}      # Outputs "zip"

Example:

foo=file.txt.zip
echo ${foo%.*}       # Outputs "file.txt"
echo ${foo%%.*}      # Outputs "file"

Example:

foo=JPG.JPG
echo ${foo/JPG/jpg}     # Outputs "jpg.JPG"
echo ${foo//JPG/jpg}    # Outputs "jpg.jpg"
echo ${foo/#JPG/jpg}    # Outputs "jpg.JPG"
echo ${foo/%JPG/jpg}    # Outputs "JPG.jpg"

Case Conversion

Bash supports the uppercase/lowercase conversion of strings with four parameter expansions and two declare command options. These conversions are useful for normalizing user input for database lookups, ensuring consistent data entry. For example, ${parameter,,} converts all characters in parameter to lowercase, ${parameter,} changes only the first character to lowercase, ${parameter^^} converts all characters to uppercase, and ${parameter^} changes only the first character to uppercase.

Example:

foo="aBc"
echo "${foo,,}"  # Outputs "abc"
echo "${foo,}"   # Outputs "aBc"
echo "${foo^^}"  # Outputs "ABC"
echo "${foo^}"   # Outputs "ABc"

Arithmetic Evaluation and Expansion

Arithmetic expansion allows performing various arithmetic operations on integers using the basic form $((expression)). The shell supports integer constants in different bases, such as decimal, octal, and hexadecimal. Unary operators, simple arithmetic operators, and more complex assignment operators provide convenient shorthand for arithmetic tasks. Notably, the increment (++) and decrement (--) operators, derived from the C programming language, increase or decrease the value of their parameters by one, with a subtle difference between prefix and postfix usage.

Example:

echo $((2 + 3))         # Outputs "5"
echo $((5 / 2))         # Outputs "2" (integer division)
echo $((5 % 2))         # Outputs "1" (modulo operation)
foo=1
echo $((foo++))         # Outputs "1", then foo becomes 2
echo $foo               # Outputs "2"
echo $((++foo))         # Outputs "3"

Bit Operations

Bit operators manipulate numbers at the bit level, often for setting or reading bit-flags. These include bitwise negation (~), left and right bitwise shifts (<< and >>), and bitwise AND, OR, and XOR (&, |, ^). Corresponding assignment operators, such as <<=, exist for all but bitwise negation.

Example:

echo $((1 << 2))  # Outputs "4" (1 shifted left by 2 bits)
echo $((4 >> 1))  # Outputs "2" (4 shifted right by 1 bit)

Logic and Comparison Operators

The (( )) compound command supports a variety of comparison operators, including less than (<), greater than (>), equal to (==), not equal to (!=), logical AND (&&), and logical OR (||). The ternary operator (expr1?expr2:expr3) performs a stand-alone logical test, acting as a compact if/then/else statement.

Example:

a=0
((a < 1 ? ++a : --a))
echo $a  # Outputs "1"
((a < 1 ? ++a : --a))
echo $a  # Outputs "0"

Practical Example

To demonstrate the efficiency of parameter expansion, we modified the longest-word script from the previous chapter to use ${#j} in place of `$(echo -n $j

| wc -c)`, resulting in a significant performance improvement. The original script took 3.618 seconds to scan the text file, while the new version using parameter expansion took only 0.06 seconds.

#!/bin/bash
# longest-word3: find longest string in a file
for i; do
  if [[ -r "$i" ]]; then
    max_word=
    max_len=0
    for j in $(strings $i); do
      len="${#j}"
      if (( len > max_len )); then
        max_len="$len"
        max_word="$j"
      fi
    done
    echo "$i: '$max_word' ($max_len characters)"
  fi
done

In conclusion, mastering these shell features enhances your ability to manipulate strings and numbers efficiently, improving the performance and readability of your scripts.

Troubleshooting Bash Scripts

As our scripts become more complex, it's crucial to address what happens when things go wrong. This chapter examines common errors in scripts and techniques to track down and eradicate these issues.

Syntactic Errors

One general class of errors is syntactic. These involve mistyping elements of shell syntax, causing the shell to stop executing the script. Here's an example script to demonstrate common types of errors:

#!/bin/bash
# trouble: script to demonstrate common errors
number=1
if [ $number = 1 ]; then
    echo "Number is equal to 1."
else
    echo "Number is not equal to 1."
fi

As written, this script runs successfully:

[me@linuxbox ~]$ trouble
Number is equal to 1.

Missing Quotes

Let's edit the script by removing the trailing quote from the argument following the first echo command:

#!/bin/bash
# trouble: script to demonstrate common errors
number=1
if [ $number = 1 ]; then
    echo "Number is equal to 1.
else
    echo "Number is not equal to 1."
fi

Running the script results in:

[me@linuxbox ~]$ trouble
/home/me/bin/trouble: line 10: unexpected EOF while looking for matching `"' /home/me/bin/trouble: line 13: syntax error: unexpected end of file

The line numbers reported by the error messages are not where the missing quote was removed but much later in the program. The shell continues looking for the closing quote until it finds one, immediately after the second echo command, confusing subsequent syntax. To avoid this, use an editor with syntax highlighting, like vim with the command :syntax on.

Missing or Unexpected Tokens

Another common mistake is forgetting to complete a compound command, such as if or while. For example, removing the semicolon after the test in the if command:

#!/bin/bash
# trouble: script to demonstrate common errors
number=1
if [ $number = 1 ] then
    echo "Number is equal to 1."
else
    echo "Number is not equal to 1."
fi

Running the script results in:

[me@linuxbox ~]$ trouble
/home/me/bin/trouble: line 9: syntax error near unexpected token `else'
/home/me/bin/trouble: line 9: `else'

The error occurs later than the actual problem. The if command's test list includes the word then due to the missing semicolon, causing syntax errors.

Unanticipated Expansions

Errors can occur intermittently due to the results of an expansion. For example, changing number to an empty variable:

#!/bin/bash
# trouble: script to demonstrate common errors
number=
if [ $number = 1 ]; then
    echo "Number is equal to 1."
else
    echo "Number is not equal to 1."
fi

Running the script results in:

[me@linuxbox ~]$ trouble
/home/me/bin/trouble: line 7: [: =: unary operator expected
Number is not equal to 1.

The test command fails due to the empty number variable, resulting in [ = 1 ], which is invalid. This can be corrected by adding quotes around the first argument in the test command:

[ "$number" = 1 ]

Then, the expansion results in [ "" = 1 ], which is valid.

Logical Errors

Logical errors do not prevent a script from running but cause it to produce incorrect results. Common types include:

  1. Incorrect conditional expressions: Misconfigured if/then/else logic.
  2. “Off by one” errors: Loop counters starting from incorrect values.
  3. Unanticipated situations: Unexpected data or expansions causing failures.

Defensive Programming

Verify assumptions by carefully evaluating the exit status of programs and commands. For example, consider this script:

cd $dir_name
rm *

If dir_name does not exist, the script will delete files in the current working directory. Improve it by quoting dir_name and checking if cd is successful:

cd "$dir_name" && rm *

Further, check that dir_name contains an existing directory:

[[ -d "$dir_name" ]] && cd "$dir_name" && rm *

Terminate the script if the directory does not exist:

# Delete files in directory $dir_name
if [[ ! -d "$dir_name" ]]; then
    echo "No such directory: '$dir_name'" >&2
    exit 1
fi

if ! cd "$dir_name"; then
    echo "Cannot cd to '$dir_name'" >&2
    exit 1
fi

if ! rm *; then
    echo "File deletion failed. Check results" >&2
    exit 1
fi

Watch Out for Filenames

Unix allows nearly any character in filenames, which can cause problems. For example, a filename starting with - can be interpreted as an option. Prevent this by using ./*:

rm ./*

Portable Filenames

To ensure portability, limit filenames to the POSIX Portable Filename Character Set: uppercase and lowercase letters, numerals, period, hyphen, and underscore.

Verifying Input

A program must handle any input it receives. Use specific tests to verify valid input. For example, to verify a menu selection:

[[ $REPLY =~ ^[0-3]$ ]]

This returns a zero exit status only if REPLY is a numeral between 0 and 3.

Design is a Function of Time

Design effort varies with the time available. Quick scripts for single use may need minimal checks, while production scripts require thorough development and testing.

Testing

Testing is crucial in software development. Use stubs to verify program flow and modify scripts to make tests safe. For example:

if [[ -d $dir_name ]]; then
    if cd $dir_name; then
        echo rm * # TESTING
    else
        echo "cannot cd to '$dir_name'" >&2
        exit 1
    fi
else
    echo "no such directory: '$dir_name'" >&2
    exit 1
fi
exit # TESTING

Test Cases

Develop good test cases to reflect edge and corner cases. For example, test the dir_name script with:

  1. dir_name contains an existing directory.
  2. dir_name contains a nonexistent directory.
  3. dir_name is empty.

Debugging

Debugging involves finding out what a script is actually doing. Isolate problem areas by commenting out sections of code:

if [[ -d $dir_name ]]; then
    if cd $dir_name; then
        rm *
    else
        echo "cannot cd to '$dir_name'" >&2
        exit 1
    fi
# else
# echo "no such directory: '$dir_name'" >&2
# exit 1
fi

Tracing

Tracing shows the program's flow. Add informative messages or use the -x option:

#!/bin/bash -x
# trouble: script to demonstrate common errors
number=1
if [ $number = 1 ]; then
    echo "Number is equal to 1."
else
    echo "Number is not equal to 1."
fi

Enable tracing for selected parts with set -x and set +x:

#!/bin/bash
# trouble: script to demonstrate common errors
number=1
set -x # Turn on tracing
if [ $number = 1 ]; then
    echo "Number is equal to 1."
else
    echo "Number is not equal to 1."
fi
set +x # Turn off tracing

Examining Values During Execution

Display variable contents during execution with echo statements:

#!/bin/bash
# trouble: script to demonstrate common errors
number=1
echo "number=$number" # DEBUG
set -x # Turn on tracing
if [ $number = 1 ]; then
    echo "Number is equal to 1."
else
    echo "Number is not equal to 1."
fi
set +x # Turn off tracing

Summing Up

This chapter covered common script problems and debugging techniques. Debugging is a skill developed through experience, involving constant testing and effective use of tracing to find and fix bugs.