CIS120 Linux Fundamentals by Scott Shaper

Linux Pipelines

Think of Linux pipelines like a factory assembly line. Just as raw materials move from one station to another, getting transformed at each step, data flows through a series of commands, getting processed and transformed along the way. The pipe symbol (|) is like the conveyor belt that moves the data from one command to the next.

When to Use Pipelines

Use pipelines when you want to:

Basic Pipeline Concepts

A pipeline connects commands using the pipe symbol (|). The output of the command on the left becomes the input for the command on the right. Think of it like a chain of commands, where each link processes the data in some way.

Common Pipeline Patterns

Pattern What It Does When to Use It
command1 | command2 Basic two-command pipeline Simple data processing
command1 | command2 | command3 Three-command pipeline Complex data processing
command1 | tee file.txt Save and display output When you need both output and a file

Practical Examples

First, create practice files for the pipeline examples:

Create text files with "error" in their names:

cat > error_log.txt << 'EOF'
Error log file content
EOF
cat > error_report.txt << 'EOF'
Error report content
EOF

Create regular text files:

cat > normal_file.txt << 'EOF'
Normal file content
EOF

Create Python files:

cat > script1.py << 'EOF'
print("Hello from script 1")
EOF
cat > script2.py << 'EOF'
print("Hello from script 2")
EOF
cat > script3.py << 'EOF'
print("Hello from script 3")
EOF

Create a text file for text processing examples:

cat > file.txt << 'EOF'
This is a sample file.
It contains multiple lines of text.
We can use it for pipeline examples.
Each line has different content.
EOF

Create a passwd-like file for cut examples:

cat > users.txt << 'EOF'
alice:x:1001:1001:Alice User:/home/alice:/bin/bash
bob:x:1002:1002:Bob User:/home/bob:/bin/bash
charlie:x:1003:1003:Charlie User:/home/charlie:/bin/sh
diana:x:1004:1004:Diana User:/home/diana:/bin/bash
EOF

Create an access log file with IP addresses:

cat > access.log << 'EOF'
IP: 192.168.1.100 Request successful
IP: 192.168.1.101 Request successful
IP: 192.168.1.100 Request failed
IP: 192.168.1.102 Request successful
IP: 192.168.1.100 Request successful
IP: 192.168.1.103 Request successful
EOF

Create log files for advanced examples:

cat > app.log << 'EOF'
2024-01-01 10:00:00 error: Connection failed
2024-01-01 10:05:23 System started
2024-01-01 10:10:45 error: File not found
2024-01-01 10:15:12 Operation completed
EOF
cat > server.log << 'EOF'
2024-01-01 10:20:30 error: Timeout occurred
2024-01-01 10:25:55 Request processed
2024-01-01 10:30:18 error: Connection failed
EOF

Create a CSV data file:

cat > data.csv << 'EOF'
Alice,Engineer,50000
Bob,Manager,75000
Charlie,Engineer,55000
Diana,Designer,60000
Alice,Engineer,52000
Bob,Manager,76000
Charlie,Engineer,56000
EOF

File Operations

# Find all text files containing "error"
ls *.txt | grep "error"

# Count how many Python files are in a directory
ls *.py | wc -l

# Sort files by size
ls -l | sort -k5 -n

Text Processing

# Convert text to uppercase and count lines
cat file.txt | tr '[:lower:]' '[:upper:]' | wc -l

# Extract usernames from users.txt (similar to /etc/passwd format)
cat users.txt | cut -d: -f1 | sort

# Find unique IP addresses in a log file
grep "IP:" access.log | cut -d' ' -f2 | sort -u

Tips for Success

Common Mistakes to Avoid

Best Practices