CIS120 Book

CIS120 Linux Fundamentals by Scott Shaper

wget, ftp and sftp Commands

Think of file transfers on the internet like delivering packages. The wget command is like a delivery service that picks up a package (file) for you with no questions asked - you just give them the address and they bring it back. The ftp command is like going to a warehouse yourself to pick up or drop off packages, but everyone can see what you're carrying. The sftp command is like that same warehouse visit, but with a private, secure tunnel that keeps your packages hidden from prying eyes. These commands help you transfer files between computers over the internet, with varying levels of automation and security.

Quick Reference

Command	What It Does	Common Use
`wget URL`	Downloads files from the web	Getting files from websites without a browser
`wget -c URL`	Resumes interrupted downloads	Continuing large downloads after connection issues
`ftp hostname`	Connects to FTP server (insecure)	Legacy systems that don't support SFTP
`sftp user@hostname`	Connects to SFTP server (secure)	Securely transferring files between computers
`get file`	Downloads a file (in FTP/SFTP)	Retrieving files from a remote server
`put file`	Uploads a file (in FTP/SFTP)	Sending files to a remote server

When to Use These Commands

When you need to download files from websites via command line
When you need to transfer files between your computer and a remote server
When you're working on servers without graphical interfaces
When you want to automate file transfers
When you need to securely move files across the internet
When you're managing websites or remote systems

The wget Command

Think of wget as your personal delivery service on the internet. You give it an address (URL), and it goes and fetches whatever is at that location without you needing to interact with it further. It's perfect for downloading files from websites directly to your computer or server without using a browser.

Option	What It Does	When to Use
`-O filename`	Saves the download with a different name	When you want to rename the file as it downloads
`-c`	Resumes a partially downloaded file	When your download was interrupted and you want to continue
`-q`	Quiet mode (no output)	When running in scripts where you don't need progress info
`-r`	Downloads recursively (follows links)	When you want to download an entire directory or website
`-l depth`	Limits the recursion depth	When downloading recursively but want to limit how deep it goes
`--limit-rate=rate`	Limits download speed	When you need to avoid consuming all available bandwidth

Practical Examples

# Download a single file
wget https://example.com/file.txt
# Downloads file.txt to current directory

# Resume a download that was interrupted
wget -c https://example.com/large-file.iso
# Continues downloading from where it left off

# Download and save with a different name
wget -O myfile.pdf https://example.com/document.pdf
# Saves as myfile.pdf instead of document.pdf

# Download quietly (no progress information)
wget -q https://example.com/file.txt
# No output during download

# Download an entire website (be careful!)
wget -r -l 2 https://example.com/docs/
# Downloads docs directory and up to 2 levels of subdirectories

# Limit download speed to 500KB/s
wget --limit-rate=500k https://example.com/large-file.iso
# Prevents the download from using all available bandwidth

Understanding wget Output

Sample Output Explained

--2024-07-10 10:00:00--  http://example.com/file.txt
Resolving example.com (example.com)... 93.184.216.34
Connecting to example.com (example.com)|93.184.216.34|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1234 (1.2K) [text/plain]
Saving to: 'file.txt'

file.txt          100%[===================>]   1.21K  --.-KB/s    in 0s      

2024-07-10 10:00:01 (234 MB/s) - 'file.txt' saved [1234/1234]

Let's break down what this means:

--2024-07-10 10:00:00-- http://example.com/file.txt: Shows the time and URL being downloaded
Resolving example.com (example.com)... 93.184.216.34: Converting the domain name to an IP address
Connecting to example.com...: Establishing a connection to the server
HTTP request sent, awaiting response... 200 OK: Server responded with status 200 (success)
Length: 1234 (1.2K) [text/plain]: The file is 1,234 bytes (1.2 KB) and it's a plain text file
Saving to: 'file.txt': Where the file is being saved
The progress bar shows download completion percentage
2024-07-10 10:00:01 (234 MB/s) - 'file.txt' saved [1234/1234]: Shows completion time, average speed, and confirms all bytes were downloaded

The ftp and sftp Commands

Think of ftp (File Transfer Protocol) as walking into a public storage facility where you can deposit or retrieve your files. Everyone can see what you're carrying in and out. In contrast, sftp (Secure File Transfer Protocol) is like that same facility but with a private, secured tunnel that keeps your files hidden from view. sftp uses SSH encryption to protect your data and login credentials, making it much safer for transferring sensitive files over the internet.

Due to security concerns, sftp should always be your first choice, as ftp transmits passwords and data in plaintext that can be intercepted. Think of it like the difference between shouting your credit card number across a crowded room versus whispering it directly to the cashier.

Option	What It Does	When to Use
`-b batchfile`	Runs commands from a file	When automating multiple file transfers
`-C`	Enables compression	When transferring over slow connections to save bandwidth
`-P port`	Specifies a different port	When the server uses a non-standard port
`-i identity_file`	Uses a private key file	When using key-based authentication instead of a password
`-v`	Verbose mode (shows more details)	When troubleshooting connection problems

Common SFTP Interactive Commands

Once connected to an SFTP server, you'll see an sftp> prompt where you can type various commands. Think of this like being in the file storage facility with a walkie-talkie to your computer - you can give instructions to retrieve files from either location.

Command	What It Does	When to Use
`ls`	Lists files on the remote server	When you need to see what files are available on the server
`lls`	Lists files on your local computer	When you need to check your local files without leaving SFTP
`cd directory`	Changes directory on the remote server	When you need to navigate to different folders on the server
`lcd directory`	Changes directory on your local computer	When you need to change where files will be saved locally
`get file`	Downloads a file from server to your computer	When you need to retrieve a file from the server
`get -r directory`	Downloads an entire directory	When you need to download multiple files at once
`put file`	Uploads a file from your computer to server	When you need to send a file to the server
`put -r directory`	Uploads an entire directory	When you need to upload multiple files at once
`mkdir directory`	Creates a new directory on the server	When you need to organize files on the server
`rm file`	Deletes a file on the server	When you need to remove a file from the server
`exit` or `bye`	Closes the SFTP connection	When you're finished transferring files

Practical Examples for SFTP

# Connect to an SFTP server
sftp user@example.com
# Prompts for password, then connects to the server

# Connect with a specific port
sftp -P 2222 user@example.com
# Connects using port 2222 instead of the default port 22

# Navigate through directories
sftp> cd /var/www/html
sftp> lcd ~/Downloads
# Changes to /var/www/html on server and ~/Downloads locally

# Download a file
sftp> get important.txt
# Downloads important.txt from server to local computer

# Upload a file
sftp> put report.pdf
# Uploads report.pdf from local computer to server

# Download multiple files
sftp> get *.txt
# Downloads all .txt files from current remote directory

# Download an entire directory
sftp> get -r projects
# Downloads the projects directory and all its contents

# Upload an entire directory
sftp> put -r website
# Uploads the website directory and all its contents

Understanding SFTP Session Output

Sample Session Explained

$ sftp user@example.com
Connecting to example.com...
user@example.com's password: 
sftp> ls
documents  images  index.html  styles.css
sftp> cd images
sftp> ls
logo.png  background.jpg  icon.svg
sftp> lcd ~/Downloads
sftp> get logo.png
Fetching /home/user/images/logo.png to logo.png
/home/user/images/logo.png                   100%   24KB  350.5KB/s   00:00
sftp> put new-image.jpg
Uploading new-image.jpg to /home/user/images/new-image.jpg
new-image.jpg                                100%   58KB  425.2KB/s   00:00
sftp> bye
$

Let's break down what happened:

$ sftp user@example.com: Started the SFTP client and tried to connect to example.com as "user"
user@example.com's password:: Prompted for the user's password (not shown when typing)
sftp> ls: Listed files in the current remote directory
sftp> cd images: Changed to the "images" directory on the server
sftp> lcd ~/Downloads: Changed to the Downloads directory on local computer
sftp> get logo.png: Downloaded logo.png from server to local Downloads folder
The system showed progress of the download (24KB at 350.5KB/s)
sftp> put new-image.jpg: Uploaded new-image.jpg from Downloads to server's images folder
The system showed progress of the upload (58KB at 425.2KB/s)
sftp> bye: Closed the SFTP connection

Tips for Success

Always use sftp instead of ftp whenever possible for security
Use wget -c to resume interrupted downloads, especially for large files
Before uploading files with put, check available space with df -h command
Use tab completion in SFTP to avoid typing long filenames
Use get -r and put -r for transferring entire directories
When using wget to download large websites, be considerate and use --limit-rate
Create scripts with wget commands for tasks you perform regularly
Test your SFTP connection with -v (verbose) flag if having trouble connecting

Common Mistakes to Avoid

Using ftp instead of sftp for sensitive data (passwords and files can be intercepted)
Forgetting which directory you're in (use pwd for remote and lpwd for local)
Downloading files to the wrong location (check with lls before downloading)
Using wget -r without limits, which could download an entire website
Ignoring error messages during file transfers
Not using bye or exit to properly close connections
Trying to transfer binary files in ASCII mode (in traditional FTP)
Forgetting to use quotes for filenames with spaces in SFTP commands

Best Practices

Use SSH keys instead of passwords for more secure SFTP connections
Create bookmarks or aliases for frequently used SFTP connections
Use wget's --spider option to check if links are valid before downloading
Create a batch file with common SFTP commands for automation
Always verify file integrity after transfers (especially for critical files)
Use compression (-C option) when transferring over slow connections
Set up proper file permissions after uploading files to a server
Document your file transfer processes for team use or future reference

Real-World Scenarios

Website Maintenance

# Scenario: Updating a website's files

# 1. Connect to the web server
sftp user@webserver.com

# 2. Navigate to the website directory
sftp> cd /var/www/mywebsite

# 3. Check what files are there
sftp> ls
index.html  styles.css  images/  scripts/

# 4. Upload the updated files
sftp> put updated-index.html index.html
sftp> put new-style.css styles.css

# 5. Add new images
sftp> cd images
sftp> put -r ~/project/new-images/
sftp> ls
banner.jpg  logo.png  new-images/

# 6. Verify the changes
sftp> exit

# 7. Test the website in browser
# (Open web browser and navigate to your website)

Batch Downloading

# Scenario: Downloading a dataset for analysis

# Create a download script
echo "https://data.example.com/dataset1.csv
https://data.example.com/dataset2.csv
https://data.example.com/dataset3.csv" > downloads.txt

# Download all files listed in the text file
wget -i downloads.txt

# Resume any interrupted downloads
wget -c -i downloads.txt

# Download with rate limiting during work hours
wget --limit-rate=500k -i downloads.txt