wget, ftp and sftp Commands

Think of file transfers on the internet like delivering packages. The wget command is like a delivery service that picks up a package (file) for you with no questions asked - you just give them the address and they bring it back. The ftp command is like going to a warehouse yourself to pick up or drop off packages, but everyone can see what you're carrying. The sftp command is like that same warehouse visit, but with a private, secure tunnel that keeps your packages hidden from prying eyes. These commands help you transfer files between computers over the internet, with varying levels of automation and security.

The course setup.sh script creates ~/playground/chapter7/sftp_practice/ with sample files (important.txt, report.txt) for SFTP get/put practice and urls_to_download.txt listing several URLs for wget -i practice (batch downloads from one file). Run cd ~/playground/chapter7/sftp_practice before trying the wget and SFTP examples that use local files.

Quick Reference

When to Use These Commands

• When you need to download files from websites via command line
• When you need to transfer files between your computer and a remote server
• When you're working on servers without graphical interfaces
• When you want to automate file transfers
• When you need to securely move files across the internet
• When you're managing websites or remote systems

The wget Command

Think of wget as your personal delivery service on the internet. You give it an address (URL), and it goes and fetches whatever is at that location without you needing to interact with it further. It's perfect for downloading files from websites directly to your computer or server without using a browser.

# Download the college website index page (saves as index.html in current directory)
wget https://www.wccnet.edu/
# Or save with a specific name:
wget -O wcc_homepage.html https://www.wccnet.edu/

cd ~/playground/chapter7/sftp_practice
wget -i urls_to_download.txt
# -i (--input-file) tells wget to read URLs from urls_to_download.txt: one URL per line;
# lines starting with # are treated as comments. Each file is downloaded to this directory.

# Resume a download that was interrupted
wget -c https://www.wccnet.edu/

# Download quietly (no progress information)
wget -q https://www.wccnet.edu/

# Download with a limit on speed (be considerate when practicing)
wget --limit-rate=500k https://www.wccnet.edu/

Understanding wget Output

--2024-07-10 10:00:00--  https://www.wccnet.edu/
Resolving www.wccnet.edu (www.wccnet.edu)... 192.0.2.1
Connecting to www.wccnet.edu (www.wccnet.edu)|192.0.2.1|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: 'index.html'

index.html                 [  <=>                ]  45.2K  --.-KB/s    in 0.1s

2024-07-10 10:00:01 (450 KB/s) - 'index.html' saved [46280]

The ftp and sftp Commands

Think of ftp (File Transfer Protocol) as walking into a public storage facility where you can deposit or retrieve your files. Everyone can see what you're carrying in and out. In contrast, sftp (Secure File Transfer Protocol) is like that same facility but with a private, secured tunnel that keeps your files hidden from view. sftp uses SSH encryption to protect your data and login credentials, making it much safer for transferring sensitive files over the internet.

Due to security concerns, sftp should always be your first choice, as ftp transmits passwords and data in plaintext that can be intercepted. Think of it like the difference between shouting your credit card number across a crowded room versus whispering it directly to the cashier.

Common SFTP Interactive Commands

Once connected to an SFTP server, you'll see an sftp> prompt where you can type various commands. Think of this like being in the file storage facility with a walkie-talkie to your computer - you can give instructions to retrieve files from either location.

# From the course practice directory you have important.txt and report.txt to get/put
cd ~/playground/chapter7/sftp_practice

# Connect to an SFTP server
sftp user@example.com
# Prompts for password, then connects to the server

# Connect with a specific port
sftp -P 2222 user@example.com
# Connects using port 2222 instead of the default port 22

# Navigate through directories
sftp> cd /var/www/html
sftp> lcd ~/Downloads
# Changes to /var/www/html on server and ~/Downloads locally

# Download a file
sftp> get important.txt
# Downloads important.txt from server to local computer

# Upload a file
sftp> put report.pdf
# Uploads report.pdf from local computer to server

# Download multiple files
sftp> get *.txt
# Downloads all .txt files from current remote directory

# Download an entire directory
sftp> get -r projects
# Downloads the projects directory and all its contents

# Upload an entire directory
sftp> put -r website
# Uploads the website directory and all its contents

Understanding SFTP Session Output

$ sftp user@example.com
Connecting to example.com...
user@example.com's password: 
sftp> ls
documents  images  index.html  styles.css
sftp> cd images
sftp> ls
logo.png  background.jpg  icon.svg
sftp> lcd ~/Downloads
sftp> get logo.png
Fetching /home/user/images/logo.png to logo.png
/home/user/images/logo.png                   100%   24KB  350.5KB/s   00:00
sftp> put new-image.jpg
Uploading new-image.jpg to /home/user/images/new-image.jpg
new-image.jpg                                100%   58KB  425.2KB/s   00:00
sftp> bye
$

Tips for Success

• Always use sftp instead of ftp whenever possible for security
• Use wget -c to resume interrupted downloads, especially for large files
• Before uploading files with put, check available space with df -h command
• Use tab completion in SFTP to avoid typing long filenames
• Use get -r and put -r for transferring entire directories
• When using wget to download large websites, be considerate and use --limit-rate
• Create scripts with wget commands for tasks you perform regularly
• Test your SFTP connection with -v (verbose) flag if having trouble connecting

Common Mistakes to Avoid

• Using ftp instead of sftp for sensitive data (passwords and files can be intercepted)
• Forgetting which directory you're in (use pwd for remote and lpwd for local)
• Downloading files to the wrong location (check with lls before downloading)
• Using wget -r without limits, which could download an entire website
• Ignoring error messages during file transfers
• Not using bye or exit to properly close connections
• Trying to transfer binary files in ASCII mode (in traditional FTP)
• Forgetting to use quotes for filenames with spaces in SFTP commands

Best Practices

• Use SSH keys instead of passwords for more secure SFTP connections
• Create bookmarks or aliases for frequently used SFTP connections
• Use wget's --spider option to check if links are valid before downloading
• Create a batch file with common SFTP commands for automation
• Always verify file integrity after transfers (especially for critical files)
• Use compression (-C option) when transferring over slow connections
• Set up proper file permissions after uploading files to a server
• Document your file transfer processes for team use or future reference