Complete tutorial on the wget command in Linux: a practical guide

If you use Linux daily, sooner or later you're going to need download files from the terminal quickly and flexiblyThat's where wget comes in, a little gem from the GNU project that has been quietly doing the dirty work for decades.

With just a few options, wget lets you Download a single file, hundreds of files, entire websites, content via FTPLimit bandwidth, resume interrupted downloads, automate tasks with cron, and much more. All without windows or wizards: just you, the console, and one very powerful command.

What exactly is the wget command in Linux?

The wget command starts the wget command-line program from the GNU projectDesigned to download content from the network using different protocols. Its name comes from "World Wide Web" + "get", meaning "to obtain from the web".

This tool is developed in C portable and distributed as free softwareso it works seamlessly on virtually any type of system Unix: GNU/Linux, BSD, and macOS distributions using additional tools, and even in Windows through environments such as Cygwin or the built-in Ubuntu Bash console.

With wget you can Retrieve files via HTTP, HTTPS, and FTPand even use it with authentication when a server requires a username and password, for both HTTP and FTP. Furthermore, it understands redirects, TLS certificates, and typical web server responses.

One of its greatest virtues is that the downloads are non-interactivewget doesn't require you to have an open session at the computer. You can launch it from a script, leave a cron job running on a remote server or use it in the background without worrying about the session SSH It cuts out.

In most popular distributions such as Debian, Ubuntu or Linux Mint wget is installed by defaultIn others, you just need to pull the corresponding package manager and you'll have it up and running in a minute.

How to install wget on Linux and other systems

In a typical Linux environment, a simple command is all that's needed to Install or check that wget is availableFor example, in Ubuntu or Debian you can do something as straightforward as:

Quick Install: sudo apt-get install wget

After completing the installation, you will have the binary file. wget ready in your PATHYou can verify this with:

Check version: wget --version

In other distributions the package manager will change, but the idea is the same: yum, dnf, pacman or zypper They allow you to install wget from the official repositories with a single command.

On macOS, the system doesn't include wget by default, but you can use it. Homebrew to install the console toolOnce you have brew configured, simply:

Homebrew: brew install wget

In the traditional Windows ecosystem, there isn't such direct access to wget from the symbol of the system classic, although Cygwin and the Ubuntu Bash console on Windows 10/11 They include wget and other GNU utilities. If you work with the Linux layer of Windows, you'll use it practically the same as in any other distribution.

Basic syntax and most important options of wget

The overall structure of the command is very simple: wget followed by options and one or more URLsThe canonical form would be something like this:

Format: wget [opciones] URL1 URL2 URL3 ...

If you don't add anything else, wget will download the file specified by the URL and... It will automatically save to the current working directory. using the same name it has on the remote server.

The true power of the command comes from the vast number of options available. Some of the most commonly used in everyday use are:

• -co –continue: resumes a download that was interrupted, without starting from scratch.
• -N: only download the file if the remote copy is newer than the one you already have on disk.
• -t [number]: indicates how many attempts wget should make when downloading a file before giving up.
• -w [seconds]: marks a wait between consecutive downloads to avoid overloading the server.
• -r: activates recursive mode to follow links or traverse directories.
• -P [route]Choose the directory where you want the downloaded files to be saved.
• –limit-rate=[speed]: limits the bandwidth consumed by wget (for example, 200k).

These options can be combined to adapt wget's behavior exactly to what you needfrom a simple one-off download to full-fledged website crawlers or regular backup tasks.

Basic examples: downloading one or more files

The most basic use of wget is to download a single file to the folder you are inFor example, to download the ISO image of an Ubuntu distribution, you could run something like:

Quick example: wget http://releases.ubuntu.com/14.04/ubuntu-14.04-desktop-amd64.iso

During the download you will see on screen the percentage completed, the total size, the speed, and There employeeThis includes DNS resolution and connecting to the server. Once finished, you'll have the file in your current directory with the same name as in the link.

If instead of a file you want Download multiple files at once by specifying all URLs in a single commandIt is enough to chain together links separated by spaces, even mixing HTTP and FTP protocols:

Multiple files: wget http://ejemplo.com/programa.tar.gz ftp://otrositio.com/descargas/video.mpg

Furthermore, wget understands certain expansion patterns in the URLFor example, if you have a collection of images numbered from 1 to 50 on a server, you could do the following:

Expansion pattern: wget http://example.com/images/{1..50}.jpg

Another very practical way to manage multiple downloads is to Prepare a text file with a URL on each line and then tell wget to read that list. For example:

Create list: nano archivos.txt

Within files.txt you write something like:

http://ejemplo.com/programa.tar.gz
http://ejemplo.com/rpm/paquete.rpm
ftp://otrositio.com/descargas/distro.iso

Then you just need to run:

Download from list: wget -i archivos.txt

With option -iwget will read the text file and download each of the listed addresses, perfect for leaving your computer working in the background while you do something else.

Change folder name and limit download speed

When you are not interested in preserving the original name of the remote file, you can use the option -Or to rename the resultFor example, if you want to save the previous ISO with a shorter name:

Rename: wget -O ubuntu_amd64.iso http://releases.ubuntu.com/14.04/ubuntu-14.04-desktop-amd64.iso

This same trick is very useful when you always want download something with a fixed name to overwrite it in scripts, such as a backup diary.

If what you want is to control the destination directory where the downloaded files are savedYou can use the -P option followed by the path:

Destination directory: wget -P documentos/isos/ http://releases.ubuntu.com/14.04/ubuntu-14.04-desktop-amd64.iso

In this way, the file will end up in documents/isos/ instead of the current directorywhich is great for keeping downloads organized without having to make any additional moves.

Another key aspect is bandwidth consumption. Wget allows Limit the speed to avoid monopolizing the entire connection.This is especially relevant on servers or shared connections. For example, to set a maximum of 500 KB/s:

Limit speed: wget --limit-rate=500k https://wordpress.org/latest.zip

You can also specify a value like 50K, 2M, etc., adjusting the rate to your needsThis way you can continue browsing or using other services while large files are being downloaded in the background.

Resume downloads, control retries, and work in the background

One of the typical cases is that you cut the connection or the power goes out in the middle of a large dischargeInstead of losing all downloaded data, wget allows you to resume the file where it left off with the -c option:

Resume downloads: wget -c http://releases.ubuntu.com/14.04/ubuntu-14.04-desktop-amd64.iso

While the server supports resuming partial downloads, wget will continue from the point where it stoppedIf you don't use -cy and the file already exists, the program will save it with a ".1" suffix to avoid overwriting the original file.

In environments with unstable connections or heavily overloaded servers, it is also important to monitor the number of download attemptsBy default, wget makes 20 attempts, but you can increase or decrease that value with -t:

Attempt control: wget -t 50 http://ejemplo.com/pelicula.mpg

If you wish, you can even use -t inf so that wget keeps trying indefinitely until it manages to connect, something that can be useful in scripts where it is a priority not to fail.

When dealing with huge files, it's best to run the download in the background. to be able to close the terminal or continue using that sessionWget offers the -b option for this:

In the background: wget -b http://example.com/beefy-file.tar.gz

By using -b, the program "detaches" from the terminal and It records its progress in a file called wget-log within the current directory. You can see how it progresses with:

View progress: tail -f wget-log

If you also want a detailed report of the entire download process—ideal for debugging problems—you can redirect the output of wget to a log file using the option -o:

Detailed record: wget -o reporte.txt http://ejemplo.com/programa.tar.gz

Working with protected servers and FTP using wget

Many online resources are not public and require a username and password to download filesWget knows how to handle these environments by adding the credentials on the command line.

For example, on an HTTP server with basic authentication, you could use something like:

HTTP authenticated: wget --http-user=admin --http-password=secreto http://ejemplo.com/archivo.mp3

Similarly, the command works perfectly with traditional FTP serversTo download a specific file from a private FTP server, simply:

FTP with credentials: wget --ftp-user=TU_USUARIO --ftp-password=TU_PASSWORD ftp://example.com/something.tar

If what you want is to take a complete directory from an FTP serverYou can combine recursive mode with FTP routes:

Download FTP directory: wget -r ftp://example.com/folder

If you add the option –no-parent You ensure that wget does not go up to higher directories in the FTP tree and focuses only on the specified folder and its subdirectories.

Download entire websites to view them offline

One of wget's most powerful features is its ability to download the content of an entire website and be able to view it locally No internet connection. This can be incredibly useful for long trips, environments without network access, or offline website analysis.

Advanced example: wget --mirror --convert-links --page-requisites --no-parent -P documentos/websites/ https://some-website.com

Here, each option contributes something very specific to the website cloning process:

• –mirror: activates a mode designed for mirrors, which combines deep recursion with timestamp management.
• –convert-links: modifies internal links to point to downloaded files, allowing offline browsing.
• –Page-requisites: includes all the resources needed for the page to look good (CSS, JavaScript, images, etc.).
• –no-parent: avoid uploading to higher directories, limiting the download to the scope of the site you are interested in.
• -P documents/websites/: indicates the directory where the entire downloaded structure will be saved.

Once the download is complete, you will be able to Open the main index with your favorite browser and navigate the copy as if you were online. The files will be organized within the documents/websites/ folder with subdirectories that replicate the structure of the remote server.

For certain sites, it is also useful to use –html-extension or the abbreviation -EThis will ensure that all documents are saved with the .html extension, even if they were originally .php, .asp, or .cgi, making it easier to open them later in local browsers.

Locate broken links and use wget as a web “spider”

Beyond downloading content, wget can be used as a kind of Lightweight crawler to assess the status of a site's linksThanks to spider mode, the program crawls the pages without saving them, simply checking what the server responds to.

Spider mode: wget -o wget-log -r -l 5 --spider http://example.com

In this line, each parameter fulfills its role:

• -o wget-log: sends all output to the wget-log file for later review at your leisure.
• -r: activates recursion to follow internal links.
• -l 5: defines the maximum recursion depth in 5 levels.
• –spider: turns wget into a spider that checks links but doesn't download the files.

Later you can process the log file to quickly extract links that return 404A typical example of a command to filter them would be:

Filter 404s: grep -B 2 '404' wget-log | grep "http" | cut -d " " -f 4 | sort -u

This way you'll get a clean list of problematic URLs, ideal for website administrators to fix redirects or deleted content.

Recover a complete website with advanced options

There are scenarios where you want Download a website with much more control over what wget brings up.For example, for detailed analysis, content audits, or long-term offline queries.

Advanced download: wget -m -F -p -np -k -erobots=off -U mozilla --limit-rate=50K --wait=2 --html-extension sitioweb

This line combines several interesting parameters:

• -m: mirror mode, equivalent to certain combinations of recursion and timestamps.
• -F: forces the download even if there are small errors in the answers.
• -p: ensures that all auxiliary resources for each page are downloaded.
• -np: no parent, prevents moving up to higher directories during recursion.
• -k: converts the links to point to the local copy.
• -erobots=off: ignores the site's robots.txt file and does not respect its restrictions.
• -U mozilla: sets a user agent string similar to a Mozilla browser.
• –limit-rate=50K: limits the speed to 50 KB/s to avoid overloading the line.
• –wait=2: introduces a 2-second pause between consecutive requests.
• –html-extension: forces the .html extension on all downloaded documents.

The result is a very complete and navigable copy of the site, designed for offline consultation and detailed analysisHowever, it's advisable to keep an eye on disk space, because this type of operation can fill up gigabytes without you realizing it, especially if recursion is enabled on a large site.

Extreme examples: filtering file types and bypassing robots

Wget also lends itself to more “extreme” uses, such as search only for certain types of files in a long list of sitesA typical example is downloading files mp3 from addresses stored in a text file:

Filter mp3: wget -r -l1 -H -t1 -nd -N -np -A.mp3 -erobots=off -i sitiosmp3.txt

The behavior of each option in this combination is very illustrative of wget's potential:

• -r -l1: recursion enabled but limited to a single level of subdirectories.
• -H: allows following links to domains other than the main one (span hosts).
• -t1: only one connection attempt per URL; if it fails, it moves on to the next one.
• -ndIt does not create local subdirectories; it saves all files in the same place.
• -N: respects the timestamp of the original file and only downloads if there are changes.
• -np: avoids uploading to parent directories.
• -A.mp3: limits downloads to files with the .mp3 extension.
• -erobots=off: ignores robots.txt restrictions to explore all directories.
• -i sitiosmp3.txt: reads the initial list of sites to crawl from the file sitiosmp3.txt.

Commands like these allow mounting automated processes that run daily using cron, downloading only new files thanks to the -N option and keeping a local repository always up to date.

Additional useful options and security best practices

In addition to all of the above, wget incorporates other very practical parameters for controlling its behaviorSome of the most interesting ones are:

• -b: runs the download in the background, ideal for scripts or remote sessions.
• -t 10: defines a specific number of retries in case of temporary errors.
• –limit-rate=200k: adjusts the maximum bandwidth per download to a specific amount.
• -pWhen you download HTML pages, also drag images, CSS, and other resources.
• –convert-links: transforms absolute links into relative links for offline work.

However, it's advisable to use wget with a minimum of care to avoid causing problems related to either safety or resource consumptionSome basic recommendations are:

• Always check the URL before downloading anything, especially if you run the command as rootto avoid going down malware or unwanted content.
• Understand the options you use, especially recursion (-r, -mirror, -H, -np) to avoid downloading more data than you intended or overloading a server.
• Control recursion To avoid bringing up huge sites; adjusting -ly -np helps limit the scope of the crawl.
• Respect robots.txt whenever possibleUnless you have explicit permission to ignore it, it marks areas of the website that their owners prefer not to expose to spiders and downloaders.
• Speed ​​limits with –limit-rate They are highly recommended on shared networks so as not to bother other users.
• Do not disable HTTPS validation Unless you really know what you're doing, certificates are checked by default and are an important layer of security.

This entire set of features makes wget A Swiss Army knife for downloading content on LinuxFrom downloading a simple file with a short command to cloning entire sites, filtering specific file types, resuming interrupted downloads, working with FTP, or generating broken link reports, always from the terminal and with options that can be easily automated in scripts and cron jobs.