Server Howto: In-house mirror
If you wish to set-up an in-house PXE / UEFI boot solution, it is rather handy to set-up an in-house mirror of your favourite Linux distribution(s) first. I would recommend an in-house mirror if you are a small to large business with Linux running on both servers and desktops. You can modify yum/dnf/apt-get sources to point at your in-house mirror to minimise bandwidth usage and to save your in-house IT estate from hammering the public mirrors.
Before you get carried away, be aware that the server you use for your in-house mirror will need a large amount of storage space. A server with at least 2.0 TiB of storage could easily be required. If you have a large IT estate, connecting this server to your network with 10 GbE is a good idea!
The scripts in this howto make extensive use of the rsync application. Check and install it if your server does not already have it installed. You may also want to install httpd and vsftpd if you want to offer FTP and HTTP access to the in-house mirror.
Mirror
I am assuming you have a server with plenty of storage space, with the largest partition assigned to / (root). Go ahead and create a mirror directory with: mkdir /mirror.
Now we need to populate that mirror directory. For that, I offer a number of script examples you can use and modify to suit your needs. As with my other howtos, my scripts live in /root/scripts. In order to control when the scripts run, I have a custom cron.d control script. More on that later.
Rocky Linux
Here is a simple rsync script to mirror Rocky Linux from an Internet-based rsync source. Check the mirror lists of your distro choice to find the nearest rsync source to you.
| #!/bin/bash # # Script to mirror Rocky Linux # Options # #OPT=vrhltDP OPT=vrhltD rsync -$OPT --delete-after --no-motd --exclude="aarch64" \ rsync://mirrors.vinters.com/rocky/ /mirror/rocky/ unset OPT exit 0 |
The script is called rocky-mirror and has 755 permissions to allow it to run. The two OPT lines can be swapped if you plan to run the script manually for its first run so you can ensure it is working correctly. The P option shows the progress of each file. This is not included by default as this file is called via cron, and the email generated after would be megabytes long if you leave the P option set.
CentOS Linux
Here is a simple rsync script to mirror CentOS Linux from an Internet-based rsync source. Check the mirror lists of your distro choice to find the nearest rsync source to you.
| #!/bin/bash # Options # #OPT=vrhltDP OPT=vrhltD rsync -$OPT --delete-after --no-motd --delete-excluded --exclude="/8*" \ --exclude="/6*" --exclude="/5*" --exclude="/4*" --exclude="/3*" --exclude="/2*" \ --exclude="/.8*" \ rsync://rsync.mirrorservice.org/mirror.centos.org/ /mirror/mirror.centos.org/ unset OPT exit 0 |
The above script is called centos-mirror and has 755 permissions. This script is more restrictive as I am only downloading CentOS 7 to my mirror.
Fedora
Here is a simple rsync script to mirror Fedora Linux from an Internet-based rsync source. Check the mirror lists of your distro choice to find the nearest rsync source to you.
| #!/bin/bash # Options # OPT=vrhltD VER=33 # Release # rsync -$OPT --delete --no-motd --exclude="aarch64" --exclude="armhfp" \ rsync://rsync.mirrorservice.org/dl.fedoraproject.org/pub/fedora/linux/releases/$VER/Everything/ \ /mirror/dl.fedoraproject.org/pub/fedora/linux/releases/$VER/Everything/ # Updates # rsync -$OPT --delete --no-motd --delete-excluded --exclude="aarch64" --exclude="armhfp" \ --exclude="source" --exclude="SRPMS" \ rsync://rsync.mirrorservice.org/dl.fedoraproject.org/pub/fedora/linux/updates/$VER/Everything/ \ /mirror/dl.fedoraproject.org/pub/fedora/linux/updates/$VER/Everything/ exit 0 |
The above script is called fedora-mirror and has 755 permissions. This script is very restrictive in that it only downloads Fedora 33, and excludes the versions for Risc processors. You can of course, mirror everything if you want - and have the space.
Ubuntu
Here is a simple rsync script to mirror Ubuntu Linux from an Internet-based rsync source. Check the mirror lists of your distro choice to find the nearest rsync source to you.
| #!/bin/bash # Options # OPT=vrhltD rsync -$OPT --delete-after --no-motd rsync://rsync.mirrorservice.org/releases.ubuntu.com/ /mirror/releases.ubuntu.com/ rsync -$OPT --delete-after --no-motd rsync://rsync.mirrorservice.org/archive.ubuntu.com/ubuntu/ /mirror/archive.ubuntu.com/ exit 0 |
The above script is called ubuntu-mirror and has 755 permissions.
/etc/cron.d
Now you have your distro scripts, you will want to run them at set times of the day/night, perhaps every 24 hours, and at a time when your Internet connection is fairly quiet.
The trick to achieve this is to create a time-controlled cron job file in /etc/cron.d as in the example below. The file is called mirror and contains the following examples:
| # Cron job to run the mirror jobs MAILTO=root 00 01 * * * root /root/scripts/centos-mirror 30 01 * * * root /root/scripts/rocky-mirror |
The example above will email the results of the jobs to root, which should then end-up in your mailbox, assuming you have the MTA configured to handle that... The CentOS mirror job is run at 01:00 (UTC in the case of my server) and the Rocky mirror job is run at 01:30. As these jobs are likely to only be downloading small changes, it is rare for them to overlap. Point-releases and major version releases being the exception.
FTP | Apache | NFS
Now we have our in-house mirror, it might be handy if we can access it! There are three ways we can achieve this: via FTP, via Apache, and via NFS.
ftp and http connections work best with a DNS name. From the DNS howto, you will want to add an 'A' record to point 'mirror.yourdomain.com' at the IP address of your mirror server.
FTP
Some quarters of the Internet are trying to kill-off FTP due to its lack of security. For an in-house mirror, FTP is a rather handy and speedy way of delivering packages to servers/desktops that are being installed via PXE / UEFI boot.
You will need to install vsftpd and enable it for anonymous read-only access. By default anonymous FTP access points to /var/ftp (on RHEL and its clones). We need to change that to suit our mirror set-up. Edit /etc/passwd with your favourite text-editor and change the ftp entry to read: ftp:x:14:50:FTP User:/mirror/./:/sbin/nologin
The above change re-points the anonymous FTP connections to the root of the /mirror directory and the /./ chroot-gaols the connection and stops anonymous FTP clients from wandering around the rest of the file system.
Apache
You will need to install httpd and create a basic configuration to access your mirror server via http. Configuration files for RHEL+clones can be found in /etc/httpd/conf.d. Create a file called mirror.conf with the following:
| # mirror.conf # <VirtualHost *> ServerAdmin webmaster@domain.net CustomLog logs/mirror_log combined ServerName mirror.domain.net DocumentRoot /var/www/html/mirror </VirtualHost> |
Adjusting the domain names and email to suit your own requirements.
In order to use the cheat of /var/www/html/mirror, change to the /var/www/html directory and create a symlink to /mirror with ln -s /mirror. This saves having to make extra declarations in the conf file for Options Indexes Includes FollowSymLinks MultiViews as these are already declared for /var/www/html.
Typing "mirror" into a browser on your local network should resolve and take you straight to the root directory of /mirror and list the directories of the distributions you have established.
NFS
You will need to install, enable, and start the NFS daemon of your server. Once it is up and running, it is simply a case of adding an entry in /etc/exports like this:
| /mirror 192.168.30.0/24(rw,sync,no_root_squash) |
The above example assumes your network is 192.168.30.x/24. Adjust as necessary.
Linux clients on your network who want access to the share can simply add this line to their /etc/fstab file:
| 192.168.30.253:/mirror /mirror nfs defaults 0 0 |
This assumes the server IP is 192.168.30.253. A resolvable name will also work, and is perhaps a better method in case you move the server to another IP address.
This method may have the least use unless you have developers who want to keep grabbing large ISO images from the in-house mirror; or as SysAdmin, you want a simple way to push config files to the /mirror area for PXE / UEFI boot.
Yum/DNF/apt-get
To make use of the local mirrors, and to save on bandwidth, you will need to create repo files that point to your mirror server. RHEL+clones need this to be carried out post-install. Ubuntu, when given a local mirror in a preseed file, will set that mirror in its apt-get configuration.
Caution: laptops that roam off the home/business network will fail to find updates as your in-house mirror is not public ally accessible. There are other ways of creating a mirror list to get around this, or you can simply leave the laptops to use public mirrors.
Is it a good idea to collect the modified files and store them in the /mirror directory where they can be copied via kickstart scripts. I detail that in the PXE / UEFI boot howto !PENDING!
CentOS
The following is an example from my own systems of the CentOS-Base.repo file found in /etc/yum.repos.d:
| # CentOS-Base.repo # # The mirror system uses the connecting IP address of the client and the # update status of each mirror to pick mirrors that are updated to and # geographically close to the client. You should use this for CentOS updates # unless you are manually picking other mirrors. # # If the mirrorlist= does not work for you, as a fall back you can try the # remarked out baseurl= line instead. # # [base] name=CentOS-$releasever - Base #mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=os&infra=$infra #baseurl=http://mirror.centos.org/centos/$releasever/os/$basearch/ baseurl=http://mirror.gaztronics.net/mirror.centos.org/$releasever/os/$basearch/ gpgcheck=1 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7 #released updates [updates] name=CentOS-$releasever - Updates #mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=updates&infra=$infra #baseurl=http://mirror.centos.org/centos/$releasever/updates/$basearch/ baseurl=http://mirror.gaztronics.net/mirror.centos.org/$releasever/updates/$basearch/ gpgcheck=1 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7 #additional packages that may be useful [extras] name=CentOS-$releasever - Extras #mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=extras&infra=$infra #baseurl=http://mirror.centos.org/centos/$releasever/extras/$basearch/ baseurl=http://mirror.gaztronics.net/mirror.centos.org/$releasever/extras/$basearch/ gpgcheck=1 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7 #additional packages that extend functionality of existing packages [centosplus] name=CentOS-$releasever - Plus #mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=centosplus&infra=$infra #baseurl=http://mirror.centos.org/centos/$releasever/centosplus/$basearch/ baseurl=http://mirror.gaztronics.net/mirror.centos.org/$releasever/centosplus/$basearch/ gpgcheck=1 enabled=0 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7 |
Text in red has been added to comment-out the mirrorlist and add the local mirror server as a baseurl. The FQDN path for this will depend on your domain name, plus the path you established to mirror the distribution.
Rocky
Rocky Linux has split the yum.repos.d files up so you have more to edit. Here is an example of Rocky-BaseOS.repo:
| # Rocky-BaseOS.repo # # The mirrorlist system uses the connecting IP address of the client and the # update status of each mirror to pick current mirrors that are geographically # close to the client. You should use this for Rocky updates unless you are # manually picking other mirrors. # # If the mirrorlist does not work for you, you can try the commented out # baseurl line instead. [baseos] name=Rocky Linux $releasever - BaseOS #mirrorlist=https://mirrors.rockylinux.org/mirrorlist?arch=$basearch&repo=BaseOS-$releasever #baseurl=http://dl.rockylinux.org/$contentdir/$releasever/BaseOS/$basearch/os/ baseurl=http://mirror.gaztronics.net/rocky/$releasever/BaseOS/$basearch/os/ gpgcheck=1 enabled=1 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-rockyofficial |
Again, the text in red has been added to comment-out the mirrorlist and add the local mirror server as a baseurl. The FQDN path for this will depend on your domain name, plus the path you established to mirror the distribution.
Ubuntu
Here is an example of a rather old Ubuntu 16 LTS sources.list file pointing to a local mirror, plus public mirrors for security updates:
| deb http://mirror.domain.com/archive.ubuntu.com/ubuntu/ xenial main restricted deb http://mirror.domain.com/archive.ubuntu.com/ubuntu/ xenial-updates main restricted deb http://mirror.domain.com/archive.ubuntu.com/ubuntu/ xenial universe deb http://mirror.domain.com/archive.ubuntu.com/ubuntu/ xenial-updates universe deb http://mirror.domain.com/archive.ubuntu.com/ubuntu/ xenial multiverse deb http://mirror.domain.com/archive.ubuntu.com/ubuntu/ xenial-updates multiverse deb http://mirror.domain.com/archive.ubuntu.com/ubuntu/ xenial-backports main restricted universe multiverse deb http://security.ubuntu.com/ubuntu xenial-security main restricted deb http://security.ubuntu.com/ubuntu xenial-security universe deb http://security.ubuntu.com/ubuntu xenial-security multiverse |
Page updated: 28th August 2021