When performing automated Kubernetes cluster construction using Kubespray in an air-gapped environment completely isolated from external networks, it is necessary to procure the required packages for bootstrap processing and OS-level initialization in advance. Simply obtaining and bringing in single RPM packages from an internet-connected environment frequently leads to package installation errors during provisioning due to OS minor version mismatches or missing dependency trees.

This article outlines the procedure for fully collecting all required RPM packages using the dependency resolution flags of dnf on an internet-connected staging host, assuming a Rocky Linux 9.7 environment, and building and verifying a local Yum repository on an internal network Web server.

1. Staging Environment Preparation and Dependency Resolution Mechanism

The internet-connected staging host is configured with the same OS distribution and architecture as the target air-gapped nodes. To traverse the package dependency tree and collect all downstream dependencies without omission, the dnf download command is used after installing the dnf-plugins-core plugin.

sudo dnf install -y dnf-plugins-core

# Create payload storage directories
mkdir -p /data/kubespray-rpms/os
mkdir -p /data/kubespray-rpms/docker-ce-stable

Key Option Specifications for the dnf download Command

Specify the following flags for downloading dependencies:

• --resolve: Automatically analyzes the entire required dependency graph for the specified packages.
• --alldeps: Forces all RPM files included in the dependency tree to be targeted for download, even if the packages are already installed on the staging host.

dnf download --resolve --alldeps --destdir=<target_directory> <package_name>

2. RPM Package Acquisition Procedure

2.1 Downloading OS Base Packages and Kubernetes Network Dependencies

Collect the basic tools, network control utilities, and Python bindings required for Ansible execution and Kubespray node setup.

sudo dnf download --resolve --alldeps \
  --destdir=/data/kubespray-rpms/os \
  python3 \
  python3-libselinux \
  conntrack-tools \
  socat \
  iproute \
  iproute-tc \
  iptables \
  ipset \
  ipvsadm \
  ethtool \
  chrony \
  rsync \
  tar \
  unzip \
  curl \
  openssl \
  ca-certificates \
  ebtables

*Note: If the ebtables package is not provided or is deprecated in Enterprise Linux 9 series repositories such as Rocky Linux 9, exclude ebtables from the target list before execution.

sudo dnf download --resolve --alldeps \
  --destdir=/data/kubespray-rpms/os \
  python3 python3-libselinux conntrack-tools socat iproute iproute-tc \
  iptables ipset ipvsadm ethtool chrony rsync tar unzip curl openssl ca-certificates

2.2 Acquiring Container Runtime Dependencies (containerd.io)

Because the standard OS repositories do not contain a production-level containerd.io package, register the official Docker CE repository to retrieve it.

# Add the Docker CE Stable repository
sudo dnf config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo

# Download containerd and SELinux policies
sudo dnf download --resolve --alldeps \
  --destdir=/data/kubespray-rpms/docker-ce-stable \
  containerd.io \
  container-selinux

When adopting the container_manager: containerd configuration in Kubespray, the full docker-ce engine set is not required; configuration is possible with only containerd.io and container-selinux.

3. Creating an Archive and Deploying the Local Repository

Check the volume of the acquired packages and bundle them as a compressed archive.

cd /data
tar czf kubespray-rpms-rocky9.7.tgz kubespray-rpms

Transfer the created archive to an internal mirror server (e.g., Nginx) and extract it under the document root.

sudo mkdir -p /usr/share/nginx/html/ROCKY_9.7
sudo tar xzf kubespray-rpms-rocky9.7.tgz -C /usr/share/nginx/html/ROCKY_9.7

After extraction, execute the createrepo_c utility to generate repository metadata (repodata).

sudo createrepo_c /usr/share/nginx/html/ROCKY_9.7/kubespray-rpms/os
sudo createrepo_c /usr/share/nginx/html/ROCKY_9.7/kubespray-rpms/docker-ce-stable

4. Client Node Configuration and Verification

On the target nodes within the air-gapped environment, disable external repository configurations and create /etc/yum.repos.d/kubespray-local.repo to reference the created local repository.

[kubespray-os]
name=Kubespray OS RPMs
baseurl=http://harbor.devstack.co.kr/ROCKY_9.7/kubespray-rpms/os
enabled=1
gpgcheck=0

[kubespray-containerd]
name=Kubespray Containerd RPMs
baseurl=http://harbor.devstack.co.kr/ROCKY_9.7/kubespray-rpms/docker-ce-stable
enabled=1
gpgcheck=0

Troubleshooting

Case 1: createrepo_c Does Not Exist on the Mirror Server

If the createrepo_c command is not installed on the repository server itself within the air-gapped environment, metadata generation cannot be performed, causing dnf makecache to fail.

Remediation Procedure:
On the internet-connected staging host, acquire createrepo_c itself along with its dependencies in advance, and include it in the payload to bring over.

sudo dnf download --resolve --alldeps \
  --destdir=/data/kubespray-rpms/os \
  createrepo_c

Case 2: Repository Index Synchronization Error and Package Not Found Issues

After configuring the local repository, if old cache remains when executing dnf list, errors indicating missing packages will occur. Explicit execution of cache clearing and index rebuilding is required.

Verification Log Example:

$ sudo dnf clean all
15 files removed

$ sudo dnf makecache
Kubespray OS RPMs                                3.2 MB/s | 2.1 MB     00:00    
Kubespray Containerd RPMs                        1.8 MB/s |  12 kB     00:00    
Metadata cache created successfully.

$ sudo dnf repolist
repo id               repo name
kubespray-containerd  Kubespray Containerd RPMs
kubespray-os          Kubespray OS RPMs

$ dnf list containerd.io conntrack-tools socat
Available Packages
conntrack-tools.x86_64            1.4.7-2.el9                kubespray-os        
containerd.io.x86_64              1.7.25-3.1.el9             kubespray-containerd
socat.x86_64                      1.7.4.1-5.el9              kubespray-os

Lessons Learned

1. Forced Full Dependency Acquisition: Without the --alldeps option, packages already installed on the staging host will be skipped, posing a risk of missing dependencies on the nodes in the air-gapped environment.
2. Isolated Management of Container Runtime: Version conflicts can be avoided by individually retrieving containerd.io from the Docker CE repository and indexing it separately from standard OS repositories.
3. Pre-generation of repodata: Creating metadata with createrepo_c on the repository server before making it available to clients is a mandatory requirement for healthy Yum/DNF package distribution.</package_name></target_directory>