wiki / storage / disks Edit this page
Edited (ad8b8f7) at 2023-06-30 by Dylan Araps
STORAGE
________________________________________________________________________________
This article will provide general guidance on the process of preparing a disk
device (SSD or HDD) for a new installation of KISS Linux. Advanced storage
formats (e.g. RAID, LVM) are considered out of scope.
Index
________________________________________________________________________________
- Overview [0.0]
- Terminology [0.1]
- Tools [0.2]
- Partitioning Schemata [1.0]
- Minimum Partition Layouts [1.1]
- Swap Partition vs File [1.2]
- Partitioning Example [2.0]
- Formatting [3.0]
- Filesystems [3.1]
- Swap Partition [3.2]
- Swap File [3.3]
- fstab [4.0]
- References [5.0]
[0.0] Overview
________________________________________________________________________________
Next to kernel configuration, preparing a disk device for installing a Linux
distribution is one of the more intimidating topics for a new UNIX user to
tackle. This process is not only highly dependent on user circumstances (the
type of hardware you have access to, your file system needs, your backup
requirements, etc.), but the process varies based on the tools used for the job.
Therefore, the first step to a successful installation is understanding the
terminology and the tools which are required throughout the process.
[0.1] Terminology
____________________________________________________________________________
+------------------+-------------------------------------------------------+
| | |
| Block Device | Represents an abstract interface to the disk. |
| | User programs can use these block devices to |
| | interact with the disk without worrying about |
| | whether the drives are SATA, SCSI, or something |
| | else. (e.g. /dev/sd*, /dev/nvme0n*, etc) |
| | |
| Partition | Represent smaller, more manageable block devices. |
| | These can be thought of as the places where user |
| | data lives. |
| | |
| Filesystem | A filesystem allows for data to actually be written |
| | to a given partition. Filesystems vary in features, |
| | requirements, and read/write access. In short, a |
| | filesystem is the structure for a given partition. |
| | |
| DOS | The DOS disklabel setup uses 32-bit identifiers |
| | for the start sector and length of the partitions, |
| | and supports three partition types: primary, |
| | extended, and logical. Primary partitions have |
| | their information stored in the master boot |
| | record itself - a very small (usually 512 bytes) |
| | location at the very beginning of a disk. Due to |
| | this small space, only four primary partitions |
| | are supported (for instance, /dev/sda1 to |
| | /dev/sda4). This is the old format for disklabels. |
| | |
| GPT | GPT (GUID Partition Table) setup uses 64-bit |
| | identifiers for the partitions. The location in |
| | which it stores the partition information is much |
| | bigger than the 512 bytes of a DOS disklabel, |
| | which means there is practically no limit on the |
| | amount of partitions for a GPT disk. Also the |
| | size of a partition is bounded by a much greater |
| | limit (almost 8 ZB - yes, zettabytes). A GPT disk |
| | is required for UEFI systems. |
| | |
| ROOTFS | ROOTFS (Root Filesystem) is the primary partition |
| | where the entire system is directly or indirectly |
| | mounted to. This is commonly referred to as '/'. |
| | |
| BOOT | The boot partition is where important files |
| | required at boot time, such as the kernel, are |
| | stored. This partition is required only on UEFI |
| | systems. It is usually mounted at /boot. |
| | |
| HOME | This partition is where most user data is stored. |
| | It is usually mounted at /home. |
| | |
| SWAP | Swap space can take the form of a disk partition |
| | or a file. Users may create a swap space during |
| | installation or at any later time as desired. |
| | Swap space can be used for two purposes: to |
| | extend the virtual memory beyond the installed |
| | physical memory (RAM), and also for hibernation |
| | or for suspend-to-disk support. |
| | |
+------------------+-------------------------------------------------------+
Some useful facts and things to keep in mind:
Due to using 32-bit identifiers, DOS partitioning tables cannot handle disks
that are >2 TBs in size.
Unless an extended partition is created, DOS disk labels supports a maximum
of four partitions.
Using GPT on a BIOS-based computer works, and is referred to as a hybrid
setup. This type of setup requries a 1MB BIOS boot partition (unformatted)
so that extra data can be stored by the bootloader (like GRUB2). This setup
is useful in cases where more than four primary or secondary partitions are
required for a system.
Unfortunately this has several limitations. For instance, you cannot
dual-boot with a Microsoft Windows operating system, as Windows will boot in
UEFI mode if it detects a GPT partition label. Also note that some buggy
(old) motherboard firmware configured to boot in BIOS/CSM/legacy mode might
also have problems with booting from GPT labeled disks. As such, this style
is undesireable, and in general it is a good idea to use GPT in cases where
UEFI can be used.
UEFI/GPT and BIOS/MBR are not synonymous. UEFI/BIOS refer to particular
types of systems that utilize the Unified Extensible Firmware Interface or
the Basic Input/Output System. This is determined strictly by the
motherboard used in a given system. GPT/DOS, however, refer to particular
disk labels: GUID Partition Tables versus the DOS label. These types
directly limit the structures available on any given disk drive - both the
number of partitions available, as well as the size of any given partition.
A good rule of thumb, however, is that UEFI systems use GPT disk labels,
BIOS systems use DOS disk labels, and hybrid systems use BIOS + GPT.
The BOOT partition is a source of much confusion for users. For instance,
it is erroneously claimed that it must be mounted to /efi, or that kernel
images should be named vmlinuz.efi.
In point of fact, it is only necessary on UEFI systems, though it may be
useful on BIOS systems. The BOOT partition need not be mounted to /efi.
Indeed, the only requirements are that it be formatted to FAT32 (in the case
of UEFI systems), and be at least 100MB in size. The recommended minimum
size for UEFI systems is 256MB.
That the kernel be named vmlinuz.efi serves only to make the fact that it
is a UEFI system explicit.
If users intend on using a multiboot system, a single BOOT partition can be
shared by all operating systems.
An important point of consideration is whether or not a separate HOME
partition should be used. This is useful in cases where users wish to keep
their data in a location separate from their operating system, for instance
in cases of critical failures requiring the OS to be reinstalled, or if
users wish to encrypt their personal data separately from their core OS.
This separation between / and /home allow users to easily migrate their data
across many different operating systems; they need only remount the HOME
partition to /home on the new system. In general, this is recommended.
In addition to a separate HOME, several other additional partition options
exist. For instance, a separate var (/var) and data partitions can be used.
Separate data partitions are useful in instances where users would like to
easily share data across different operating systems, as allowing write
access to a user's home directory is a security risk. A separate /var is
useful in niche cases where /usr is read-only. For more detailed information
on filesystem structures, refer to the Filesystem Hierarcy Standard [0].
Additionally, the Linux From Scratch project has some useful rationales [1].
[0.2] Tools
____________________________________________________________________________
Depending on which live environment the user chooses for installing a KISS
Linux system, the tools available vary. On most UNIX systems, the following
can usually be expected:
- fdisk: a dialogue driven partition manipulator [2]
- gdisk: GPT fdisk (note that fdisk also supports GPT) [3]
- cfdisk: a partition editor with a curses-based UI, part of util-linux [4]
- parted: a GNU partition editor [5]
It is also possible to use the live CD version of parted, GParted [6], to
partition disks, and it offers many other useful features for disk
management and data recovery.
In general, most programs can handle the majority of use-cases. The tool of
choice is almost entirely user preference. However, some utilities have
powerful and advanced features that can aid in fixing damaged disks. fdisk
is one such program, and is useful in cases where more fine-grained control
is required. For general system setup, however, any tool will do.
[1.0] Partitioning Schemata
________________________________________________________________________________
Great care should be taken when picking a partition structure, as changing it
afterwards is (generally) a difficult endeavor. Ensure that the chosen structure
meets your use case. Things to consider include whether or not a swap partition
is needed, whether to use a separate home partition, whether separate partitions
are needed for general data storage (like music or photos), and whether
dual-booting is desired.
[1.1] Minimum Partition Layouts
____________________________________________________________________________
The number of partitions required is highly variable and dependent on the
use case. The following table provides examples of common partition
schemata, formats, and recommended minimum sizes:
+----------------------------+---------------------------------------------+
| Schemata Description | Partition (Type, Format, Size) |
+----------------------------+---------------------------------------------+
| | |
| GPT + UEFI (with /home) | /dev/sda1 (BOOT, FAT32, 256MB) |
| | /dev/sda2 (ROOTFS, EXT4, 30GB ) |
| | /dev/sda3 (HOME, EXT4, * ) |
| | |
| Legacy DOS (with swap) | /dev/sda1 (SWAP, swap, RAM ) |
| | /dev/sda2 (ROOTFS, EXT4, * ) |
| | |
| Hybrid BIOS/GPT | /dev/sda1 (BIOS BOOT, XXXX, 1MB ) |
| | /dev/sda2 (SWAP, swap, RAM ) |
| | /dev/sda3 (ROOTFS, EXT4, xxxGB) |
| | /dev/sda4 (HOME, XFS, yyyGB) |
| | /dev/sda5 (DATA, BTRFS, zzzGB) |
| | |
+----------------------------+---------------------------------------------+
The boot partition is generally only used for storing kernels or an
initramfs. As these files are usually very small (<20MB), 256MB is more
than enough to accomodate most use-cases. 512MB is plenty. A separate BOOT
is only required on UEFI systems.
In cases where no other additional partition is desired, the ROOTFS or HOME
partition should span the remainder of the disk.
For source-based distributions like KISS, maximizing the amount RAM that can
be used is paramount for successful builds, especially for large packages
such as rust, firefox, or qt5-webengine. If the amount of RAM available is
less than 8GB, the usage of a swap partition or swap file is strongly
recommended to avoid build failures due to out-of-memory (OOM) issues. A
decent rule of thumb for swap size is to have the same amount of swap as
you have RAM.
These examples only serve to show a minimum partition layout required for
each type of disk style (UEFI + GPT, BIOS + DOS, hybrid).
[1.2] Swap Partition vs File
____________________________________________________________________________
When considering swap space, there are benefits and tradeoffs to certain
choices. The traditional recommendation has been to create a swap partition
as close to the first partition as possible, that is at least half the
amount of RAM available - this number is a sliding scale and varies by use
case. For instance, if few RAM intensive programs are going to be used, and
little compiling will be done on the host system, less swap space is needed
(if any at all).
Historically, swap was placed at the beginning of disks as this allowed for
the fastest possible seek times on traditional spinning disk drives (hard
drives). This was preferred because disk access times are orders of
magnitude slower than RAM access times, and long swap operations result in
a desktop feeling 'slow' or 'laggy'. However, with the rise in popularity
of solid state drives (and, more recently, NVMe based storage), along with
the precipitous decrease in storage space cost, the requirement of a
separate swap partition has laxed. If users have access to low latency,
large storage drives, a swap file may be a preferable alternative.
Swap files allow for ondemand, resizeable swap spaces. Swap may not be
necessary for day-to-day operation but only in cases where large builds are
happening. With access to these newer technologies, users can simply create
a file and define it to be a dedicated swap location which is used
identically to the traditional swap partition. This has left the need for a
swap partition largely to cases where solid state storage is not an option.
For more modern systems, swap files are in general preferable to partitions.
Care should be taken, however. Changes may occur outside of the user's
control that necessitate intervention. For instance, if a swapfile
disappears but is still referenced in /etc/fstab, disk mounting will fail
at boot and the system will drop users into an emergency shell.
Additionally, file fragmentation can cause swap files to become unreliable.
Finally, kernel updates could potentially cause issues [7].
In addition to swap, @/zswap and @/zram are useful options to consider for
maximizing swap usage and memory management.
[2.0] Partitioning Example
________________________________________________________________________________
The following is a step-by-step partitioning example.
PLEASE BE AWARE: The following commands can cause irreparable damage to the
data on your drives. Ensure that the data on these drives is backed up to a
separate device, and make sure you select the correct device for partitioning.
Again, these commands are dangerous and WILL CAUSE DATA LOSS.
In order to identify the correct disk, there are several commands that can help
+------------------------------------------------------------------------------+
| |
| $ fdisk -l |
| $ sfdisk -l |
| $ parted -l |
| $ lsblk |
| |
+------------------------------------------------------------------------------+
While partitioning disks, you will be asked multiple questions. These include
the starting sector of the partition, the last sector of the partition, and the
hex code for that partition.
Generally, using the default as the first sector is the best choice - this will
ensure there are no strange breaks between partitions.
Most programs support relative last sector locations. Therefore, instead of
typing out the desired sector number, one need only write '+400GB' to make the
partition start at the first available sector and continue for the next 400GB.
Hex codes or partition IDs are hexadecimal identifiers which the kernel and
filesystem programs can interpret, and programs like fdisk and cfdisk will use
the hex code of a given partition to display a useful fact about the partition,
like that it's specifically an EFI partition, or a Linux root partition, or a
Solaris partition. Because GPT uses 64-bit partitions identifiers, far more
partition types are available for use on GPT systems over MBR systems.
Be aware that altering partitions on in-use disks could cause data corruption,
and the changes in partition layout may not be available until the next reboot.
This example assumes the target device block of /dev/nvme0n1, using gdisk
for the actual partitioning process.
To create a GPT + EFI disk with a separate /home,
+------------------------------------------------------------------------------+
| |
| $ gdisk /dev/nvme0n1 |
| |
| # Enter '?' for a list of available commands |
| |
| # Delete partition data by creating a new GPT: |
| $ o |
| This option deletes all partitions and creates a new GPT. |
| Proceed? (Y/N): y |
| |
| # Create Partition 1 (/boot): |
| $ n |
| Partition Number: (RETURN key for 1) |
| First sector: (RETURN key for first available) |
| Last sector: +128M |
| Hex Code (L to show codes): ef00 |
| |
| # Create Partition 2 (ROOTFS): |
| $ n |
| Partition Number: (RETURN key for 2) |
| First sector: (RETURN key for first available) |
| Last sector: +30G |
| Hex Code: 0304 |
| |
| # Create Partition 3 (HOME): |
| $ n |
| Partition Number: (RETURN key for 3) |
| First sector: (RETURN key for first available) |
| Last sector: (RETURN key for rest of disk) |
| Hex Code: 0302 |
| |
| # Write Partition Table To Disk: |
| $ w |
| Do you want to proceed? (Y/N): Y |
| |
+------------------------------------------------------------------------------+
cfdisk may be preferable to users who prefer a more visual representation of
partition manipulation and a constantly updating table of information on the
current partition layout and structure.
[3.0] Formatting
________________________________________________________________________________
Now that the disks have been partitioned, it is time to create the relevant
filesystems for each partition.
[3.1] Filesystems
____________________________________________________________________________
There are many different filesystems to choose from on any given Linux
system. Which filesystem is used depends almost entirely on the use-case of
the system. For simple data storage read only by UNIX systems, EXT4 is a
perfectly normal choice. For data that needs to be accessible to Windows,
FAT or NTFS is required. The Apple Filesystem (APFS) is used by Macs.
In the simple case where only Linux systems require access to a given
partition, many choices exist. Below is a list of filesystems which are
available in KISS either in the official repository or in community.
+--------------+------------------------+----------------------------------+
| Filesystem | Package | Command |
+--------------+------------------------+----------------------------------+
| | | |
| swap | core/busybox | mkswap |
| EXT2/3/4 | extra/e2fsprogs | mkfs.ext2/3/4 |
| DOS/FATxx | extra/dosfstools | mkfs.dos/fat/vfat |
| XFS | extra/xfsprogs | mkfs.xfs |
| BTRFS | community/btrfs-progs | mkfs.btrfs |
| NTFS | community/ntfs-3g | mkfs.ntfs |
| | | |
+--------------+------------------------+----------------------------------+
Other filesystems exist with varying degrees of popularity, including JFS,
ReiserFS, and ZFS. The community repository is an excellent place to share
work in including these filesystems in KISS! Others are available in
user-created repositories. ZFS is one such example [8]. Due to licensing
restrictions, ZFS requires more work to use than other filesystems.
EXT4 is a solid, general purpose filesystem. For UEFI systems, FAT32 is a
required filesystem for the BOOT partition. BTRFS is an experimental
filesystem which sees heavy development by organizations like Facebook,
and offers a different way of managing storage than more traditional (EXT,
DOS, NTFS) filesystems. All of these have built-in kernel support.
For a more complete list of other filesystem options and what limitations
and features exist for them, refer to the Arch Wiki [9].
Continuing from the partition example in [2.0], we will make a FAT32
parition on BOOT (/dev/nvme0n1p1), and an EXT4 partition on ROOTFS and HOME
(/dev/nvme0n1p2 and /dev/nvme0n1p3, respectively):
+--------------------------------------------------------------------------+
| |
| $ mkfs.ext4 /dev/nvme0n1p2 |
| $ mkfs.ext4 /dev/nvme0n1p3 |
| $ mkfs.fat -F32 /dev/nvme0n1p1 |
| |
| $ mount /dev/nvme0n1p2 /mnt |
| $ mount /dev/nvme0n1p3 /mnt/home |
| $ mount /dev/nvme0n1p1 /mnt/boot |
| |
+--------------------------------------------------------------------------+
[3.2] Swap Partition
____________________________________________________________________________
To set up a partition as a Linux swap space, the mkswap command is used.
Replace X with the drive where the swap partition is located, and Y by the
partition on that drive that will be formatted as swap.
+--------------------------------------------------------------------------+
| |
| $ mkswap /dev/sdXY |
| $ swapon /dev/sdXY |
| |
+--------------------------------------------------------------------------+
[3.3] Swap File
____________________________________________________________________________
As an alternative to creating an entire partition, a swap file offers
similar functionality, and the added benefits of being able to vary its size
on-the-fly, as well as being more easily removed. This may be especially
desirable if disk space is at a premium.
The most straightforward way to create a swap file is to use dd:
+--------------------------------------------------------------------------+
| |
| $ dd if=/dev/zero of=$LOCATION bs=$BYTES count=$BLOCKS |
| |
+--------------------------------------------------------------------------+
$LOCATION is where the swapfile should be made. Assume $LOCATION=/swapfile.
$BYTES is the size of each block to be written to the file. Disks have
varying supported block sizes; choosing the correct size for the partition
where the swapfile will be is important for optimizing read/write
throughput. dd defaults to 512 bytes, but this is suboptimal for drives
with larger supported block sizes. To determine the block size for your
device, you can use stat:
+--------------------------------------------------------------------------+
| |
| $ stat -fc %s . |
| |
+--------------------------------------------------------------------------+
For instance, if the result is 4096, $BYTES=4k should be chosen.
$BLOCKS is the number of blocks of size $BYTES you wish to write to
$LOCATION. $BLOCKS determines the final size of the swapfile.
$BYTES and $BLOCKS can be suffixed by b (512 bytes), kB (1,000 bytes), k
(1,024 bytes), MB (a thousand kB), M (a thousand k), GB (a million kB), or
G (a million k).
To create a 16G swapfile at /swapfile with a 4k block size,
+--------------------------------------------------------------------------+
| |
| $ dd if=/dev/zero of=/swapfile bs=4k count=4M |
| |
+--------------------------------------------------------------------------+
Set the permissions, create the swap filesystem, and activate it as swap,
+--------------------------------------------------------------------------+
| |
| $ chmod 600 /swapfile |
| $ mkswap /swapfile |
| $ swapon /swapfile |
| |
+--------------------------------------------------------------------------+
To remove a swap file, it's as simple as disabling and deleting it,
+--------------------------------------------------------------------------+
| |
| $ swapoff /swapfile |
| $ rm -f /swapfile |
| |
+--------------------------------------------------------------------------+
[4.0] fstab
________________________________________________________________________________
The fstab file contains a list of partitions with their filesystem type, their
mount location, and the options they should be mounted with. The mount program
will read the fstab file (by default /etc/fstab) and will mount all of the
partitions and filesystems for you. This is useful for automatically mounting
things like BOOT, HOME, and tmpfs during the init process.
The fstab is NOT required - the kernel location and ROOTFS should be specified
in the bootloader entry. If you find yourself mounting the same partitions
repeatedly with consistent options, the fstab file serves to automate this
prcess for you.
Here is an example fstab, which will mount ROOTFS to /, BOOT to /boot, HOME to
/home, enable the swap file, and mount some important virtual filesystems:
+------------------------------------------------------------------------------+
| |
| # device mount-point type options dump fsck order |
| |
| /dev/sda2 / ext4 defaults 0 1 |
| /dev/sda1 /boot vfat defaults 0 2 |
| /dev/sda3 /home ext4 defaults 0 2 |
| /swapfile none swap defaults 0 0 |
| tmpfs /tmp tmpfs rw,nodev,nosuid 0 0 |
| proc /proc proc defaults 0 0 |
| sysfs /sys sysfs defaults 0 0 |
| devpts /dev/pts devpts gid=4,mode=620 0 0 |
| |
+------------------------------------------------------------------------------+
tmpfs, proc, sysfs, and devpts should be mounted during the init process.
Therefore, proc, sysfs, and devpts can be left out of the fstab. A reason to
keep tmpfs in the fstab is to use it for building packages in RAM. To only do
this for certain packages, see [10].
Devices can be referred to either by their /dev pathname, by a label, by UUID,
or by Part-UUID. For systems with multiple drives, it is recommended to use
UUIDs or labels instead of /dev pathnames, as these are volatile and could
change during each system reboot. See [11] for details on identifying disks.
The dump entry refers to the backup utility, dump [12].
fsck order gives the order in which a filesystem check should be ran in. ROOTFS
should be specified with a 1. All other filesystems should be specified with 2
or 0. swap and virtual filesystems should not be fsck'd.
There are a large number of options that you can choose from for mounting
partitions. 'defaults' is a basic, filesystem-independent option which will
mount the partition rw,suid,dev,exec,auto,nouser,async. For an explanation of
these options (and far more detailed information on the fstab) see [13].
CAUTION: an improperly configured fstab will result in the user being dumped
into an emergency shell! Even a small typo will result in this. If it occurs,
simply check the fstab for errors, mount the partitions that failed to be
mounted, and exit the emergency shell.
[5.0] References
________________________________________________________________________________
[0] https://refspecs.linuxfoundation.org/fhs.shtml
[1] http://linuxfromscratch.org/lfs/view/stable/chapter02/creatingpartition.html
[2] https://man7.org/linux/man-pages/man8/fdisk.8.html
[3] http://rodsbooks.com/gdisk/
[4] https://linux.die.net/man/8/cfdisk
[5] https://gnu.org/software/parted/manual/parted.html
[6] https://gparted.org/
[7] https://bugzilla.kernel.org/show_bug.cgi?id=207585
[8] https://github.com/jedavies-dev/kiss-zfs
[9] https://wiki.archlinux.org/index.php/file_systems
[10] #/package-manager#6.3
[11] https://wiki.archlinux.org/index.php/Fstab#Identifying_filesystems
[12] https://linux.die.net/man/8/dump
[13] https://man7.org/linux/man-pages/man5/fstab.5.html
________________________________________________________________________________
Dylan Araps (C) 2019-2021
The registered trademark Linux(R) is used pursuant to a sublicense from the
Linux Foundation, the exclusive licensee of Linus Torvalds, owner of the mark
on a worldwide basis.