- Kdump reserves a capture kernel via kexec for dumping
vmcoreafter a panic. - Configure crashkernel and dump path properly to avoid crashes and duplications.
- Secure Boot and lack of contiguous slot are the most common errors when loading kdump.
- Use crash with
vmlinuxSymbols for in-depth and reliable analysis.
When the system Linux suffers a kernel panic hang (see Differences between kernel panic and system crash), the only reliable clue is usually a core dump. Configure kdump and analyze vmcore with crash It can mean the difference between guessing and accurately diagnosing what happened, especially if the log leaves no trace.
In this practical guide, I've compiled the essentials from the best technical sources on the topic to give you a clear roadmap: What is kdump, how to configure it without common problems, how to solve typical errors (Secure Boot, crash kernel, save paths) and how to take advantage of crash to investigate vmcore. If your system crashes several times a day, here's a realistic and detailed action plan to get things in order.
What are kdump and crash, and why do they matter?
The kdump feature enables a kernel dump mechanism in the event of a catastrophic failure. Kdump reserves memory for a second capture kernel which is started by the system call kexec when the main kernel panics, and from there dumps the memory of the stopped system to disk.
In many enterprise distribution images, the system comes partially or fully prepared to generate crash dumps depending on the date of the image. However, it's a good idea to check and complete the configuration to ensure the dump is produced and saved in the correct location.
Red Hat's crash tool is the de facto standard for kernel dump analysis. Crash allows you to inspect dumps obtained with kdump, makedumpfile, diskdump and others, and can even work on running systems using /dev/mem or, in Red Hat derivatives, /dev/crash. It is more powerful than using gdb on /proc/kcore due to limitations on access to internal kernel structures and the requirements for the appropriate binary.
Red Hat recommends, especially in critical environments, that administrators Regularly update and test kexec-tools within the normal kernel update cycle, and even more so when new kernel features are released. This practice reduces surprises when system availability is most at stake.
Correct configuration and dump paths
First of all, understand what files you're going to get. In a typical dump, you'll see at least vmcore with kernel memory and auxiliary files such as vmcore-dmesg.txt o kexec-dmesg.log, useful for reconstructing the context of calls and messages prior to the crash.
It is key to clearly define where the dump is saved. A typical case in systems that mount the dump destination in /var/crash is to end up, unintentionally, with nested routes of the style /var/crash/var/crash. This occurs when the file system is mounted on /var/crash and also in /etc/kdump.conf is established path /var/crashThe result is that the dump falls into that duplicate path.
The solution is straightforward: if the dump mount point is already /var/crash, defines in kdump.conf path / instead of path /var/crashThis way, you avoid nesting and your dumps will appear in the expected directory. Use a configuration file filter (ignoring comments and blank lines) to ensure that the active options match the actual mount of the destination.
On distributions with GRUB, the memory allocation for the capture kernel is set with the parameter crashkernel= in the line of Boot. It is common to test values such as crashkernel=128M, 256M, 512M o auto, and regenerate the configuration with grub2-mkconfig o update-grub According to the distro's family, however, reserving memory doesn't, by itself, guarantee that the service will start if there are other crashes.
In Ubuntu/Debian environments, a typical sequence is to install the package linux-crashdump, enable the service kdump-tools and adjust /etc/default/grub with parameter crashkernel= before update-grub. The command kdump-config show It tells you the status: dump directory (/var/crash), links to vmlinuz e initrd from the capture kernel, the reserved address, and whether it's ready to run kdump. If you see "current state: Not ready to kdump" or "no kexec command recorded," the capture kernel is missing an effective load.
Another basic detail for analysis: crash requires a kernel binary with symbols. The appropriate file for analysis is vmlinux (uncompressed, with debug symbols), while vmlinuz This is the compressed version loaded at boot. Without the correct binary, crash may complain that it can't find the booted kernel and ask for the correct namelist. Install the -dbg/-debuginfo packages for your kernel to ensure the best scanning experience.
Common problems, how to solve them, and crash analysis
If after restarting you run systemctl status kdump.service And if you see that it's failing, the first thing to do is read the messages carefully. A very common real-life case shows: “Secure Boot is enabled. Using kexec file based syscall” followed by “failed to load kdump kernel”. With Secure Boot enabled, the system forces the use of the “file-based” variant of kexec, which can fail if the capture kernel or its initrd are not signed as expected by the firmware.
What to check in that scenario: confirm that The capture kernel and its initrd are suitable for Secure Boot, that the package kexec-tools is up-to-date and there are no unsigned critical modules in the kdump initrd. If your security framework allows it, temporarily disabling Secure Boot to isolate the problem helps distinguish whether the root cause of the issue is the signature or the memory allocation.
On Ubuntu/Debian, another illustrative error when loading kdump is: “Could not find a free area of memory of 0x943c000 bytes… locate_hole failed”. This indicates that a contiguous memory slot of the required size for the capture kernel cannot be found. Typical causes include memory fragmentation, insufficient allocation, or conflicts with the current layout.
Practical remedies for “locate_hole failed” and similar:
- Reduce or adjust the reserve: try with
crashkernel=256Mif you were on 512M or vice versa; on some computers, intermediate values work better thanauto. - It boots with fewer parameters that fragment memory at startup, and make sure to regenerate GRUB after the changes (
grub2-mkconfigoupdate-grub). - Check that the links of
/var/lib/kdump/vmlinuzeinitrd.imgpoint to the correct capture kernel and that there is your kdump initrd. - On machines with demanding devices or IOMMUs, crashkernel values with segments (e.g., zone-reserved schemes) can help; if your distro documents predefined profiles, try them before you invent one.
If the service reports “kdump: Starting kdump: ” and “Main process exited, status=1/FAILURE”, go back to basics: Was the capture kernel actually loaded? Run kdumpctl start o kdump-config load depending on your system, and examine the full output. The messages about initrd creation, symbolic links, and then a memory allocation failure all point to the same root: a missing contiguous space of the requested size.
Also remember that the graphical environment (KDE, GNOME, etc.) has no place in this film: kdump operates at the kernel and boot levelIf someone asks you “how to get it working in KDE,” the answer is that the DE isn’t the factor, but the firmware (Secure Boot), the bootloader, memory allocation, and proper packaging of the capture kernel/initrd.
We return to the peculiarity of the routes: if you see overturned in /var/crash/var/crash, check out /etc/kdump.conf. When the dump target is mounted on /var/crash and the option path it is also /var/crash, the result is the duplicate route. Establishes path / and matter resolved.
Once you generate the dump (e.g., vmcore together with vmcore-dmesg.txt y kexec-dmesg.log), it's time to use crash. Crash loads the vmcore and binary vmlinux with symbols and opens a rich analysis environment: you can list processes, call stacks, modules, memory, locks, etc. It is the ideal approach compared to trying gdb with /proc/kcore, which is often limited and dependent on how the kernel was compiled.
To work on a system that is not in production (or after copying the files), launch crash indicating the vmcore and the vmlinux suitable. Without the correct vmlinux the analysis is limping, because symbols are missing to interpret internal structures. If your distro offers -debuginfo or -dbg kernel packages, install them and use their vmlinux corresponding to the vmcore version.
It's important to understand the security implications. A vmcore contains "everything" that was in memory at the time of the crash: sensitive data, keys, fragments of conversations and anything that was alive in processes and kernel. Therefore, the location and permissions of /var/crash And dumps should be treated with the same care as other secrets. Don't leave them accessible to users who shouldn't see them.
If you don't need a full kernel dump, and the goal is to debug a specific process, a less invasive strategy is to generate a process dump with tools like gcore. Process dumps are smaller and more manageable, and allow you to isolate use cases (e.g., a specific app that crashes). However, the risk of exposing sensitive process data remains, so the same precautions apply storage and permissions.
To close the loop on good operational practices: In mission-critical environments, ensure that Testing kdump after updating the kernel and kexec-toolsIt's better to discover that the "capture kernel won't boot" during a maintenance window than in the middle of a real incident. And document the path, retention, and safe removal/transfer procedure for vmcore for the development or support team.
Practical summary of signs and actions:
- Status says “Not ready to kdump” or does not register kexec command: missing capture kernel load; check crashkernel, kdump initrd, and reload with your distro's tool.
- Secure Boot active and “kexec file based syscall”: verifies signatures and compatibility from the kernel/initrd capture or test without Secure Boot to isolate the fault.
- “locate_hole failed”: set size of
crashkernel=and reduces fragmentation at boot; confirm links and regenerate GRUB. - Dumped in
/var/crash/var/crash: correctspath /enkdump.confwhen the FS is mounted on/var/crash.
When everything is in place, the ideal flow is: panic occurs, kexec starts the capture kernel, makedumpfile collects the memory and saves the vmcore to the agreed path, and you or your team open the case with crash and the vmlinux . That's the shortest path between incident and learning., and the one that saves you from repeating the error without visibility.
While each platform has its nuances, the fundamentals remain the same: reserve sufficient and contiguous memory, respect Secure Boot requirements when active, properly align the dump path with the mount point, and always have a vmlinux with hand-drawn symbols for crash analysis. With these pieces in place, kdump stops being a mystery and becomes your black box. when the kernel decides to land without warning.
Passionate writer about the world of bytes and technology in general. I love sharing my knowledge through writing, and that's what I'll do on this blog, show you all the most interesting things about gadgets, software, hardware, tech trends, and more. My goal is to help you navigate the digital world in a simple and entertaining way.