Link
WebDoc
Search

Debug the Linux

See medium.com/@daeseok.youn if you want a step-by-step guide book.

tools:

  • catchsegv (LD_PRELOAD=/lib/libSegFault.so)
  • calltree
  • gporf => gmon.out
  • gcov, -fprofile-arcs -ftest-coverage
  • Memwatch, Valgrind, mtrace

reference:

Table of contents

  1. Compile the Kernel
    1. vmlinux
    2. bzImage
  2. Installation
  3. qemu
  4. qemu-img
  5. KVM
  6. Create Root file system
    1. minimal
    2. Buildroot
    3. ready-to-use Debian
    4. Install by Image (Debian / Ubuntu / …)
  7. GDB
  8. Networking
    1. How to get SSH access to a guest

Compile the Kernel

Kernel compile flags can usually be checked by looking at /proc/config.gz (gzip -d to unzip it) or /boot/config-<kernel-version>.

make clean
make mrproper    # clean up 
make kernelversion
make help

make defconfig   # default config
make allnoconfig / allyesconfig / allmodconfig
make menuconfig  # menu-driven interface
make nconfig     # ncurses-based interface

make rpm         # make rpm package
  • Kernel hacking -> Compile the kernel with debug info (CONFIG_DEBUG_INFO)
  • Kernel hacking -> Compile the kernel with debug info -> Provide GDB scripts for kernel debugging (New)
  • Others related CONFIGs:
CONFIG_DEBUG_KERNEL=y
CONFIG_DEBUG_INFO=y
CONFIG_KALLSYMS=y
CONFIG_KALLSYMS_ALL=y
CONFIG_DEBUG_BUGVERBOSE=y
CONFIG_STACKTRACE=y

make arguments:

  • ARCH: arm64
  • CROSS_COMPILER: aarch64-linux-gnu-

Note that: You can’t de-optimize the kernel. You might be able to de-optimize certain functions, like this:

void __attribute__((optimize("O0"))) foo(unsigned char data) {
    // unmodifiable compiler code
}

But the kernel as a whole requires -O2 because the code itself is written with assumptions that certain functions will be optimized in a certain way. (Some people say that -O1 also works for kernel.)

make
make modules
make modules_install

vmlinux

On Linux systems, vmlinux is a statically linked executable file that contains the Linux kernel in one of the object file formats supported by Linux, which includes ELF, COFF and a.out. The vmlinux file might be required for kernel debugging, symbol table generation or other operations, but must be made bootable before being used as an operating system kernel by adding a multiboot header, bootsector and setup routines. It will be added to /boot.

bzImage

The default target of make. As the Linux kernel matured, the size of the kernels generated by users grew beyond the limits imposed by some architectures, where the space available to store the compressed kernel code is limited.

The bzImage (big zImage) format was developed to overcome this limitation by splitting the kernel over discontiguous memory regions.

Installation

make headers_install
make modules_install
make install

qemu

https://people.debian.org/~aurel32/qemu/

QEMU’s own “Copy On Write” image format: qcow qemu-system-x86_64 -hda debian.img -cdrom debian-testing-amd64-netinst.iso -boot d -m 512

https://en.wikipedia.org/wiki/Vmlinux

https://wiki.debian.org/QEMU#Usage

command line option:

  • -m 2G: RAM
  • -smp 4: VCPUs
  • -hda fs.raw: filesytem
  • -hdb fs2.raw: the second filesytem, (or hdc, hdd)
  • -nographic: affects what is done with video output and changes the destination of the serial and parallel port data.
  • -display none: affects what is done with video output
  • -serial dev: Redirect the virtual serial port to host character device dev.
    • stdio [Unix only] standard input/output; use -serial mon:stdio to keep Ctrl+C working
    • tcp:[host]:port[,server][,nowait][,nodelay][,reconnect=seconds]
    • pty: [Linux only] Pseudo TTY (a new PTY is automatically allocated)
    • and others
  • -s: enables the gdb stub
  • -S: instructs QEMU to stop after system restart. Waits for gdb to connect
  • -kernel bzImage: Use bzImage as kernel image. The kernel can be either a Linux kernel or in multiboot format.
  • -append cmdline: Use cmdline as kernel command line
  • -boot d: Specify boot order drives as a string of drive letters. Valid drive letters depend on the target architecture. The x86 PC uses: a, b (floppy 1 and 2), c (first hard disk), d (first CD-ROM), n-p (Etherboot from network adapter 1-4), hard disk boot is the default.
  • -monitor dev: Redirect the monitor to host device dev (same devices as the serial port). example: -monitor telnet:127.0.0.1:55555,server,nowait. connect to it by running telnet 127.0.0.1 55555

Kernel boot command line option:

  • console=ttyS0: write output to the terminal
  • rw: Mount root device read-write on boot
  • nokaslr: When CONFIG_RANDOMIZE_BASE is set, this disables kernel and module base offset ASLR (Address Space Layout Randomization).
  • acpi=off: disable ACPI (Advanced Configuration and Power Interface) if default was on

Examples:

qemu-system-x86_64 -s -S -no-kvm -kernel arch/x86/boot/bzImage \
  -hda /dev/zero -append "root=/dev/zero console=ttyS0 nokaslr"\
  -serial stdio -display none

qemu-system-x86_64 -kernel arch/x86/boot/bzImage \
  -boot c -m 2049M -hda ../buildroot/output/images/rootfs.ext4 \
  -append "root=/dev/sda rw console=ttyS0,115200 acpi=off nokaslr" \
  -serial stdio -display none

qemu-system-x86_64 -kernel ./linux-4.19.101/arch/x86/boot/bzImage \
  -boot c -m 256M -hda ./rootfs.ext2   \
  -append "root=/dev/sda rw console=ttyS0,115200 acpi=off nokaslr" \
  -display none -serial mon:stdio -s\
  -monitor telnet:127.0.0.1:55555,server,nowait

ctrl+a + :

  • c: enter control plane
  • x: exit

qemu monitor:

  • qtree: show the bus
  • mtree: memory-mapped I/O
  • network
  • pci

qemu-img

convert from a sparse file into the qemu’s own “Copy On Write” image. This conversion will save the same space and still be runnable

qemu-img convert -c debian.img -O qcow debian_recompressed.img

  • qemu-img create -f raw fs.raw 8G

https://wiki.qemu.org/Documentation/CreateSnapshot

KVM

It has advantage of performance with KVM if the architecture is same with host and qemu guest.

Create Root file system

initial ramdisk (initrd): a temporary root file system that contains essential modules that kernel need to get to the rest of the hardware, and that is used by the Linux kernel during initialization process while other filesystems are not mounted.

minimal

https://github.com/ivandavidov/minimal

a tiny educational Linux distribution, which is designed to be built from scratch by using a collection of automated shell scripts.

Buildroot

Buildroot is a simple, efficient and easy-to-use tool to generate embedded Linux systems through cross-compilation. It can automate the process of generating initrd.

git clone git://git.buildroot.net/buildroot
cd buildroot
make menuconfig
  • Target Options -> Target Architecture
  • Choose ext4 file system to be mounted as default file system for the system.
  • Target packages -> Networking applications -> openssh
make

You can input root as user to login buildroot shell.

To set up the network, add these lines to /etc/network/interfaces:

auto eth0
iface eth0 inet dhcp

ready-to-use Debian

Download from debian.org. Boot with root=/dev/sda1.

  - Root password:  root
  - User account:   user
  - User password:  user

Install by Image (Debian / Ubuntu / …)

qemu-img create -f qcow2 fs.qcow 2G
qemu-system-x86_64 -hda fs.qcow -cdrom img.iso -boot d -m 512

After installation:

qemu-system-x86_64 -hda fs.qcow -m 512

GDB

set architecture i386:x86-64:intel
file ./vmlinux
target remote :1234
apropos lx

https://01.org/linuxgraphics/gfx-docs/drm/kernel-hacking/index.html

The symbolic lookup table is at /boot/System.map.

Q&A:

Networking

User Networking (SLIRP):

guest (10.0.2.15)  <--+--->  Firewall/DHCP server <-----> Internet
                      |          (10.0.2.2)
                      |
                      +--->  DNS server (10.0.2.3)
                      |
                      +--->  SMB server (10.0.2.4)

How to get SSH access to a guest

-device e1000,netdev=net0 \
-netdev user,id=net0,hostfwd=tcp::5555-:22

ssh root@localhost -p 5555

add these line to etc/ssh/sshd_config:

PermitRootLogin yes
PermitEmptyPasswords yes