This blog is about the whole procedure of running riscv on qemu. It also records some pitfalls I encountered.

Preparation

If you ever downloaded source code from GitHub, you must have experienced the desperation that the download speed is only 50kb/s. Moreover, downloads are interrupted from time to time due to network problems.

So, you need git proxy to solve this problem. Please refer to my last blog(最终发现没啥用…)

Possible problem

When I git clone riscv-gnu-toolchain from GitHub, it was often stuck while receiving objects.(usually 99%…)

Someone says that ‘VMware running on NAT has this problem. Changing it to Bridged will fix the issue.’[1]

However, the same problem appeared on my host machine. Finally, I failed to fix this issue.

There are two possible solutions:

  1. Wait until it continues.
  2. Interrupt and download again.

The first one doesn’t make sense for me and the second one is to try your luck…

I have tried to download it again and again but fail.

Finally, I asked a senior studying in Japan to help me clone it and send it to me.

Fig 1 Envy
# Get riscv-gnu-toolchain

Prerequisites

1
sudo apt-get install autoconf automake autotools-dev curl python3 libmpc-dev libmpfr-dev libgmp-dev gawk build-essential bison flex texinfo gperf libtool patchutils bc zlib1g-dev libexpat-dev git

To make sure you have git on your system before the next step. Other standard packages are needed to build the toolchain.

1
2
cd ~/my-riscv
git clone --recursive https://github.com/riscv/riscv-gnu-toolchain

Because there are many submodules in this repository, so we need to add --recursive to fetch the submodules automatically.

Installation(Newlib)

Make

Pick an install path by using --prefix

1
2
./configure --prefix=/home/clickmouse/my-riscv/opt/riscv
make

You will get riscv-64-unknown-elf-gcc and its cousins in opt/riscv. The c runtime library is Newlib.

Specfication

Newlib is a runtime c standard library.

A runtime library (RTL) is a set of low-level routines used by a compiler to invoke some of the behaviors of a runtime environment, by inserting calls to the runtime library into compiled executable binary.[2]

A runtime library is needed by any c program. Functions such as _read(), _write(), _getpid() and so on are implemented in runtime library.

Test

1
$ ./riscv64-unknown-elf-gcc -v

You will get a lot of information about riscv64-unknown-elf-gcc.

Some useful information is listed below.

1
2
3
4
5
--target=riscv64-unknown-elf-gcc #tool is riscv64-unknown-elf-gcc
--with-newlib #runtime lib --> newlib
--with-abi=lp64d #The abi(Application Binary Interface)supported by the toolchain is lp64
--with-arch=rv64imafdc #architecture supported is rv64imafdc.
#imadfdc are names of ISA base and extensions

Installation(Linux)

Pick an install path by using --prefix

1
2
./configure --prefix=/home/clickmous/my-riscv/opt/riscv-linux
make

You will get riscv-64-unknown-linux-gcc and its cousins. The c runtime library is glibc.

Installation(Others)

Please refer to riscv各种版本gcc工具链编译与安装[3]

RISC-V QEMU

Make

current directory: riscv-gnu-toolchain

1
2
3
4
cd qemu
./configure --target-list=riscv64-softmmu #set target list (default: build everything)
 make -j$(nproc)
make install

If Errors are like this

1
2
3
install -d -m 0755 "/usr/local/share/qemu"
install: cannot change permissions of ‘/usr/local/share/qemu’: No such file or directory
Makefile:828: recipe for target 'install-datadir' failed

Please run sudo make install

You can test by running following commands

1
2
3
$ qemu-system-riscv64 -version
QEMU emulator version 4.0.50 (v4.0.0-1854-g57dfc2c4d5-dirty)
Copyright (c) 2003-2019 Fabrice Bellard and the QEMU Project developers

Specification

The RISC-V QEMU port supports the following targets:[4]

  • riscv32-softmmu (RV32 full-system emulator)
  • riscv64-softmmu (RV64 full-system emulator)
  • riscv32-linux-user (RV32 linux user-mode emulator)
  • riscv64-linux-user (RV64 linux user-mode emulator)

Tips: In this step, you may need to install some libraries. You can find related library package name you need use sudo apt-cache search package-name and then use sudo apt install package-name. For example, if you type sudo apt-cache search gcc, then you will get a series of packages with a string of gcc in its name.[5]

Linux source code

Download

It is not an essential step if you have Linux source code on your system.

To finish the whole procedure of running riscv on qemu, I downloaded a new copy of Linux source code.

1
wget http://ftp.sjtu.edu.cn/sites/ftp.kernel.org/pub/linux/kernel/v5.x/linux-5.5.tar.xz

If you want to download it to specific directory, please add -P dir after wget.

The second step is to decompress the source code.

1
2
xz -dkv linux-5.5.tar.xz #d:decompression. k:keep the Compressed file. v:show the procedure
tar -xvf linux-5.5.tar

It may consume some time and don’t forget to add -v. Or you can only use top to know it is running(hhh).

Make(do this after building Busybear)

1
2
3
4
5
cd linux
cp ~/my-riscv/busybear-linux/conf/linux.config .config 
make ARCH=riscv CROSS_COMPILE=riscv64-unknown-linux-gnu- olddefconfig
make ARCH=riscv CROSS_COMPILE=riscv64-unknown-linux-gnu- menuconfig #if needed
make ARCH=riscv CROSS_COMPILE=riscv64-unknown-linux-gnu- vmlinux -j $(nproc)

Busybear

Prerequisites

riscv-64-unknown-linux-gcc

Make

Run this command to get the source code of busybear

1
git clone https://github.com/michaeljclark/busybear-linux

By the way, you can use this command to download riscv-pk which is needed next part in busybear-linux directory

1
git clone --recurisve https://github.com/michaeljclark/busybear-linux

If you don’t want busybear to download linux source code here, you need to modify the shell file.

1
2
cd scripts
vi build.sh

You can delete all lines about downloading, extracting and building linux. If you are not clear how to delete, please replace your build.sh with build.sh in Appendix of this blog.

Compile them and prepare a root filesystem image named busybear.bin

1
2
cd busybear-linuxx
make

Tips: It will cost much time. But don’t use make -j4 or make -j$(nproc) because build.sh will download something from Internet. Use multi-processor will probably result into connection error.

Specification

busybear-linux is a RISC-V Linux root filesystem image that targets the virt machine in riscv-qemu.[6]

riscv-pk

Make

Get the source code of riscv-pk

1
git clone https://github.com/riscv/riscv-pk

Build BBL

Currently, you are in riscv-pk. And riscv-gnu-toolchain is added to $PATH

1
2
3
mkdir build && cd build
../configure --enable-logo --host=riscv64-unknown-elf --with-payload=~/build_kernel/linux-5.5/vmlinux
make

Specification

We only need bbl. So, make install is not necessary.

bbl is a supervisor execution environment for tethered RISC-V systems.(I haven’t figure it out yet)

Run[7]

Run this command

1
2
3
4
5
qemu-system-riscv64 -nographic -machine virt \
-kernel ~/my-riscv/riscv-pk/build/bbl \
-append "root=/dev/vda ro console=ttyS0" \
-drive file=~/my-riscv/busybear-linux/busybear.bin,format=raw,id=hd0 \
-device virtio-blk-device,drive=hd0

username: root

password: busybear

Fig.2 Run riscv on qemu

If you want to shutdown QEMU. There are two methods.

  1. ctrl+A+X but it doesn’t work on my system.
  2. ps aux|grep qemu find the pid of qemu-riscv64. kill pid

Debug[8]

Run this command

1
2
3
4
5
6
qemu-system-riscv64 -nographic -machine virt \
-kernel ~/my-riscv/riscv-pk/build/bbl \
-append "root=/dev/vda ro console=ttyS0" \
-drive file=~/my-riscv/busybear-linux/busybear.bin,format=raw,id=hd0 \
-device virtio-blk-device,drive=hd0
-S -s

Open qemu and wait for gdb to attach

Open another terminal and run this command

1
2
riscv64-unknown-linux-gnu-gdb ~/build_kernel/linux5.5/vmlinux
(gdb) target remote localhost: 1234 #attach

Acknowledgement

Chao Liang helps me download the source code of riscv-gnu-toolchain.

Reference

[1] stuck while receiving objects

[2] Runtime library

[3] riscv各种版本gcc工具链编译与安装

[3] Risc-V QEMU

[5] Download essential library

[6] busybear-linux

[7] riscv-getting-started

[8] how to debug

Appendix

build.sh

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
#!/bin/bash

set -e

. conf/busybear.config

#
# test environment
#
for var in ARCH ABI CROSS_COMPILE BUSYBOX_VERSION \
    DROPBEAR_VERSION LINUX_KERNEL_VERSION; do
    if [ -z "${!var}" ]; then
        echo "${!var} not set" && exit 1
    fi
done

#
# find executables
#
for prog in ${CROSS_COMPILE}gcc sudo nproc curl openssl rsync; do
    if [ -z $(which ${prog}) ]; then
        echo "error: ${prog} not found in PATH" && exit 1
    fi
done

#
# download busybox, dropbear and linux
#
export MAKEFLAGS=-j4
test -d archives || mkdir archives
test -f archives/busybox-${BUSYBOX_VERSION}.tar.bz2 || \
    curl -L -o archives/busybox-${BUSYBOX_VERSION}.tar.bz2 \
        https://busybox.net/downloads/busybox-${BUSYBOX_VERSION}.tar.bz2
test -f archives/dropbear-${DROPBEAR_VERSION}.tar.bz2 || \
    curl -L -o archives/dropbear-${DROPBEAR_VERSION}.tar.bz2 \
        https://matt.ucc.asn.au/dropbear/releases/dropbear-${DROPBEAR_VERSION}.tar.bz2
:<<!
test -f archives/linux-${LINUX_KERNEL_VERSION}.tar.gz || \
    curl -L -o archives/linux-${LINUX_KERNEL_VERSION}.tar.gz \
        https://git.kernel.org/torvalds/t/linux-${LINUX_KERNEL_VERSION}.tar.gz
!

#
# extract busybox, dropbear and linux
#
test -d build || mkdir build
test -d build/busybox-${BUSYBOX_VERSION} || \
    tar -C build -xjf archives/busybox-${BUSYBOX_VERSION}.tar.bz2
test -d build/dropbear-${DROPBEAR_VERSION} || \
    tar -C build -xjf archives/dropbear-${DROPBEAR_VERSION}.tar.bz2
:<<!
test -d build/linux-${LINUX_KERNEL_VERSION} || \
    tar -C build -xzf archives/linux-${LINUX_KERNEL_VERSION}.tar.gz
!

#
# set default configurations
#
cp conf/busybox.config build/busybox-${BUSYBOX_VERSION}/.config
#cp conf/linux.config build/linux-${LINUX_KERNEL_VERSION}/.config

#
# build busybox, dropbear and linux
#
test -x build/busybox-${BUSYBOX_VERSION}/busybox || (
    cd build/busybox-${BUSYBOX_VERSION}
    make ARCH=riscv CROSS_COMPILE=${CROSS_COMPILE} oldconfig
    make ARCH=riscv CROSS_COMPILE=${CROSS_COMPILE} -j$(nproc)
)
test -x build/dropbear-${DROPBEAR_VERSION}/dropbear || (
    cd build/dropbear-${DROPBEAR_VERSION}
    ./configure --host=${CROSS_COMPILE%-} --disable-zlib
    make -j$(nproc)
)
:<<!
test -x build/linux-${LINUX_KERNEL_VERSION}/vmlinux || (
    cd build/linux-${LINUX_KERNEL_VERSION}
    make ARCH=riscv CROSS_COMPILE=${CROSS_COMPILE} olddefconfig
    make -j$(nproc) ARCH=riscv CROSS_COMPILE=${CROSS_COMPILE} vmlinux
)
!

#
# build bbl
#

:<<!
test -d build/riscv-pk || mkdir build/riscv-pk
test -x build/riscv-pk/bbl || (
    cd build/riscv-pk
    ../../src/riscv-pk/configure \
        --host=${CROSS_COMPILE%-} \
        --with-payload=../linux-${LINUX_KERNEL_VERSION}/vmlinux
    make -j$(nproc)
)
!

#
# create filesystem image
#
sudo env PATH=${PATH} UID=$(id -u) GID=$(id -g) \
./scripts/image.sh