Thanks to Clément Deschamps `<clement.deschamps@greensocs.com>` for this nice tutorial!

Activity 1 - Build a Simple System

Overview:

Create a simple virtual platform
- Implement a simple memory (with DMI support)
- Add a CPU
- Add a UART
Build and run a simple firmware
Build more advanced firmware
- Add a libc and run Dhrystone
- Build and run Linux
- Create a minimal filesystem using Builroot

1. Presentation

During this course, we will build the following simple virtual platform.

We will then use it to explore some of the features of SystemC and QBox.

The virtual platform will contain the following components:

ARM Cortex A57 CPU (provided by QBox)
PL011 UART (source code provided)
Simple memory (skeleton provided)

2. Create a simple virtual platform

2.1 Prepare environment

This tutorial requires either Debian (8 or 9), or Ubuntu (16.04 or 18.04).

If you are not using one of these distributions, or if you want a clean environment, you can create a temporary docker container with:

docker run --name gs-tuto -d -i -t --cap-add=SYS_PTRACE ubuntu:18.04
docker attach gs-tuto
<ENTER>
apt update && apt install sudo
cd $HOME

Install required Ubuntu packages:

sudo apt install -y --no-install-recommends \
     bison flex cmake wget git htop device-tree-compiler socat gcc-aarch64-linux-gnu\
     libboost-dev libboost-filesystem-dev libfdt-dev libglib2.0-dev libgtk-3-dev liblua5.2-dev \
     libncurses5-dev libpixman-1-dev python-dev swig texi2html zlib1g-dev tree vim gdb bc

Build SystemC 2.3.1a and install it in /opt/systemc-2.3.1a

wget http://www.accellera.org/images/downloads/standards/systemc/systemc-2.3.1a.tar.gz
tar -xf systemc-2.3.1a.tar.gz
sed -i -e 's/auto_ptr/unique_ptr/g' systemc-2.3.1a/src/sysc/packages/boost/get_pointer.hpp
sed -i -e '/using std::gets/d' systemc-2.3.1a/src/systemc.h
mkdir systemc-2.3.1a/objdir
pushd systemc-2.3.1a/objdir
../configure --prefix=/opt/systemc-2.3.1a
make -j$(nproc)
sudo make install
popd

2.2 Implement a simple Memory

The objective is to implement a very simple TLM memory, supporting Loosely Timed (LT) b_transport, and Direct Memory Access (DMI).

The memory will be tested with a MemoryTester component that we provide.

Please download the project skeleton from:

wget https://darwin.greensocs.com/s/YF9Bk3yFEdyEqnN/download -O hands-on-vp.tar.gz
tar xvf hands-on-vp.tar.gz

The project contains the following tree:

cmake/ folder contains a script for finding SystemC
CMakeLists.txt is the cmake description of the project
src/main.cc contains the sc_main() that instantiates a Memory and a MemoryTester
src/memory.h is the skeleton of the memory you need to implement
src/memory-tester.h is a sc_module for testing the memory

We provide a SystemC module named MemoryTester that will access your memory through TLM transactions, and through DMI.

Build the project

The build process is based on CMake, and takes place in two stages. First, standard build files are created from configuration files. Then the platform’s native build tools (gcc, binutils, …) are used for the actual building.

cd hands-on-vp
mkdir build
cd build
cmake -DSYSTEMC_PREFIX=/opt/systemc-2.3.1a ..

And finally, build using make command.

This will generate an executable file named vp.

Run

Simply execute the vp binary file.

You should see an error as the Memory component is not fully implemented (it’s only a stub):

Implement LT transport

Implement a b_transport function in memory.h, and register it on the socket.

Note: a good memory would support endianness, and take notice of the streaming width. In our case, we will keep things simple and support 1, 2, 4 and 8 byte words only. We’ll be running a Little Endian machine on a Little Endian host, so you can ignore Endianness, and we wont worry about streaming width.

Once you’ve finished implementing the b_transport, re-build the platform with make and run it.

If the memory test passes, you should see a success, which means your memory is working!

Take a look at the memory-tester.h code, to see what it’s testing in your memory. One of the test if a full memory write followed by a full memory read.

Now measure the time it takes to run your virtual platform with the following command:

time ./vp

Note the time somewhere, we will compare with the DMI approach later.

Implement DMI

Now implement get_direct_mem_ptr DMI function in memory.h, and register it on the socket.

Don’t forget to set the DMI hint in Memory::b_transport, so the initiator knows the memory supports DMI.

Build the platform with make and run it.

You should see a speed up.

We can now construct a simple system using the memory you just implemented!

2.3 Add a CPU

Clone QBox and dependencies

First we need to clone qbox, tlm2c, greenlib and simple_cpu.

Navigate to libs directory and clone the following projects:

cd ~/hands-on-vp/libs
git clone --depth 1 https://git.greensocs.com/tlm/tlm2c.git
git clone --depth 1 https://git.greensocs.com/greenlib/greenlib.git
git clone --depth 1 https://git.greensocs.com/models/simple_cpu.git
git clone --depth 1 https://git.greensocs.com/qemu/qbox.git

It is possible, and much faster, to only clone the last commit without the full history using option --depth 1 right after the clone command.

Your libs directory should look like this:

TLM2C is the adapter between Qemu and SystemC, while Simple_cpu wraps the result and provides the actual TLM model. GreenLib contains the critical infrastructure items we’ll be using, and QBox is the actual Qemu itself.

Edit the cmake file in libs directory to tell CMake that new projects were added. Add the following lines in the currently empty libs/CMakeLists.txt

Now you can re-build the platform with themake command executed from build directory. CMake should detect the changes in the configuration and build the new subprojects you added.

Modify the platform

Edit the file src/main.cc.

Remove the MemoryTester, and add a Cortex-A53 CPU:

For instantiating the CPU you need the following include: #include "SimpleCPU/arm/armCortexA53CPU.h"

The Cortex-A53 class is called ArmCortexA53CPU. The class is templated with BUSWIDTH. We will use a BUSWIDTH of 32 during this course.

Instantiate an ArmCortexA53CPU<32>, named "cpu", and connect it directly to the memory as pictured above. The cpu socket is named master_socket (Initiator) :

ArmCortexA53CPU<32> *cpu = new ArmCortexA53CPU<32>("cpu");
cpu->master_socket(memory->socket);

It is necessary to add the dependency on this new components in the original top level CMakeLists.txt :

Build the platform to check that there is no syntax error in the code.

Configure the CPU

The CPU contains some parameters (gs_params) that you can set either programmatically or by using a Lua description. For example the number of cores is a CPU parameter named ncores.

GreenLib provides a Lua parser to simplify the configuration of the parameters.

You can find more information about Lua on lua.org.

First, we need to add the Lua parser to our code:

#include "greencontrol/config_api_lua_file_parser.h"

gs::cnf::LuaFile_Tool luareader("luareader");
if (luareader.config("../platform.lua") != 0) {
    printf("lua file not found.\n");
    exit(EXIT_FAILURE);
}

This code will load the lua file from ../platform.lua. Note the .. as we are executing our vp from ./build folder, and we want to use the platform.lua in the base directory.

Create an empty platform.lua in the base directory, and add the following content:

cpu = {
    library = "libs/qbox/build/aarch64-softmmu/libqbox-cortex-a53.so",
    kernel = "../fw/hello.elf",
    ncores = 1,
    cores = {
        { AA64nAA32 = 1, CfgEnd = 0, CfgTe = 0, Rvbaraddr = 0, Vinithi = 0 },
    },
}

The name ‘cpu’ is the name you gave to the Qbox instance. The library is the qemu library code that will be loaded, and the kernel is the file that qemu will be told to load into memory for you. We’re asking for 1 core, which we configure.

Build the platform as before, and this time run it again. The hello.elf firmware will attempt to write to a UART (which for now is not there).

To exit the simulation hit ctrl+c

Your terminal might be a bit broken because Qemu modifies a bit the configuration. If you experience the issue, use the reset command to set it back to normal.

2.4 Add a Router and a UART

Add a router

GreenLib provides a simple router implementation, we are going to use it in our platform.

Include:

#include "greenrouter/simpleRouter.h"

The router class is SimpleRouter. The router’s sockets are named init_socket (Initiator) and target_socket (Target).

Now you can:

Instantiate a router SimpleRouter<32> named router
Bind the cpu initiator socket to the router target socket
Bind the memory target socket to the router initiator socket

Once the memory is bound, we need to tell the router where the memory is mapped on the Bus, you can do this by calling the following router function right after the socket binding:

Memory *memory = new Memory("memory", 0x40000000);
router->init_socket(memory->socket);
router->assign_address(0, memory->size());

It will map the memory in the range [0, memory-size].

Add a UART

First we need to clone the pl011 project.

Navigate to libs directory and clone the following project:

https://git.greensocs.com/models/pl011.git

Your libs directory should look like this:

Add the new subproject in the CMakeLists.txt

Now you can:

Instantiate a PL011 named pl011
Bind the pl011 target socket to the router initiator socket
Bind the pl011 irq socket to the cpu irq socket

#include "PL011/PL011.h"

[...]

PL011 *pl011 = new PL011("pl011");
pl011->irq_socket(cpu->irq_socket);
router->init_socket(pl011->target_port);

Once the pl011 is bound, we need to tell the router where it is mapped on the Bus, you can do this by modifying the lua file. NOTE: This is a more elegant way of configuring the address of a component. The gs_params are within the socket (which is a greensocs tlm-2 socket), and will be automatically picked up by the router.

pl011 = {
    irq_number = 1,
    target_port = {
        base_addr = 0xC0000000,
        high_addr = 0xC0001000
    }
}

It will map the pl011 in the range [0xC0000000, 0xC0001000].

It is necessary to add the dependency on this new components in the original top level CMakeLists.txt :

Add a backend to the UART

The PL011 model needs what we call a backend in order to output the characters.

GreenLib provides a ncurses based serial backend, you can create it and connect it to the pl011 using the following code:

#include "greenserialsocket/backends/ncurses_serial_backend.h"
[...]
StdioSerial *backend = new StdioSerial("serial");
pl011->serial_sock(backend->socket);

Now re-run the platform, this time the hello.elf firmware should output a message to the UART (which you should see on the terminal).

3. Build and run a simple firmware

In this step we are going to create a simple firmware and run it on our platform.

First we need an aarch64 cross compiler in order to compile the firmware.

Downloadgcc-linaro-4.9-2016.02-x86_64_aarch64-elf.tar.xz toolchain from

https://releases.linaro.org/components/toolchain/binaries/4.9-2016.02/aarch64-elf/gcc-linaro-4.9-2016.02-x86_64_aarch64-elf.tar.xz

Extract it and add the bin folder of the extracted toolchain to your path. Check it works by issuing the command:

aarch64-elf-gcc --version

Now download the firmware skeleton from:

wget https://darwin.greensocs.com/s/kWxJm49PxWm5sYM/download -O skel.tar.gz

Explore the different files (Makefile, sources, linker script)

You need to:

initialize the stack pointer in boot.s
jump to c_entry() from boot.s
create a kernel.c file containing the c_entry() function
write a print() function, that print to the UART

Build the firmare and run it on your platform.

4. Build advanced firmware

The little hello firmware is fine for testing, but now we will move to larger applications.

For that we will use a C Standard Library (libc). The C standard library provides macros, type definitions and functions for tasks such as string handling, mathematical computations, input/output processing, memory management, and several other operating system services.

As an example we will use a baremetal dhrystone, built using libc. So first we must build libc for the target.

4.1 C standard library (libc)

In this chapter, we are hoing to see how to build Newlib.

Newlib is a C library intended for use on embedded systems. It is a conglomeration of several library parts, all under free software licenses that make them easily usable on embedded products.

Newlib is only available in source form. It can be compiled for a wide array of processors, and will usually work on any architecture with the addition of a few low-level routines.

Download Newlib version 2.5.0.201712 from Newlib website.

Extract the archive, and build newlib with:

./configure --target aarch64-elf --prefix=$PWD/prefix
make
make install

Use newlib in your firmware

We need to modify the Makefile to include libc headers and to link with libc.

The includes are in the /include.

The library to link against is libc.a and is located in /lib

Add a call to printf in your firmware, and build it.

Do you see a linker issue ?

You’re missing some function that are required for newlib to work, we’ll explain that in the next section.

System Calls

The C subroutine library depends on a handful of subroutine calls for operating system services.

We need to implement some of these subroutines.

More information here : http://www.sourceware.org/newlib/libc.html#Syscalls

Create a file named syscalls.c and add it to the sources files in your Makefile.

You need to implement:

_close
_fstat
_isatty
_lseek
_read
_write
_sbrk

For _close, _fstat, _isatty, _lseek, _read you can use the stubs provided on Newlib Syscalls page linked above.

For _write, you can use the stub provided on Newlib Syscalls page, but you’ll need to declare a subroutine named void outbyte(unsigned char b) that will output the character toward the pl011 uart.

For _sbrk, you need to write a small allocator, you could for example allocate a static byte array and use it as a memory pool, so _sbrk would return pointers inside that array.

Once you have implemented syscalls.c, you can build and run your program, the printf function provided by newlib should output to the console (where the pl011 writes).

4.2 Dhrystone

Now that we have a libc, porting an existing application is easier.

In this section, we will port the original dhrystone application.

Dhrystone is a popular benchmark program developed in 1984.

You can find more information about dhrystone here : https://en.wikipedia.org/wiki/Dhrystone

Download original dhrystone sources from:

https://darwin.greensocs.com/s/FYWqXG5MqkJfMmz

Extract the archive and add the needed dhrystone files to your Makefile.

Dhrystone uses scanf to request from the user the number of iterations to run. The scanf function provided by newlib uses the _read function we stubbed earlier in syscalls.c. It is now time to implement this _read function, by waiting for an available character in the pl011 and returning the read character.

Build. You should have a link error.

Dhrystone also uses the function gettimeofday to measure the time of execution. The gettimeofday function provided by newlib calls _gettimeofday that we need to implement in syscalls.c. If you have time, implement a SystemC component that returns the SystemC time on register read. Or you can stub this function for now (in this case the time printed at the end will be wrong).

Build and run.

Thanks to newlib, the porting of dhrystone is pretty easy.

4.3 Linux

Now that we have seen how to build a minimal libc and link a bare metal application to it, we will see how to build more complex applications such as an operating system. The advantage of running an operating system such as linux on a virtual platform is that it allows to test a lot of things (instructions, modes, locks, timers, etc …), plus linux provides a lot of tool for testing a platform. Additionnaly, there are a lot of benchmark application running on top of linux that we can then run easily.

In this section, we will build a minimal linux kernel to run on our platform.

Build the kernel

Start by downloading the latest stable kernel archive from https://www.kernel.org/.

Extract the archive, and prepare the environment:

export ARCH=arm64
export CROSS_COMPILE=aarch64-linux-gnu-

Configure the tiniest possible kernel with:

make tinyconfig

Now we need to add a few options needed for linux to print messages in our platform.

Linux provides a ncurses based interface for modifying the configuration:

make menuconfig

We need to :

Enable support for printk
Enable TTY
Enable ARM AMBA PL011 serial port support
Enable Support for console on AMBA serial port

When you have enabled these 4 options, you can exit and save the config.

You can now build the kernel with the make command.

After the compilation, you should see your kernel image in arch/arm64/boot/Image.

This is the image that we are going to load into QBox.

But first we need to write a device tree.

Device Tree

The Device Tree is a data structure for describing hardware. Rather than hard coding every detail of a device into an operating system, many aspect of the hardware can be described in a data structure that is passed to the operating system at boot time. For example the memory mapping of a platform is passed to the kernel using a device tree loaded in memory by the bootloader.

More information about Device Tree can be found here: https://www.kernel.org/doc/Documentation/devicetree/usage-model.txt

In this section we will write a Device Tree Source (dts file), that describes our platform.

Start by downloading the skeleton of the device tree:

wget https://darwin.greensocs.com/s/3otDLnbFHLrZfgt/download -O gsboard-skel.dts

Edit the dts file and set the correct memory mapping for your memory and your uart.

Then compile your dts into a dtb (linux kernel understands dtb only).

dtc -I dts -O dtb -o gsboard-skel.dtb gsboard-skel.dts

Now edit your platform.lua to point cpu.kernel to your built kernel Image, and add new parameter called cpu.dtb with the path to your dtb file.

Run the simulation, you should see your kernel start!

Your kernel should panic beause it lacks a root filesystem to execute (When linux kernel has finished the initialization, it tries to acess the root filesystem and execute a script called /init).

4.4 Buildroot

We are going to build a minimal filesystem containing busybox, and dhrystone as a userland application.

Download buildroot 2018.02.6 archive from buildroot.org

Extract the archive.

Build buildroot

Buildroot use an interface similar to linux for configuring the options:

make menuconfig

We need to :

Set the Target Architecture to AArch64 (little endian)
cpio the root filesystem (for use as an initial RAM filesystem)
- and compress it using gzip
Enable Dhrystone in Target Packages

When you have enabled these options, you can exit and save the config.

Now we can build buildroot using make command.

Buildroot outputs the rootfs in output/images/rootfs.cpio.gz.

Piggyback the rootfs on the kernel

Our platform is very simple and does not have any SD card controller or any SATA controller, which would help us mounting a filesystem.

So we are going to use the INITRAMS option of the kernel: our root filesystem will be piggybacked on the kernel Image, and loaded in memory.

Enable the INITRAMS option in your kernel, and point to the rootfs.cpio.gz you just built in the previous section.

Build, you should see that your kernel Image is a bit bigger than earlier.

Run on your platform, login as root, and run dhrystone in userland.

Congratulations you have a simple platform running a minimal linux kernel with a minimal root filesystem!

Thanks to Clément Deschamps <clement.deschamps@greensocs.com> for this nice tutorial!

Activity 1 - Build a Simple System

Overview:

1. Presentation

2. Create a simple virtual platform

2.1 Prepare environment

2.2 Implement a simple Memory

Build the project

Run

Implement LT transport

Implement DMI

2.3 Add a CPU

Clone QBox and dependencies

Modify the platform

Configure the CPU

2.4 Add a Router and a UART

Add a router

Add a UART

Add a backend to the UART

3. Build and run a simple firmware

4. Build advanced firmware

4.1 C standard library (libc)

Use newlib in your firmware

System Calls

4.2 Dhrystone

4.3 Linux

Build the kernel

Device Tree

4.4 Buildroot

Build buildroot

Piggyback the rootfs on the kernel

Thanks to Clément Deschamps `<clement.deschamps@greensocs.com>` for this nice tutorial!