The integration of pytest with real boards (test.py) was written by Stephen Warren of Nvidia, some 9 years ago. It has certainly stood the test of time. The original code has been tweaked for various purposes over the years, but considering the number of tests added in that time, the changes are very small. Here is a diffstat for the changes up until a recent rename:
test/py/multiplexed_log.css | 11 +-
test/py/multiplexed_log.py | 133 ++++++++++---
test/py/test.py | 31 ++--
test/py/u_boot_console_base.py | 341 ++++++++++++++++++++++++++++------
test/py/u_boot_console_exec_attach.py | 40 ++--
test/py/u_boot_console_sandbox.py | 54 ++++--
test/py/u_boot_spawn.py | 212 ++++++++++++++++++---
test/py/u_boot_utils.py | 197 ++++++++++++++++++--
8 files changed, 848 insertions(+), 171 deletions(-)
When Stephen wrote the code, there was no Gitlab system in U-Boot (it used Travis). Tom Rini added Gitlab in 2019: test.py mostly just worked in that environment. One of the reasons the code has proven so stable is that it deals with boards at the console level, simply relying on shell-script hooks to talk start up and communicate with boards. These scripts can be made to do a lot of different things, such as powering boards on and off, sending U-Boot over USB, etc.
But perhaps it might be time to make a few changes. Let me give a bit of background first.
In 2020 I decided to try to get my collection of boards into some sort of lab. Picking out a board to manually test with it was quite annoying. I wrote Labman, a Python program which created various files based on a yaml description of the lab. Labman generates udev rules and an /etc/fstab file. It also creates small Python programs which know how to build U-Boot and write it to a board, including dealing with the reset/recovery sequences, SD-wire, etc. With all that in place, Tbot provides a way to get an interactive session on a board. It also provides a way to run U-Boot tests.
Early last year I decided to take another look at this. The best things about Labman were its unified lab description (including understanding how many ports each USB hub has and the address of each) and a ‘labman check’ option which quickly pointed to connection problems. The bad thing about Labman was…well, everything else. It was annoying to re-run the scripts and restart udev after each lab change. The Python code-generation was a strange way of dealing with the board-specific logic.
Tom Rini suggested looking at Labgrid. After a bit of investigation, it looked good to me. The specification of hubs is somewhat primitive and the split between the exporter and the environment is confusing. But the structure of it (coordinator, exporters and clients) is much better than Labman. The approach to connecting to boards (ssh) is better as well, since it starts ser2net automatically. Labman is a thin layer of code over some existing services. Labman is much better designed.
So overall I was pretty enthusiastic and set to work on creating an integration for U-Boot. So I can again build U-Boot, write it to a board and start it up with a simple command:
ellesmere:~/u$ ub-int rock5b
Building U-Boot in sourcedir for rock5b-rk3588
Bootstrapping U-Boot from dir /tmp/b/rock5b-rk3588
Writing U-Boot using method rockchip
DDR 9fa84341ce typ 24/09/06-09:51:11,fwver: v1.18
<...much unfortunate spam from secret binaries here...>
U-Boot Concept 2025.01-rc3-01976-g290829cc0d20 (Jul 20 2025 - 20:10:36 -0600)
Model: Radxa ROCK 5B
SoC: RK3588
DRAM: 4 GiB
Core: 362 devices, 34 uclasses, devicetree: separate
MMC: mmc@fe2c0000: 1, mmc@fe2d0000: 2, mmc@fe2e0000: 0
Loading Environment from nowhere... OK
In: serial@feb50000
Out: serial@feb50000
Err: serial@feb50000
Model: Radxa ROCK 5B
SoC: RK3588
Net: No ethernet found.
Hit any key to stop autoboot: 0
=>
I’ve also used this integration to make my lab accessible to gitlab, so that any branch or pull-request can be tested on the lab, to make sure it has not broken U-Boot.
So, back to the topic. The Labgrid integration supports test.py and it works fine. A minor improvement is ‘lab mode’, where Labgrid handles getting U-Boot to a prompt, making it work with boards like the Beagleplay, which has a special autoboot message.
But the test.py interface is (at last) showing its age. It’s only real interface to Labgrid is via the u-boot-test-console script, which just runs the Labgrid client. Some tests restart the board, perhaps because they boot and OS or do something destructive to the running U-Boot. This results in U-Boot being built again, flashed to the board again and started again. When something breaks, it could be a lab failure or a test failure, but all we can do is show the output and let the user figure it out. The current lab works remarkably well given its fairly basic setup, but it is certainly not reliable. Sometimes a board will fail a test, but trying it again will pass, for example.
So I am thinking that it might make sense to integrate test.py and Labgrid a little more closely. Both are written in Python, so test.py could import some Labgrid modules, get the required target, start up the console and then let the tests run. If a test wants to restart, a function can do this in the most efficient and reliable way possible.
This might be more efficient and it might also provide better error messages. We would then not need the hook functions for the Labgrid case.
