Pyromaniac

Pyromaniac

Introduction

RISC OS Pyromaniac is an alternative implementation of RISC OS for non-ARM systems. It provides a semi-hosting system within which RISC OS binaries (utility, absolute, and modules) may be loaded and run, within an operating system environment which matches RISC OS Classic very closely.

RISC OS Pyromaniac is:

Intended uses

Pyromaniac is intended to provide a way of developing and testing RISC OS tools within a development or automation environment, which RISC OS Classic is not suited for. To this end it is intended to be highly configurable:

Supported features

At a very high level, there is limited support for RISC OS in general. Quick high level summary:

A more detailed feature summary can be found in the Features document.

Installation

There exist multiple distributions of Pyromaniac, which have slightly different properties. All operate in similar ways, and it is just the delivery and environment that differ.

Application distribution

The Windows and macOS (OSX) distribution includes an application which has a Python 2.7 and the optional components installed (except for GTK). The application distribution is intended for ease of use in those environments. There is currently no Linux application distribution.

Certain differences exist in the use of these distributions:

macOS:

Windows:

Note: This will change in future, and the two application distributions will align.

Docker distribution

The docker distribution is intended for use in test environments where interaction with the host is not necessary, and isolation is important. Different docker distributions exist with different tools and enviroment installed. Consult the docker image documentation for the specifics of the configuration of each container, but the typical invocation would be:

docker run -it --rm -v "$PWD:/home/riscos/fs" docker-registry.gerph.org/gerph/pyromaniac pyro <options>

At the current time the following containers can be provided:

Package distribution

The 'package' distribution for macOS and Linux is an archive which can be unpacked into the $PATH. This includes the Unicorn distribution, but otherwise uses the existing Python 2.7 installation. Packages may need to installed by the user, as for the source installation.

The usual invocation with the package distribution would be:

pyro <options>

Source installation

Pyromaniac is a Python 2.7 application, so an installation of Python 2.7 is required. This has been tested on Windows 10, macOS, and Ubuntu Linux. It has been tested on Alpine linux, but difficulties in setting up the environment mean that this is not currently a supported environment. BSD systems are not currently supported.

In order to use Pyromaniac you will need to install Unicorn Engine. On macOS and Linux this can be achieved with:

git clone -b cjf-dev https://github.com/gerph/unicorn.git
cd unicorn
UNICORN_ARCHS="arm" ./make.sh
cd bindings/python
python setup.py install

Note: The forked repository is required because there remain patches that have not been accepted upstream.

Optionally, the following Python modules may be required depending on the functionality needed:

Using it

Invoking utilities or absolute files from the host filesystem:

python pyro.py <options> -- <filename> <parameters>

Invoking modules:

python pyro.py <options> --load-module <module-filename> --enter-module <module-name> -- <parameters>

Invoking commands (which may include files on the RISC OS filesystem):

python pyro.py <options> --command <command> -- <parameters>

Booting from the RISC OS filesystem:

python pyro.py <options> --config pyro.system_boot=true

Configuration

RISC OS Pyromaniac is intended to be used to debug behaviour in a way that RISC OS Classic cannot do as easily. One way in which RISC OS Pyromaniac is different to RISC OS Classic is that can be reconfigured in many different ways easily without rebuild, and sometimes without even restarting.

The configuration system is described in separate configuration documentation.

Debugging

A number of debug options are available to help work out what's gone wrong within the system. Two switches are provided for controlling the debug options:

The switch --list-debug can be used to show the debug options which are available. Some of the common options you might use are:

Within Pyromaniac itself, the *PyromaniacDebug command can be used to control the debug in use, which may be additive by prefixing options with a +.

Implementation information

The functionality supported by default are limited to just those SWIs and commands provided by the Kernel, and a few modules which have been stubbed. This is only useful in very limited cases where the rest of the OS is not required.

The SWIs and commands supported may be extended by loading the supplied Python modules with --load-internal-modules as described above. The SWIs can be listed by running the tool with the --list-swis switch:

python pyro.py --list-swis

The existance of a SWI in the list does not mean that is functional. It only means that there is a registered handler for that SWI, although usually this provides a good indication that there is some implementation behind it.

Some SWIs use other reason codes to sub-divide their operations. These are grouped internally as handlers. For example OS_Byte calls have registered handlers which allows them to be extended more easily. The groups which have handlers can be listed with:

python pyro.py --list-handlers

The registered handlers, or all handlers for each group, can be listed with:

python pyro.py --list-registered <group>
python pyro.py --list-handler <group>

When listed with --list-handler, those which are registered are marked with a +.

The commands which are known within modules can be listed with one of (without the internal modules loaded, only the UtilityModule will be present):

python pyro.py --list-commands
python pyro.py --load-internal-modules --list-commands

Some modules can provide different functionality for their operation, either because the RISC OS Classic behaviour is not appropriate, or because different facilities are available within RISC OS Pyromaniac. For example, the Hourglass offers a number of different ways of working - it can provide the pointer that we know from RISC OS Classic, or it can report the state through the VDU system. The latter is more appropriate for use when no display is present.

The implementations registered are described with:

python pyro.py --list-implementations

RISC OS Pyromaniac does not have any 'legacy' mappings of memory. All memory areas are managed centrally through the OS_DynamicArea interfaces. The memory map registered with OS_DynamicArea can be shown with:

python pyro.py --list-memory

Internally, 'resources' provide the equivalent of hardware or internal state, which the system manages. These resources are able to be communicated with by different modules, and provide effectively named interfaces for communication of non-RISC OS module components. You might think of resources like hardware (like the PCF8583 chip used for configuration), or internal parts of the Kernel which are shared (like the error system, which turns Python errors into RISC OS errors). These resources can be shown with:

python pyro.py --list-resources

System lifecycle

The system will always initialise modules, and perform certain initialisation time operations, similar to those performed by RISC OS Classic. This startup includes (but is not restricted to):

  1. Service_PostInit
  2. Mode selection (--config kernel.reset_modechange)
  3. System banner (--config kernel.reset_banner)
  4. Service_Reset
  5. Boot bell (--config kernel.reset_bell)

Following these initialisation operations, the supplied invocation options will then be run. Command line options for invocation (--enter-module, system boot, --command, host binaries) are added after the configuration options read from the configuration file. Although the executed actions in the configuration file may appear in any order, the options supplied on the command line are ordered according to their type, rather than the position on the command line. If you need to mix different types of invocation actions the --action switch allows actions to be used on the command line.

The command line options will be executed in the following order (after those execute actions specified in the configuration file):

  1. --enter-module (only one module may be specified)
  2. --command (multiple command may be specified and will be executed in order)
  3. Host binary execution (only one host binary may be specified)
  4. System boot configuration (if pryo.system_boot is True then the system boot actions will be performed).

If the system is rebooted (and does not power off; --config pyro.reset_effect), it goes through a similar sequence of operations to RISC OS Classic.

  1. Service_PreReset
  2. Clear all memory and CPU state.
  3. Load modules.
  4. Initialise as above

After the initialisation has been performed, any unexecuted actions will continue. If there are no further actions and the system is configured with pyro.system_boot, the system boot will be triggered. Any action which reports an error will cause the system to exit. If pyro.stop_on_failing_rc is enabled, the value of Sys$ReturnCode will be checked and non-0 values will cause the system to exit.

Once all the actions which have been configured are completed and exited, the system will shut down. The shutdown sequence is similar to that which happens on Ctrl-Break on RISC OS Classic

If the emulation.kernel_shutdown option is set to True, a Service_PreReset is issued. This is the default, and is able to be disabled if it's known that problems occur during this service.

After the Kernel has shut down, resources will finalise, which allows them to perform any cleanup operations that they need. Examples of such clean up might be:

System boot

System boot can be configured a number of different ways, and once triggered, different configuration options take place.

Execution speed

As Pyromaniac is powered by the QEMU engine, it can execute ARM quite quickly. On a MacBook Pro (late 2013), with a 2.6GHz i7, the timings show about 6100 MIPS if the QEMU engine is left to run ARM code. However, interaction with the host system through SWIs is slower - only 0.1 MIPS if the instructions are exclusively SWI calls.

Tracing

The purpose of Pyromaniac is to make it easier to find problems with RISC OS binaries. As such the execution of the code can be traced using the --debug trace switch. This will show the disassembly of the code as it is executed, and can be valuable in finding out how a program works (or fails).

A slightly faster trace is available with the --debug traceblockregs, which reports each 'basic block' which is executed and the registers which are different on entry to the block relative to the prior block that was reported.

In addition, there exists the --debug traceswi switch, which outputs just the SWI calls and the registers on entry and exit from them. This is comprehensive, but requires the reader to know the meaning of any given SWI.

The format output by this debug option is changed through the --debug traceswiargs switch. This changes the register dump from raw numbers to an interpretation of the parameters that the SWI takes, giving the parameters a name, and trying to decode blocks of data and strings where possible. The information for this output has been extracted from the def files provided with OSLib.

The 'region' of execution can be determined from the dynamic area in which the execution is happening, and (where relevant) the module. The region can be displayed during the trace through the switch --debug traceregion, which shows the regions in regular execution, or --debug traceswiregion, which shows the region only on the execution of a SWI.

When exceptions occur, and tracing is enabled (but not using the full disassembly), information about the recently executed code blocks is output. This should help to indicate what code was operated on recently.

The --debug tracemsr option allows specialised debugging of the MSR operations, displaying the contents of the status register when it is encountered.

Within the disassembly, some of the output is processed from Capstone to make it easier to follow the execution:

Some instructions are in non-standard RISC OS formats:

The disassembly may recognise that a block has been repeated a number of times, and turn off the full disassembly processing. This makes the execution of these repeated executions significantly faster, and reduces the size of the trace log. These repetitions will be reported within the trace once the code leaves the repeated block.

The trace system has configuration for some of its behaviour:

The syntax of the trace output is described in separate trace documentation.

Watchpoints, Tracepoints and SWI traps

The trace system allows for reporting of certain operations within the execution:

Watchpoints are configured through trace.watchpoints as a comma separated list of addresses for words that will trigger the watch. Whenever these addresses change, a report will be written to the trace output to give details about the context.

Tracepoints are configured throgh trace.tracepoints as a comma separated list of addresses whose executions will trigger the watch. The addresses can be expressed in a number of forms:

Function names may be wildcarded with * matching any number of characters, and ? matching a single character.

Tracepoint locations are recached when executable regions change (triggered by OS_SynchroniseCodeAreas), which means that they should update as new commands are loaded.

SWI are configured through trace.switraps as a comma separated list of SWI numbers, names or SWI prefixes. The names may be suffixed by a :<operation> to indicate what operation should be performed when the SWI is executed. The operations that can be performed are:

By default the SWI traps will use the report operation if none is supplied.

If a SWI prefix is given, all SWIs with that prefix will be trapped. As modules are loaded and killed, the SWI names will change, and this should cause the traps to be updated appropriately. Thus, if you are trying to debug a SWI call which happens whilst the module is not loaded and thus does not have a name, the SWI number must be used instead.

Debugging the heap

The heap operations have significant configuration options which can help with diagnostics. This is documented in the heap debugging documentation.

Information signal

If the system becomes stuck executing code and does not leave, it is possible to request that the system dump its current state. Sending the SIGUSR1 signal to the process will cause an interrupt which dumps the register state and other information. This is only available on Linux and macOS systems, as the signal is only available there.

If the system does not respond to the first request, this indicates that it has not been able to return to the Python execution thread. Usually this should not happen, but if it does, a second request will produce an information dump as an interrupt. Some of the register details may be incorrect as the system may still be executing underneath, or execution may be within a Python implementation which is not updating register state. However, it may indicate where the problem lies.

Timings and counts

The timings configuration group allows the counts of the number of calls to SWIs, and timings of execution to be tracked by the system.

Python modules

Modules may be implemented in Python. A collection of modules are provided with the system, which by default are not loaded. To use these modules, a command line switch is used to register the modules for use --load-internal-modules.

Python modules inherit from the riscos.pymodules.PyModule class. The internal modules are present in the riscos.pymods package. Additional modules, or packages of modules may be added using the switch --load-pymodule <filename|module|package>.

The UtilityModule is always initialised, even if the above switch is not supplied. The module provides the GOS command, which allows OS_CLI calls to be made.

The Python modules may access many of the RISC OS operations through the ro.kernel.api object. This object provides interfaces to some of the RISC OS interfaces using more pythonic calls. For example, RISC OS files can be opened with a simple open call, which returns a file handle implementing the Python io protocols.

It is possible to generate a template for writing a Python module from an OSLib def file. For example, to create a template wimp.py from the OSLib 'wimp' file:

utils/oslib_parser.py oslib/User/def/wimp --create-pymodule-template wimp.py

These templates will need some changes, as the generator can only infer so much (and the SWIs themselves will do nothing as generated), but this goes a long way to providing a skeleton to implement and extend.

Constants from the def file may also be exported with:

utils/oslib_parser.py oslib/User/def/wimp --create-pymodule-constants wimp.py

GTK/WxWidgets installation

Using the GTK graphics implementation allows the rendering into an application on both OS X and Linux. For OS X, it is necessary to install the GTK+3 components, or the WXWidgets:

brew install gobject3 gtk+3

or pip install wxpython

Once installed, the tool can be configured:

python pyro.py --config graphics.implementation=gtk ...

or python pyro.py --config graphics.implementation=wx ...

The UIs provided must use a 'main thread' for their dispatch. This is the default, which may be disabled with the --no-main-thread. Using this switch will prevent the UI components from initialising, but may be useful for diagnostics. It is not expected that most users need to user this option.

Command server

RISC OS Pyormaniac may be invoked through to run as a Command Server. In this form the RISC OS environment is started as a server, and commands may be issued to the command server through a TCP socket, with the output sent through the same socket. This allows scripted use of RISC OS within host build and test processes.

More information can be found in the Pyromaniac Command Server documentation.

Diagrams

There are a few diagrams within the diagrams directory, in graphviz dot format, which are supplied with the RISC OS Pyromaniac source. They are not supplied with the distributions.

These may not represent the actual operation of Pyromaniac itself, but a reference for the actual behaviour of RISC OS for certain operations. These diagrams were created as part of a process of understanding what needs to be implemented (and what can be skipped) and its implications.

A memory layout diagram is present within the diagrams directory, and can be processed using the supplied mld.sh tool, to create a SVG file alongside the diagram file:

utils/mld.sh diagrams/memory-layout.mld

Tests

There are some tests in the 'testcode' directory. They require the RISCOS toolchain to build, but are supplied with built binaries.

make tests

Test structure

The tests are held in the testcode directory. This contains both the source for the ARM utilities that are used to test the execution of Pyromaniac, and the binaries that were built from them. The testcode can be built with the command:

make testcode

This requires the riscos-objasm tool to be on the path.

The util directory contains any built utilities taken from RISC OS itself which may be used for testing. These are not included as source because the source is present in source control elsewhere.

The tests are controlled by a custom Perl script, test.pl which is used as a harness to execute the tests in a controlled manner and report the results. It is written in Perl so that it may work with the Last Good Perl which supported RISC OS filenames natively, which was a Perl 5. The tool is not quite compatible with this yet. The intention is that the tests that are executed with the tool are runnable on Linux, Darwin and RISC OS using the same Perl script - this is not so important for Pyromaniac, but is retained for the rest of the RISC OS cross-compilation toolchain.

The tests that will be run are defined in the file tests.txt, which contains the definitions of how the tool (pyro or pyro.py) should be run, and what results it should expect. Thie file may include other tests-*.txt files, which contain collections of related test groups.

Tests are grouped together by the functional area they exercise. Each group may contain multiple tests, which use the main group definition plus their own definition to define what sould be run (the Command) and what the result should be (Expect, RC). There are more commands which allow a variety of different checks to be performed - see the file prologue comment in test.pl for more detail.

Preloading Python code

Usually the interfaces provided are sufficient to work with many of the internal functions that are available. However, to test some interfaces it is necessary to rely on either platform specific Python modules or optional Python modules which won't be installed. In these cases it is often necessary to mock the interfaces so that the tests can exercise the Pyromaniac code if working as expected.

Usually such mocking would happen within unit tests, but much of the testing of RISC OS Pyromaniac uses system testing. This means it is harder to inject mocks and stub interfaces in place of the real modules. A command line option, --preload-python is provided to allow Python code to be loaded early, before PyModules are initialised. This allows system modules to be replaced or fake modules to be injected into the system under the control of the custom code.

The Raspberry Pi GPIO interface (RPi.GPIO) is tested in this way, by faking the interfaces provided by that module. For example, to use the dummy GPIO interface, with the GPIO implementation you might use:

./pyro.py --load-internal-modules --load-module modules/BASIC,ffa \
          --preload testcode/preloads/dummy_rpigpio.py \
          --config gpio.implementation=rpigpio --command BASIC

Subsequent calls to the GPIO interfaces from BASIC will pass through the GPIO module, into the rpigpio implementation, and then to the dummy modules provided by the preloaded code.

Supplied utilities

There are a number of small utilities supplied alongside RISC OS Pyromaniac, in the utils directory. The utilities that might be useful to users are:

Terminology

There are some terms used within Pyromaniac which may need clarification: