Logging

2025 03 01 head

Embedded systems are hard to debug.

You do not have the luxury of connecting up most of the time with a monitor and keyboard.

Stepping through code is not much use when the application needs to continually respond to real-world events for the bug to appear in the first place.

And after a while, toggling your single GPIO LED just does not cut it, and you decide you need to be able to log messages and print values.

This is typically where printf() and a trusty UART are employed, spitting messages out to a computer via an FTDI cable, virtual COM port over USB, or similar.

As a personal choice, we tend not to use debuggers beyond getting a stack trace or the value of a variable or two. One reason is that it is easy to get lost in details of complicated data structures and control flow.

We find stepping through a program less productive than thinking harder and adding output statements and self-checking code at critical places. Clicking over statements takes longer than scanning the output of judiciously-placed displays.

It takes less time to decide where to put print statements than to single-step to the critical section of code, even assuming we know where that is. More important, debugging statements stay with the program; debugging sessions are transient.

— Brian W. Kernighan and Rob Pike
The Practice of Programming

This page aims to discuss the finer details about logging on embedded systems, and cover some interesting topics such as deferred logging.

Logging in Distributed Systems

Logs are supposed to be a consistent and reliable source of information which makes it easier for us to troubleshoot issues in our systems. A standard log file structure that is consistent across all log files is required for good logging. Each log statement line should reflect a single event and include the timestamp, hostname, service, and project logger name, among other things.

Good Practice is:

  • Care that your messages are machine parseable.

  • The format should be documented and enforced. All records should have a similar structure and provide all mandatory fields.

  • Use a standard date and time format such as ISO8601.

  • Write all your log messages in English.

  • Define what is the log level for a class of log record.

Both incoming and outgoing data of your application should be logged.

Severity Levels

Use the regular levels. All software engineers know these levels and their meaning. Avoid adding your own level. If you do, please document them clearly.

Asynchronous Systems

Beware that context such an HTTP session identifier will be lost in asynchronous systems. You are on your own. Try using IP address to help you filter the logs.

Conditional Flags

Conditional flags are used to slightly optimize the application when specific logging requests are not needed.

Logs Management

Management of the records is defined to store the most relevant information in a limited storage space.

The standard log levels are:

FATAL

Identifies extremely serious error events that are likely to cause the program to abort. Typically, this leads to catastrophic failures.

ERROR

Identifies error events that may still let the software run, but with restricted capabilities in the impacted routes.

WARN

Describes events that are less destructive than errors. They usually do not result in any reduction of the program’s functionality or its full failure. They are, however, red signs that must be investigated.

INFO

In the program behavior, it denotes the major event banners and informational messages.

DEBUG

Denotes particular and detailed data, which is mostly used for debugging. These logs assist us in debugging the code.

TRACE

To provide the greatest information on a specific event/context, it denotes the most low-level information, such as stack traces of code. These logs allow us to examine variable values as well as complete error stacks.

Java log4j2 Library

Apache Foundation log4j2 [1] defines a modern logging library based on the learnings of log4j1.x and slf4j.

If part of your application is written in Java, you should use the log4j2 library without hesitation.

It is quite reasonable to write soft real-time applications in Java. Use a raspeberry pi or a rasperberry pi zero to run your Java application. The limit would be open loop control systems with a cycle time bellow 1ms.

Therefore, job floor systems or home automation solutions are good candidates for Java.

C++ Library gogl

Google Log glog [2] defines a series of macros that simplify many common logging tasks.

  • log messages by severity level

  • control logging behavior from the command line.

  • log based on conditionals.

  • abort the program when expected conditions are not met.

  • introduce your own verbose logging levels

  • customize the prefix attached to log messages, and more.

As a C++ developer, you should consider the logging framework of the Boost library.

The boost library is the tremendeous advantage to have an active community and a huge user community. The Boost organization and wider Boost community supports research and education into the best possible uses of C++ and libraries developed for it, particularly, but not exclusively those contained in the Boost Library.

ROS-2 Logging

The logging subsystem in ROS 2 aims to deliver logging messages to a variety of targets, including:

  • To the console (if one is attached)

  • To log files on disk (if local storage is available) -To the /rosout topic on the ROS 2 network

By default, log messages in ROS 2 nodes will go out to the console on stderr, to log files on disk, and to the /rosout topic on the ROS 2 network. All targets can be individually enabled or disabled on a per-node basis.

Lessons Learnt

You shall define a logging concept for your product. Enforce the log record format in all packages. The language shall be English International.

Allocate space in your embedded system for logging information. You need some persistent memory area that will store the log record using a round-robin scheme.

Verify that your log files can programmatically be parsed and filtered.

References