PPSi (ptp Ported to Silicon) is an application which, in its basic operation, implements ieee 1588 specification in a way that is portable to several architectures, including OS-less ones.
This manual is mainly aimed at developers: people who are working with ptp and/or White Rabbit and need to use the protocol in their own hardware or software environments. Users who simply run PPSi on the host may want to read Command Line and little else.
This project originates in the context of White Rabbit, which is shortened to WR in this manual.
WR is a multi-lab research project that aims at synchronizing thousands of I/O devices distributed in a network several kilometers wide; its software protocol is an extension of ptp. A WR network is made up of two devices: the WR Switch and the WR Node. White Rabbit is developed at ohwr.org: http://www.ohwr.org/projects/white-rabbit.
The WR switch is an 18-ports Gigabit Ethernet switch that runs wr-ptp as a Linux process, synchronizing each Ethernet link as either a ptp master or a ptp slave (it is a boundary clock). The WR node is an I/O device that includes a soft core that runs wr-ptp in freestanding mode (i.e., without an operating system).
PPSi, as a software package, is designed to be the ptp implementation used in WR, running both standard ptp and wr-ptp, in all possible use cases.
The algorithm and computation routines regarding the basic ieee 1588 are derived from the PTPd project, v.2.1.0 (see AUTHORS for details about copyright); but as of March 2013 very little remains of the original code base. In addition to the basic feature set we inherited from PTPd, we support raw Ethernet frames (ETH_P_1588), according to Annex F of the specification and support for protocol extensions.
We thank Danilo Sabato for fixing the bare architectures (see Architectures).
The home page of the PPSi project and the source repository are:
http://www.ohwr.org/projects/ppsi
git://ohwr.org/white-rabbit/ppsi.git
This document tries to summarize the status of the project, but we are working a lot on the package, so information here may be slightly out of date with respect to code.
The software package is designed to be modular: each of architectures, protocols and timing engine can be replaced by providing a proper subdirectory.
After release 2013.11 we added to PPSi the Kconfig configuration engine. Configuration is now performed by running “make menuconfig” or equivalent command.
A number of ready-to use configuration files are distributed in the configs/ directory: to use any of them name it as target of make (for example: “make wrs_defconfig”).
The default configuration is unix_defconfig, that builds a Unix daemon running the standard ptp version 2 protocol.
When building PPSi, the user can specify which architecture to build for, by selecting it in make menuconfig or equivalent configuration command. When cross-compiling, you need to tell your cross-compiler prefix in the configuration step, or override it at build time by passing CROSS_COMPILE= on the commabnd line.
The package currently supports the following architectures:
We plan, over time, to support microcontrollers (a beta version for ARM7 is working, and we are considering Cortex-M) and bare-arm (to be tested on the WR switch).
At build time, the user can select the standard protocol (selected by default by most architectures) or an extension. The code base only includes one extension, White Rabbit, which is the default choice when building for one of the WR architectures. Nothing in PPSi prevents our fellow developers to support their own ptp extension using PPSi.
Each extension lives in a subdirectory called proto-ext-name. Source files in that directory may override the implementation for the standard protocol (which lives in proto-standard) to provide their own functionalities. To simplify writing extensions, the proto-standard functions already provide hooks so the extension itself can provide callbacks while still using the basic ptp engine. The set of callbacks is currently based on the needs of WR, but we are willing to accept patches to provide more hooks as needed.
If you plan to write your own protocol extension within PPSi, please refer to how proto-ext-whiterabbit is implemented.
PPSi supports custom implementations of time functions, so you can easily port the daemon to your own timing primitives.
The subdirectories named time-sth are used to implement timing functions; timing includes the methods that are part of two data structures: pp_time_operations and pp_network_operations. The former structure deals with getting and setting system time, while the latter deals with frame tx and rx. Network operations are concerned with timestamping the actual I/O, and that's why they are considered part of the “timing” of PPSi.
Each architecture defines its own TIME= choice in the specific Makefile. The user can override the default by either setting the variable on the command line of PPSi, or by setting an environment variable if the architecture supports it. For example, the bare architectures force TIME=bare, while the unix architecture suggests Unix timing code (TIME?=unix). The choice for TIME= is not currently selected using Kconfig, because nobody is expected to diverge from the default choice for the architecture being built.
If you want to support a different timing engine within the Unix build system, you can use “make TIME=xyz” to request building the time-xyz subdirectory. Please note that the Unix time structures are built anyways for CONFIG_ARCH=unix, so you can piggy-back on those functions, either within your own methods or by replacing the ppi->t_ops and ppi->n_ops for the communications links that do not include your own hardware support.
This is a summary of current and planned features for the PPSi package. As said, we are working on it these months, so the following list me be not up to date with software developments by the time you read it.
The following list if features doesn't consider known bugs, which are listed in the following sections. Please consider such bugs transient mishaps, as we are working on them right now; anyways, it would be unfair to hide them.
As of 2013-05 the project suffers from these known bugs:
PPSi support configuration files and individual configuration items passed on the command line. Such support is currently not available for freestanding architectures (the bare ones and wrpc-sw).
When PPSi starts it parses its own configuration. The command line can include a number of -f <file> and -C <item> options; they are processed in order, so later ones override earlier ones. A configuration “item” can include several directives, using the semicolon as a separator.
If no configuration file is specified, the program reads the default one, which is architecture-specific (thus, the default configuration file is processed after all the command line configuration items. The default file name is /etc/ppsi.conf, but arch-wrs tries to load /wr/etc/ppsi.conf first. The source tree of PPSi includes two example configuration files in its own etc/ subdirectory.
Configuration is made of of global options and port-specific options. To configure a port, use port <name> followed by its options. The parser allocates a new ptp state machine for each new port, but allows changing configuration of an existing port. For instance, to enable frame diagnostics for a specific port, you can use:
./ppsi -f /etc/ppsi.conf -C "port eth1; diagnostics 02"
Each configuration item is made up of a keyword and an optional argument. The argument can be either a number or a string. The parser looks up keywords in three tables: a global table, an architecture-specific table and an extension-specific table.
Currently no architectures or extensions provide configuration keywords, while the list of global items is on show in lib/conf.c, as the pp_global_arglines array. Future versions of this manual may document the keywords, if the time allows it.
Note: most current command-line options are going to be turned into configuration options. This applies to the priorities, intervals and thresholds, as well as the slave-only flag.
The clock class value (clockClass), a field of the “clock quality” structure, can be specified in the configuration file for the architectures that support such a file.
For the White Rabbit node (arch-wrpc) the class is defined according to build-time constants, set forth in constants.h.
When the node is configured as grandmaster (WRPC shell command mode gm), the class is set to 6 (a clock synchronized to a primary reference time source). However, if the internal PLL can not lock in a 60 seconds timeout, the class is set to 52: a clock that will never be a slave, but is not currently connected to a primary time source.
When the node is configured as master, it uses class 187, i.e. a clock that may be a slave of another clock.
During development of PPSi, diagnostic support used several techniques, but finally we converged on the one described here, that is here to stay. The idea is that we need to add verbosity per-feature and per-port. This fine-grained control is expected to be important while developing features or while diagnosing problems on new architectures.
The diagnostics configuration keyword can be used both as a global item and as a port-specific configuration value.
The header file <ppsi/diag-macros.h> introduces the concept of diagnostics flags, which are hosted in both a global variable and pp_instance (i.e. one flags-set for each communication link). The macros use the logical OR of both flags, so developers can activate diagnostics on either a single link or globally.
The diagnostic flags are split into topics (called things). For each diagnostic thing the header defines a few bits; so we can have diagnostic levels for each of them, but we suggest only using level 1 and 2 – the rationale is in the header itself.
The things currently defined are:
The user is expected to pass diagnostic flags as a string, specifying diagnostic levels for each of the things, where trailing zeroes are optional. So for example “01” specifies a diagnostic level 1 for time, and “102” specifies FSM at level 1 and frames at level 2. The header itself is more detailed about the conventions.
To parse the diagnostic string, PPSi offers pp_diag_parse. The function is used by both the code that reads the command line and code that parses configuration.
Within PPSi, developers should insert diagnostic messages by means of the pp_diag function:
pp_diag(ppi, thing, level, ...)
The function acts like printf, with the leading arguments ppi, thing (which is one of the names fsm, time, frames, servo, bmc, ext) and level, which should be 1 or 2. For example, the code setting system time includes this diagnostic line:
pp_diag(ppi, time, 1, "%s: %9li.%09li\n", __func__,
tp.tv_sec, tp.tv_nsec);
Finally, if you need to shrink the size of your binary file, you can build PPSi with no diagnostic code at all (i.e., the compiler won't even generate the function calls), you can define PPSI_NO_DIAG in CFLAGS while building. This can be achieved by setting USER_CFLAGS:
make USER_CFLAGS="-DPPSI_NO_DIAG"
For more details please refer to the header file, which is throughly commented.
We introduced the diagnostic macros described above at the beginning of March 2013. Earlier code used a less-structured approach, which has later been removed. If you used PP_PRINTF, it's gone by now.
The code uses pp_printf as a replacement for printf. This implementation comes from previous work by Alessandro, which in turn uses an older Linux implementation. Now pp_printf is published separately, as the “Poor Programmer's Printf” and included here in its own subdirectory. Please check pp_printf/README for more details about the size of this implementation and the different implementation choices.
By avoiding calls to the real printf function we allow the PPSi code base to be built for freestanding implementations without ugly #ifdef tricks in the code. Please note that pp_printf and printf can coexist: for example the hosted version of PPSi links with the standard library without any problem; in that case pp_printf relies on fputs to write to stdout.
If your run-time environment already includes an implementation of pp_printf, you can build with CONFIG_NO_PRINTF set. For example:
make USER_CFLAGS="-DCONFIG_NO_PRINTF"
The resulting ppsi.o will include undefined references to the pp_printf symbol, which must be provided externally. This is how we build for arch-wrpc, which already includes its own implementation if pp_printf. In that specific situation we still link the libc provided by the compiler, but we don't want to lift its own printf which would bring in most of newlib and would overflow our available RAM by a huge amount.
If your freestanding environment provides a printf that you want to use, and which is not called pp_printf, you may use the PROVIDE keyword in your linker script. You can find an example in the wrpc-sw package, which maps mprintf to pp_printf at link time, in order to accept external code that calls mprintf, which we don't provide any more.
The PPSi package includes some tools, mainly meant to help developers. Most of them live in the tools subdirectory and must be built separately with “make -C tools”.
This is a sniffer for ptp frames. It reports all Ethernet frames and UDP datagrams that talk ptp, from a specific network interface. The output format is line-oriented to help running grep over log files. The number of blank lines between frames depends on how much time elapsed between them; this should help identifying sync/follow-up pairs at a glimpse of the eye.
The program receives one optional argument on the command line, which is the name of the interface where it should listen; by default it uses eth0.
This is, for example, the dump of two UDP frames:
TIMEDELTA: 977 ms
TIME: (1362504223 - 0x51362a1f) 18:23:43.958091
ETH: 0800 (00:22:15:d7:c0:ef -> 01:00:5e:00:01:81)
IP: 17 (192.168.16.1 -> 224.0.1.129) len 72
UDP: (319 -> 319) len 52
VERSION: 2 (type 0, len 44, domain 0)
FLAGS: 0x0002 (correction 0x00000000)
PORT: 00-22-15-ff-fe-d7-c0-ef-00-01
REST: seq 29, ctrl 0, log-interval 0
MESSAGE: (E) SYNC
MSG-SYNC: 1362504223.957872054
DUMP: payload (size 44)
DUMP: 80 02 00 2c 00 00 02 00 00 00 00 00 00 00 00 00
DUMP: 00 00 00 00 00 22 15 ff fe d7 c0 ef 00 01 00 1d
DUMP: 00 00 00 00 51 36 2a 1f 39 17 f7 b6
TIMEDELTA: 0 ms
TIME: (1362504223 - 0x51362a1f) 18:23:43.958259
ETH: 0800 (00:22:15:d7:c0:ef -> 01:00:5e:00:01:81)
IP: 17 (192.168.16.1 -> 224.0.1.129) len 72
UDP: (320 -> 320) len 52
VERSION: 2 (type 8, len 44, domain 0)
FLAGS: 0x0002 (correction 0x00000000)
PORT: 00-22-15-ff-fe-d7-c0-ef-00-01
REST: seq 29, ctrl 2, log-interval 0
MESSAGE: (G) FOLLOW_UP
MSG-FOLLOW_UP: 1362504223.957953221
DUMP: payload (size 44)
DUMP: 88 02 00 2c 00 00 02 00 00 00 00 00 00 00 00 00
DUMP: 00 00 00 00 00 22 15 ff fe d7 c0 ef 00 01 00 1d
DUMP: 02 00 00 00 51 36 2a 1f 39 19 34 c5
The tool outputs a pulse-per-second signal to a parallel port or a serial port. Most likely it only works on an x86 Linux system because of direct access to I/O ports – no such portability problem is expected for serial ports. For this reason it is not built by default in the distributed tools/Makefile
The program receives a single argument on the command line, which is either an hex port number (with or without leading 0x) or an absolute filename:
./tools/pps-out 378
./tools/pps-out /dev/ttyS0
When passed a port number, the program toggles all bits assuming it refers to a parallel port. When passed a pathname, the program assumes it is a serial port and it toggles the DTR and RTS modem control signals (on pins 4 and 7 of the DB9 male connector).
By setting the VERBOSE environment variable you ask the program to report how late it was before and after generating the rising edge:
morgana$ VERBOSE=1 ./tools/pps-out /dev/ttyS0
gettimeofday takes 0.2 usec
udelay takes 4.0 usec
delayed between 0 and 2 usecs
delayed between 0 and 4 usecs
[...]
pps-out can be used to verify on a scope the level of synchronization of two or more computers, but please note that the user-space software-only approach shows some jitter; on my systems the delay is usually a few microseconds, up to around a dozen (in general, use of the parallel port has less delay and less jitter). In any case this offers a second source to check what NTP or ptp daemons report.
The program uses settimeofday to make the local time jump forward or backward by some amount, specified as floating-point seconds:
# date +%H:%M:%S; ./tools/jmptime 1.32; date +%H:%M:%S
12:24:28
Requesting time-jump: 1.320000 seconds
12:24:29
This program monitors changes in the current time, but comparing the output of clock_gettime(CLOCK_REALTIME) and clock_gettime(CLOCK_MONOTONIC). The delay between successive checks, expressed in milliseconds, can be specified on the command line and defaults to 10ms. The program only reports observed changes that are bigger than 0.5ms, to avoid excessive reporting of false positives that are simply due to process latencies induced by the system workload.
By running this program you can see the effect of tools/jmptime, or the insertion of a leap second (which is why I wrote this program in June 2012).
For example, this is what chktime reports when running “jmptime .002”. The program doesn't need superuser privileges:
% ./tools/chktime 50
./tools/chktime: looping every 50 millisecs
13-03-12-12:29:41: 1996 us
This program works like jmptime, but it requests a slow adjustment of the time. It receives the requested adjustment on the command line, as a floating point number just like jmptime described above. Additionally, it reports what was the ongoing adjustment using adjtime. See the adjtime(2) man page for details.
Adjustments requested by this program cannot be reported by tools/chktime, even when the overall change integrates to more than half a millisecond, because the change in clock speed affects both CLOCK_REALTIME and CLOCK_MONOTONIC.
You can see adjustment in one host by comparing with the time of another host, for example using tools/mtp, described next.
The following example shows how on my host the kernel adjusts the time by 15ms every 30s (i.e. 0.05%):
# ./tools/adjtime .15; sleep 30; ./tools/adjtime 0
Requesting adjustment: 0.150000 seconds
Previous adjustment: 0.000000 seconds
Requesting adjustment: 0.000000 seconds
Previous adjustment: 0.135000 seconds
The program reads and optionally changes the clock rate of the host system using the Linux-specific adjtimex system call (the same being used in the core PPSi program). It is meant to check adjustment and get acquainted with the involved values.
The numerical argument of the system call is parts-per-million scaled by 16 bits. So for example half ppm is passed as 32768.
The following are example uses of the program on an ntp-driven host:
morgana% ./tools/adjrate --help
./tools/adjrate: use "./tools/adjrate [<adj-value> [ppm]]"
morgana% ./tools/adjrate
rate: -407582 (-6.219208 ppm)
morgana% ./tools/adjrate -6
./tools/adjrate: adjtimex(rate=-6): -1 (Operation not permitted)
morgana% sudo ./tools/adjrate -6 ppm
morgana% sudo ./tools/adjrate
rate: -393216 (-6.000000 ppm)
The directory tools/mtp includes a few example programs I wrote for an article about time synchronization. MTP means “mini time protocol”; it uses the T1, T2, T3, T4 idea to report the time difference between two hosts. The program comes in two flavors: UDP and raw Ethernet.
To run a listening server on one host, you can run the program in listen mode:
tornado% ./tools/mtp/mtp_udp -l
On the other host, you can run the client that reports the time difference it measures, you can pass either an IP address or an host name:
morgana% ./tools/mtp/mtp_udp tornado
0: rtt 0.000459000 delta 0.099351500
You can continuously monitor the difference by running the program in a loop:
morgana% while true; do ./tools/mtp/mtp_udp tornado; sleep 0.1; done
The program mtp_packet works in the same way by using raw Ethernet frames (AF_PACKET). It needs an interface name as first argument and superuser privileges.
The MAKEALL script, in the top-level directory of PPSi builds the program for all known configurations, picking them from configs/. Developers are urged to run it before committing each patch, to ensure they are not breaking the program for configurations they are not using – but sometimes I forgot to do that myself and committed trivially-incomplete changes.
It may happen, however, that some developers experience errors or warnings that others didn't notice, because of differences in compiler version or library versions.
What follows is an older example run, limited to hosted compilation; currently, with the new Kconfig engine introduced after release 2013.11, MAKEALL ignores command-line arguments and always builds all configurations; but I plan to add the command line back.
% ./MAKEALL unix
###### Build for arch "unix", ext "", printf xint
text data bss dec hex filename
15801 224 344 16369 3ff1 ppsi.o
34370 984 380 35734 8b96 ppsi
###### Build for arch "unix", ext "", all messages
text data bss dec hex filename
16850 224 344 17418 440a ppsi.o
35410 984 380 36774 8fa6 ppsi
This is a summary about the build process for PPSi.
The main Makefile creates directory names from $ARCH and $PROTO_EXT, each sub-Makefile then can augment CFLAGS with "-ffreestanding" or whatever it needs. Similarly, CROSS_COMPILE may be set by sub-Makefiles but please let the environment override it (as no custom pathnames should be edited before building and the pristine package can be used).
The basic state machine is in ./fsm.c. It's a simple file released in the public domain as we'd like the idea to be reused and the code itself is not worth copylefting – even thought the file is not completely independent from PPSi itself.
All the rest of the package is built as libraries. The link order of libraries selects which object files are picked up and which are not. Additionally, "CONFIG_PRINTF_XINT" or one of the other pp_printf configurations can be set to override the default. By default PPSi builds the “full” implementation.
This state-machine source refers to a specific state machine by the name pp_state_table. The table is picked from a library: either from the extension being selected or the proto-standard one. The table includes pointers to functions, and such names will select which other object files are picked up from the libraries.
Individual architectures can add files to the "OBJ-y" make variable, in order to add their own stuff (like the main function or crt0.o for freestanding stuff). Similarly, architectures can add files to the "all" target. The main Makefile only builds ppsi.o, leaving the final link to the chosen architecture, so for example arch-unix adds ppsi to the all target.
Since code and data space is a problem in the freestanding world (for example, the whole ptp may need to fit in 64kB RAM including data and stack), each state in the state machine of the standard protocol must be implemented as a separate file. This allows an extension not using that particular function to save the overhead of binary size. Clearly an extension may implement several functions in the same file, if they are known to be all used in the final binary or of it uses -ffunction-sections and -fdata-sections).
The code is licensed according to the GNU LGPL, version 2.1 or later. Some files are individually released to the public domain, when we think they are especially simple or generic.
Both the full and the partial printf code is distributed according to the GPL-2, as it comes from the Linux kernel. This means that any code using our diagnostics fall under the GPL requirements; you may compile and use the diagnostic code internally with your own proprietary code but you can't distribute binaries with diagnostics without the complete source code and associated rights. You may avoid the GPL requirements by using different printf implementations; if so we'd love to have them contributed back in the package.
The same issue about the GPL license applies to the div64_32 function. We need this implementation in our wrpc code base because the default libgcc division is very big, and we are always tight with our in-FPGA memory space.
For binaries without diagnostic code and the size-optimized division, the LGPL applies, as detailed below.
For licensing purposes, any `arch-name' or `time-name' subdirectory "can be reasonably considered independent and separate works in themselves" (quoting the LPGL text).
Code in the directories provided here is all LGPL, and you may add your own drop-in subdirectories. The LGPL requirements for source distribution and associated permissions won't apply to any such subdirectory that you may add, even if technically such code is not linked as a library.
The coding style is the one inherited from Linux kernel project (see Documentation/CodingStyle in the kernel sources). However, structures, constants and field names defined by ieee 1588 are kept in the suggested "CamelCase" form. Similarly, the typedefs are left, even if they are really a pain to deal with. Most of this stuff is in include/ppsi/ieee1588_types.h file.
The mostly used prefix is pp_, the short prefix for 'Portable ptp', which is used for every function related to the algorithm itself (but not in the architecture-specific code).
Some prefixes to the ieee naming are added, in order to improve readability:
EN means "enumeration type". For instance, Enumeration Time Source (defined in the spec at table 7, page 57) becomes ENTimeSource.
PPM_ means "ppsi message", and is used for message types.
Msg means "message" and is used for message structures.
PPS_ means "ppsi state" and is used for state machine's states definition.
DS means "data set", and is used for the standard Data Sets (e.g. DSCurrent is the "Current Data Set"). The concept of data sets is defined in the specification at chapter 8, page 63.
The hosted build, as well as the two bare ones, include command line support.
The command:
./ppsi --help
will print help about command line options (we also support the question-mark like the original ptp, but it's a bad choice because it is a shell wildcard that should be escaped). Actually, all multi-char options will print the help at this point in time, because we only support since-char options; we don't want to rely on getopt which is not available for all architectures, nor we want to implement complex parsing.
Note: most current command-line options are going to be turned into configuration options. This applies to the priorities, intervals and thresholds, as well as the slave-only flag.
For standard operation, simply run ./ppsi with no options. It will work like the PTPd executable, with the automatic choice of master/slave defined in ieee specification (announce/timeout mechanism).
What follows is a list of the most important command line options. For a list of the other ones please see the help message.