5. Interface Implementation Specification

The interface implementation is specified using the 7-layer Open Systems Interconnection model (OSI model):
Layer 7 - Application [Specified according to the SEI Interface Documentation [AD7: Chapter 7]
Layer 6 - Presentation
Layer 5 - Session
Layer 4 - Transport
Layer 3 - Network
Layer 2 - Data link
Layer 1 - Physical
The four SDP-CSP interface implementations are subsequently described separately in terms of the OSI model.

5.1. SDP – CSP Visibility Data Interfaces: I.S1M.SDP_CSP.001

This interface will carry real-time streaming visibility data unidirectionally from the CSP to the SDP instances. The data exchange between the SDP and the CSP for the visibility data interfaces is fully described in this section.
A visibility is defined as the complex cross-correlation produced between two individual receiving elements (real or beamformed) for a single spectral channel. For the purpose of this interface the definition of a visibility shall include autocorrelations.
A visibility data frame is produced by the CSP on a regular cadence as determined by the correlator dump rate. A data frame is defined as the full dimensionality of the high-dimensional data hypercube produced by CSP. The dimensions of the visibility data frame are determined on a per scan basis and remain fixed for the duration of a scan. A scheduling block consists of multiple scans and is allocated to only one subarray at any one time and subarrays are independent (in terms of data flow, data processing, monitoring and control) of each other.
In general, this visibility data frame is a multi-dimensional array of visibilities with the following axes:

5.1.1. Application Layer

The Application layer (OSI layer 7) is defined in the SPEAD protocol [AD3]. (SDP_REQ_INT-285)
5.1.1.1. Interface identity
The SPEAD protocol is described in [AD3]. The Python SPEAD2 [RD8] implementation of the SPEAD protocol is used in this interface discussion. It is an improved implementation of the SPEAD protocol compared to the previous PySPEAD [RD9] implementation.

5.1.1.1.1. SPEAD flavour:

The SPEAD protocol is versioned and within a version allows for multiple flavours, with different numbers of bits for item pointer fields. The spead2 library supports all SPEAD-64-YY flavours of version 4, where YY is a multiple of 8.
This interface uses the SPEAD-64-48 flavour (ItemPointerWidth=16bits, HeapAddressWidth=48bits) in order to support 48bit immediate values.
5.1.1.2. Resources
The following relevant SPEAD definitions serve to describe the SPEAD visibility payload (See [AD3] for full explanation of SPEAD protocol and payload format.)

5.1.1.2.1. Data organisation (SPEAD Heaps and Streams)

The SPEAD protocol makes provision for a top-level logical group called a heap (i.e. a portion of a visibility data frame). This can consist of a number of individual packets (UDP/IP packets), each of which contain the same unique SPEAD ID and an appropriate offset into the overall heap for the specific packet in question.
As defined, the SPEAD heap will form the smallest individually routable data unit, i.e. all the packets for a specific heap need to be received by a single receiving node.
The visibility data frame is divided into heaps according to the data parallelism required by SDP and the architecture of the CSP correlator. Each heap will contain the following set of visibility data for a single correlator dump: all polarisation products of all baselines (for a subarray) of one frequency channel and one phase bin.
Subsequent SPEAD heaps (correlator dumps) of a particular scan of the same channel, phase bin, baseline and polarisation combination will be grouped as a stream of visibility data (SPEAD stream) and routed to a specific IP and port combination. Multiple streams are created by directing the transmission of SPEAD heaps to different ports or hosts. (SDP_REQ_INT-38)
The order of channels, baselines, phase bins and polarisation products (for a specific scheduling block) will be supplied to SDP by TM [RD4] (as informed by CSP) prior to the start of a scheduling block (refer to [RD5] for definition of scheduling block).

5.1.1.2.2. Data routing

From CSP, all packets belonging to the same SPEAD stream (and therefore heap) will be routed to a single SDP node [RD6] since SDP’s first order parallelisation is by frequency. Therefore it’s possible that the SPEAD packets forming a heap may come from different CSP nodes. One CSP node will generate the SPEAD heap header packet (one packet) and may also generate SPEAD packets containing its subset of visibility data and other CSP nodes may also generate SPEAD packets for the same heap with their subset of visibility data. All the SPEAD packets (from the same heap) contain the heap ID and are routed to the same SDP node. The packets forming a heap are not required to arrive at the SDP destination node in a particular order. The complexity of the SDP processing (for a particular observation) will determine how many frequency channels are processed on a single SDP node and the routing of the frequency channels (streams) will be done accordingly.
SDP will supply the routing information to CSP (via TM) [RD4] for a particular scheduling block and this routing will remain static for the duration of the scheduling block. The routing information will be supplied to CSP prior to the start of the scheduling block. Since routing is done per frequency channel and phase bin the routing information will contain the following for each frequency channel and phase bin (TBC-001):

5.1.1.2.3 Sending data

The CSP correlator nodes generate SPEAD packets directly for transmission across the I.S1M.SADT_CSP.005 interface. See [AD3 par 8] for the SPEAD Packet Data Representation (i.e SPEAD header, ItemPointers and ItemDescriptors). See Table 3 in section 5.1.1.4 for the ItemDescriptor items (field) for each SKA1 SPEAD item.
The data flow and packet exchange is fully described in [AD3]. Here is a brief overview of the typical steps that are performed to transmit data for a particular SPEAD stream from CSP to SDP:
  1. At the start of the session (scan) an ItemDescriptor is generated for each Item (variable) in the ItemGroup and all ItemDescriptors are transmitted as a heap (with a special ItemPointer in the header of the heap). These ItemDescriptor packets are issued by the CSP nodes (i.e. FPGAs).
  2. A heap is subdivided into a collection of data packets and sent as a stream of UDP datagrams (by the CSP Nodes). Heaps are sent at the correlator dump rate for the duration of the session (scan).
  3. When the SPEAD stream terminates at the end of the session (scan), a stream control Item is sent to indicate that the transmission has completed. This may be in a separate SPEAD packet and this packet could be issued by a control computer rather than an FPGA.
The shape (ItemDescriptorShape) of each SKA1 SPEAD item is defined according to the subset of the visibility data frame allocated to a heap. Therefore each SKA1 SPEAD item (if required) is only included once in a heap.

5.1.1.2.4. Receiving data

SDP receives visibility data as SPEAD packets from which it reconstructs SPEAD heaps with appropriate metadata. These are then packaged in the internal SDP format and made available for further processing.
5.1.1.3. Data pre-conditions

5.1.1.3.1. Visibility Coverage

For a given set of frequency channels the CSP shall produce all polarization products, for all logically independent cross-correlations between and auto-correlations of receiving elements (real or beamformed) for a given scheduling block (subarray). (SDP_REQ_INT-33)

5.1.1.3.2. Visibility Corrections

3: Visibility Corrections