5.
Interface Implementation
Specification
The interface implementation is specified using the 7-layer Open Systems
Interconnection model (OSI model):
Layer 7 - Application [Specified according to the SEI Interface
Documentation [AD7: Chapter 7]
Layer 6 - Presentation
Layer 5 - Session
Layer 4 - Transport
Layer 3 - Network
Layer 2 - Data link
Layer 1 - Physical
The four SDP-CSP interface implementations are subsequently described
separately in terms of the OSI model.
5.1.
SDP – CSP Visibility Data Interfaces:
I.S1M.SDP_CSP.001
This interface will carry real-time streaming visibility data
unidirectionally from the CSP to the SDP instances. The data exchange
between the SDP and the CSP for the visibility data interfaces is fully
described in this section.
A visibility is defined as the complex cross-correlation produced
between two individual receiving elements (real or beamformed) for a
single spectral channel. For the purpose of this interface the
definition of a visibility shall include autocorrelations.
A visibility data frame is produced by the CSP on a regular cadence as
determined by the correlator dump rate. A data frame is defined as the
full dimensionality of the high-dimensional data hypercube produced by
CSP. The dimensions of the visibility data frame are determined on a per
scan basis and remain fixed for the duration of a scan. A
scheduling block consists of multiple scans and is allocated to
only one subarray at any one time and subarrays are
independent (in terms of data flow, data processing, monitoring and
control) of each other.
In general, this visibility data frame is a multi-dimensional array of
visibilities with the following axes:
-
Frequency Channels
-
Phase bins
-
Baselines
-
Polarisation products
5.1.1. Application
Layer
The Application layer (OSI layer 7) is defined in the SPEAD protocol
[AD3]. (SDP_REQ_INT-285)
5.1.1.1. Interface
identity
The SPEAD protocol is described in [AD3]. The Python SPEAD2
[RD8] implementation of the SPEAD protocol is used in this interface
discussion. It is an improved implementation of the SPEAD protocol
compared to the previous PySPEAD [RD9] implementation.
5.1.1.1.1. SPEAD
flavour:
The SPEAD protocol is versioned and within a version allows for multiple
flavours, with different numbers of bits for item pointer fields. The
spead2 library supports all SPEAD-64-YY flavours of version 4, where YY
is a multiple of 8.
This interface uses the SPEAD-64-48 flavour (ItemPointerWidth=16bits,
HeapAddressWidth=48bits) in order to support 48bit immediate values.
The following relevant SPEAD definitions serve to describe the SPEAD
visibility payload (See [AD3] for full explanation of SPEAD protocol
and payload format.)
-
Item : a Variable transmitted using SPEAD protocol
-
ItemGroup a Collection of Items (SPEAD variables) to be
transmitted
-
Heap : An ItemGroup packaged for transmission as UDP data
packets
-
ItemPointer : metadata in packet header containing information
how to unpack received datagram
-
ItemDescriptor : ItemDescriptors are used to provide receiving
clients with the metadata required to decode, interpret and unpack
Heaps to form ItemGroups as part of a particular SPEAD stream. In
general, these packets will be sent at the start of a session before
data packet transmissions begin.
5.1.1.2.1.
Data organisation (SPEAD Heaps and
Streams)
The SPEAD protocol makes provision for a top-level logical group called
a heap (i.e. a portion of a visibility data frame). This can consist of
a number of individual packets (UDP/IP packets), each of which contain
the same unique SPEAD ID and an appropriate offset into the overall heap
for the specific packet in question.
As defined, the SPEAD heap will form the smallest individually routable
data unit, i.e. all the packets for a specific heap need to be received
by a single receiving node.
The visibility data frame is divided into heaps according to the data
parallelism required by SDP and the architecture of the CSP correlator.
Each heap will contain the following set of visibility data for a single
correlator dump: all polarisation products of all baselines (for a
subarray) of one frequency channel and one phase bin.
Subsequent SPEAD heaps (correlator dumps) of a particular scan of the
same channel, phase bin, baseline and polarisation combination will be
grouped as a stream of visibility data (SPEAD stream) and routed to a
specific IP and port combination. Multiple streams are created by
directing the transmission of SPEAD heaps to different ports or hosts.
(SDP_REQ_INT-38)
The order of channels, baselines, phase bins and polarisation products
(for a specific scheduling block) will be supplied to SDP by TM
[RD4] (as informed by CSP) prior to the start of a scheduling block
(refer to [RD5] for definition of scheduling block).
5.1.1.2.2. Data
routing
From CSP, all packets belonging to the same SPEAD stream (and therefore
heap) will be routed to a single SDP node [RD6] since SDP’s first
order parallelisation is by frequency. Therefore it’s possible that the
SPEAD packets forming a heap may come from different CSP nodes. One CSP
node will generate the SPEAD heap header packet (one packet) and may
also generate SPEAD packets containing its subset of visibility data and
other CSP nodes may also generate SPEAD packets for the same heap with
their subset of visibility data. All the SPEAD packets (from the same
heap) contain the heap ID and are routed to the same SDP node. The
packets forming a heap are not required to arrive at the SDP destination
node in a particular order. The complexity of the SDP processing (for a
particular observation) will determine how many frequency channels are
processed on a single SDP node and the routing of the frequency channels
(streams) will be done accordingly.
SDP will supply the routing information to CSP (via TM) [RD4] for a
particular scheduling block and this routing will remain static for the
duration of the scheduling block. The routing information will be
supplied to CSP prior to the start of the scheduling block. Since
routing is done per frequency channel and phase bin the routing
information will contain the following for each frequency channel and
phase bin (TBC-001):
-
Channel ID & Phase bin ID
-
SDP destination node IP address & MAC address
-
SDP destination node port
5.1.1.2.3 Sending
data
The CSP correlator nodes generate SPEAD packets directly for
transmission across the I.S1M.SADT_CSP.005 interface. See [AD3 par
8] for the SPEAD Packet Data Representation (i.e SPEAD header,
ItemPointers and ItemDescriptors). See Table 3 in section 5.1.1.4 for
the ItemDescriptor items (field) for each SKA1 SPEAD item.
The data flow and packet exchange is fully described in [AD3]. Here
is a brief overview of the typical steps that are performed to transmit
data for a particular SPEAD stream from CSP to SDP:
-
At the start of the session (scan) an ItemDescriptor is generated for
each Item (variable) in the ItemGroup and all ItemDescriptors are
transmitted as a heap (with a special ItemPointer in the header of the
heap). These ItemDescriptor packets are issued by the CSP nodes (i.e.
FPGAs).
-
A heap is subdivided into a collection of data packets and sent as a
stream of UDP datagrams (by the CSP Nodes). Heaps are sent at the
correlator dump rate for the duration of the session (scan).
-
When the SPEAD stream terminates at the end of the session (scan), a
stream control Item is sent to indicate that the transmission has
completed. This may be in a separate SPEAD packet and this packet
could be issued by a control computer rather than an FPGA.
The shape (ItemDescriptorShape) of each SKA1 SPEAD item is
defined according to the subset of the visibility data frame allocated
to a heap. Therefore each SKA1 SPEAD item (if required) is only included
once in a heap.
5.1.1.2.4. Receiving
data
SDP receives visibility data as SPEAD packets from which it reconstructs
SPEAD heaps with appropriate metadata. These are then packaged in the
internal SDP format and made available for further processing.
5.1.1.3. Data
pre-conditions
5.1.1.3.1. Visibility
Coverage
For a given set of frequency channels the CSP shall produce all
polarization products, for all logically independent cross-correlations
between and auto-correlations of receiving elements (real or beamformed)
for a given scheduling block (subarray). (SDP_REQ_INT-33)
5.1.1.3.2. Visibility
Corrections
3: Visibility Corrections