6: Transient Search: Logical Model


5.2.1. Application layer

5.2.1.1. Interface Identity
This interface uses FTP (RFC 959) to transfer Pulsar Search and Fast Transient/Single Pulse data from CSP to SDP.
5.2.1.2. Resources

5.2.1.2.1. Data organisation

OCLD/SPOCLDs are generated per beam per pulsar search scan (180 - 1800 seconds) and delivered from CSP to SDP. A data stream is defined as all OCLD/SPOCLD data files of a specific beam for the duration of a scheduling block. (SDP_REQ_INT-317) (refer to [RD5] for definition of scheduling block)
If possible OCLDs will be divided into smaller packets/files (on a time or candidate basis) and transferred to SDP as soon as they are ready (TBC-012 ), otherwise OCLDs will be transferred as complete entities (files or objects).

5.2.1.2.2. Data routing

Data streams (as defined above) are routed to SDP nodes. The complexity of the SDP processing will determine how many data streams are processed on a single SDP node and the routing of data streams will be done accordingly.
SDP will supply the data routing information to CSP (via TM) [RD4] for a particular scheduling block and this routing will remain static for the duration of the scheduling block (SDP_REQ_INT-316). The routing information will be supplied to CSP prior to the start of the scheduling block. Since routing is done per beam the routing information will contain the following for each beam:

5.2.1.2.3. Sending and receiving data

SDP will be the server (TBC-011) (at least one per receiving node). When a scan is configured each CSP node opens a control connection to a server. When a file or block of data is ready to be transmitted, CSP sends a STOR command with at least the Scan ID, Beam ID and Scheduling Block ID encoded in the ”file name” parameter. SDP will open data channel (if not already open) in the usual way and CSP transmits the data in BLOCK mode.
The following parameter setting should be used:
Block mode is recommended as this allows the data connection to be reused and provides reliable indication that a transfer is complete.
If an OCLD is divided into smaller packets/files for transmission, then SDP will reconstruct the OCLD once all the data for scan has been received.
5.2.1.3. Data pre-conditions
5.2.1.4. Data types and constants

5.2.1.4.1. Candidate List

The candidate list is a form of metadata, providing a summary of all significant candidate detections (all candidates found before sifting) made per beam, where significance is determined by the signal processing pipeline within CSP. Candidate lists are important, as they provide information useful for improving candidate filtering. The precise number of entries in the list will vary for each scan. However the length of the list will be capped at TBD-003 entries.
The candidate list will be formatted as a table. The columns include, but are not limited to: Timestamp, Beam ID, Scheduling Block ID, S/N, Acceleration, DM, Period, Pulse Width, duplicates, number of harmonics, and TBD-004. (SDP_REQ_INT-61) Candidates are ranked and sorted by TBD-005 manner (most likely according to S/N). (SDP_REQ_INT-297).
File Format: CSV file using ASCII encoding (RFC 4180)

5.2.1.4.2. Optimised Candidate Data

Optimised candidate data provides a summary description of each sifted candidate for scientific analysis consisting of a data cube and associated metadata. The sifted candidates are candidates surviving the sifting process undertaken within CSP, a process which aims to remove duplicate detections of the same signal.
5.2.1.4.2.1. Optimised candidate data cube
For each sifted candidate, a data cube describing the detection in time, phase, and frequency space is stored. The data cube represents the best detection of a candidate.
File format: PSRFITS (SDP_REQ_INT-301 & SDP_REQ_INT-291)
The PSRFITS version number is not specified in this ICD because it shall be defined in the header of the file.
5.2.1.4.2.2. S/N Sheets
For each sifted candidate multiple signal-to-noise (S/N) sheets are stored.
File format for the S/N sheets: HDF5
5.2.1.4.2.3. Metadata (per candidate)
Each cube is accompanied by metadata, which further helps describe the detection. Such metadata includes, but is not necessarily limited to, information associated with the scan (Beam ID, Scheduling Block ID, quality measures, etc.).
This metadata is contained in the PSRFITS and/or HDF5 files.
5.2.1.5. Error handling
Failure of the communication link or the receipt of incorrectly formatted data results in an alarm being generated.
5.2.1.6. Variability
This section shows the variability than can be expected and any constraints on specific use cases of the interface or particular aspects of the interface.
Note that parameter values for modes shown in this section are expected or typical values and are provided for information only. This interface does not constrain parameter values unless explicitly stated as a constraint. Actual parameter values will be chosen by Telescope Manager (for each scheduling block and its scans) according to the requirements of the observation and any constraints imposed by components of the telescope [AD6].

5.2.1.6.1.Pulsar search data

The pulsar search data cube represents the best detection of a candidate, ‘folded’ and dedispersed at the optimised dispersion measure, period, and period derivative values - all determined within CSP. The data cube (per candidate) has the following axes (SDP_REQ_INT-300):
The pulsar search candidate data also includes S/N sheets (three 2-dimensional matrices) which describe the resulting folded S/N ratio for different combinations of folding parameters:
Table 12 describes the list of pulsar search parameters and expected values for each parameter.
12: Pulsar search parameters