[20]r0.6 \label{comm}Demonstration of commensal fast and slow transient science with the VLA during our latest FRB survey (started October 2015). In addition to searching the fast data stream, the current realfast system produces a slow copy to search for transients on timescales of seconds to days. Circles show (low-significance) fast transient candidates overlaid on an image of the galaxy NGC 2146.

Our science goals require the arcsecond resolution, large field of view, and high sensitivity of interferometric imaging. However, millisecond imaging produces data at rates higher than 1 TB hour\({}^{-1}\), which strains the traditional collect-and-analyze observing model. Our solution is bring computing power to the telescope for real time data triage. Since transients are rare and occupy a tiny fraction of the overall data volume, real-time data reduction compresses that stream to a manageable rate.

Over the past five years, our team has developed the algorithms and code base to support fast interferometry \citep{2011ApJ...742...12L,2012ApJ...749..143L,2012ApJ...760..124L}. We now have a transient search pipeline that combines interferometric concepts (calibration, RFI removal, imaging) with time-domain concepts (dedispersion, filters). This pipeline has been developed in Python/Cython and has been run in multi-core, multi-node, CPU-based architectures at the NRAO and national supercomputer facilities at LANL and NERSC \citep{2015ApJ...807...16L}. We also recently began implementing the transient search pipeline on the Elastic Compute Cloud service of Amazon.

Our approach exploits several simplifications for interferometric calibration and imaging that are possible on timescales less than \(\sim 1\) s: (i) the VLA fringe rate is always slow enough that constant sources can be removed by differencing of visibility data, eliminating the need for source catalogs. (ii) The noise level in the images is tens to hundreds of mJy, allowing for simple gridding schemes and single-precision floating point operations. (iii) Image cleaning is not necessary, since images are either empty or contain a single point source; a simple S/N threshold is adequate to find a transient.

[38]l0.45    

\label{coobserving}Co-obsering the Crab pulsar and nebula with realfast, RAPTOR (optical), and VLITE (300 MHz; single-antenna). The spatial localization of realfast lets us associate pulses with the RAPTOR image, while the time resolution lets us associate pulses with the VLITE lightcurve. Both images show realfast candidate transients overlaid as red circles with size proportional to significance of detection.

By integrating the transient search system with the correlator and observing system, we open access to the commensal data stream of thousands of hours per year. Figure \ref{comm} shows how the primary data product (an image of a galaxy) can be derived from the same observation that finds candidate fast transients. Broadly, real-time discovery and decision-making capabilities will transform the VLA from its traditional role as a transient follower into that of a leader of observing campaigns. We refer to this real-time, fast transient search system as realfast.

Our first large-scale demonstration of the transient search pipeline was in a off-line search of a 200-hour (200 TB) VLA survey for FRBs \citep{2015ApJ...807...16L}. We used… Search took xx time… False positives controlled… Our search did not find an FRB, but has forced a reevaluation of the FRB rate of occurrence; our limit is consistent with newer measurements \citep{2015MNRAS.447.2852K} that predict roughly 1 event per 200 hours of VLA observing in our mode.

In October 2015, we began the first realfast search for FRBs with a new 185-hour campaign at the VLA ¹¹See program code VLA/15B-305 led by S. Burke-Spolaor.. This effort is key not only for its potential to discover FRBs, but for implementing the first real-time, triggered data recording at the VLA. This demonstrates a key concept of realfast: data triage \citep[e.g.,][]{2004PhRvL..93d1101G,2015APh....61...22C}. Our current implementation²²Code available at http://github.com/caseyjlaw/realfast. uses a set of daemonized processes to find incoming data, start the transient search on 24 worker nodes at the VLA site, and aggregate results for visualization within a day of observing. The proposed realfast system will improve performance by using an in-memory buffer, much more computational power, and a redesigned software pipeline.

The current realfast system is integrated with the ”correlator back-end” (CBE) compute cluster, which gives it rapid access to observing metadata like pointing direction. We have used this information to drive co-observing with other observatories, including the VLITE, the Long Wavelength Array, RAPTOR, and Skynet. Figure \ref{coobserving} shows how we used this connection to co-observe the Crab pulsar and nebula at radio and optical wavelengths.

As a commensal system, the science potential of realfast depends on the distribution of VLA observations in antenna configuration and band (see §\ref{modes}). Here, we make a gross estimate of the fraction of time the VLA will be useful for commensal fast transient searches and consider the specific example of commensal observing with the VLA sky survey (VLASS). Generally, we target use cases in the lower VLA bands (below 10 GHz), which are sensitive to fast, coherent emission processes and have a relatively large field-of-view.

By inspecting gross measures of VLA usage³³See http://www.aoc.nrao.edu/~schedsoc/tac2015b.shtml. and taking a detailed look at one month of observing logs, we find that in compact antenna configurations (D and C), roughly 60% of observations are conducted below 10 GHz with \(\sim 25\)% in our favored L band (1–2 GHz). Considering the observing efficiency of 70%, this adds up to 1200 hours below 10 GHz and 500 hours in L band per compact configuration. Over a full year, the VLA commensal system will search roughly 4200 hours below 10 GHz and 1800 hours at L-band.

We also intend to target commissioning of realfast to take advantage of the VLASS, a \(\sim\)6000 hour, all-sky survey in B configurations at S band (2–4 GHz). In S band, dedispersion is a factor of \(\sim 5\) times less than at L band, so the proposed system will easily manage the computational load (see §\ref{modes}). VLASS is scheduled to begin in May 2016, but could be delayed until mid 2017 by the ongoing NRAO review process.

[18]r0.45 \label{periodicity}Image of maximum Fourier power in a VLA observation of pulsar B0355+54. The inset shows the Fourier spectrum at the pulsar location.

The real-time processing concept can be extended to imaging searches for periodic (pulsar) emission. In the more compact configurations, the fringe rate allows averaging on timescales of 1 and 3 minutes in C and D configurations, respectively⁴⁴See http://science.nrao.edu/facilities/vla/docs/manuals/oss/performance/fov/t-av-loss.. This means that dedispersed visibilities can be Fourier transformed in time and each period bin can be imaged and searched for sources as done for transient searches described above. Searching in this domain requires a factor of 100–1000 fewer Fourier transforms than an image-domain approach. In compact array configurations, this algorithm works on timescales comparable to the time per pointing used in many current single-dish pulsar searches \citep[\(\sim\)minutes;][]2006ApJ…637..446C,2014ApJ…791…67S. Figure \ref{periodicity} shows a demonstration of a periodicity algorithm in a VLA observation of pulsar B0355+54. We will implement an algorithm like this on whatever timescale the system can support in parallel with the transient search. One use case is a periodicity search of the Galactic plane, which has the highest projected density of pulsars. This region is not suitable for our most computationally demanding, high-DM transient searches, so the system can more easily sustain both transient and periodicity searches. This work will be useful for targeted pulsar searches with the VLA and as a prototype development for future array-based pulsar instruments (SKA, etc).