Orthogonal time frequency space (OTFS) modulation holds great promise for enabling integrated sensing and communication (ISAC) systems in future mobile networks. However, computing the channel matrix in OTFS presents significant challenges due to its extremely high dimension, especially in sensing applications. This study introduces a novel approach to drastically reduce the computational complexity of the OTFS-based ISAC system. Firstly, we define four low-dimensional matrices, which are utilized in computing the channel matrix through simple algebraic manipulations. Secondly, we establish an analytic criterion independent of system parameters to identify the most informative elements in each of these matrices by leveraging the properties of the Dirichlet kernel. To gauge the impact of our approach, we derive the computational complexity of the distilled channel matrix in terms of the number of elementary operations required. Numerical results demonstrate that our proposed technique significantly reduces receiver complexity by up to three orders of magnitude without compromising sensing performance.