This paper introduces a systematic design method for decoupling
elements, which can significantly improve the isolation between two
co-located antennas, e.g. between transmit and receive antennas of an
in-band full-duplex system. The design method applies the theory of
characteristic modes for controlling the phase and amplitude of the
scattered fields of the decoupling element, in order to optimally cancel
the original incident fields which couple to the receiving antenna. We
describe concisely the effects that characteristic angle, modal
near-field, and modal excitation of the decoupling element have on the
antenna isolation. For validating the proposed method, a planar wavetrap
is designed and the isolation improvement verified with full-wave
simulations. When we use the proposed method to optimize a wavetrap that
is placed between two co-located patch antennas, we obtain an
improvement of the isolation between the antennas by 33 dB at the centre
frequency of their operational frequency band, and at least 12-dB
improvement across the whole 142-MHz operational bandwidth of the two
antennas. As a benchmark, the wavetrap is replaced by an absorber
occupying 10 times the volume of the wavetrap. The absorber gives only 6
dB of isolation improvement, substantiating the effectiveness of the
proposed wavetrap method.