Inter-observer agreement in assessment of clinically overt extra nodal extension (cN3b)

Key Points

Data Availability Statement

The data that support the findings of this study are available from the corresponding author upon reasonable request.

Introduction

Regional lymph node metastasis is known to be one of the most powerful prognostic indicators in head and neck cancer. Furthermore, the presence of extra nodal extension (ENE) in these metastatic nodes is of profound prognostic significance^{. 1,2,3,4,5.}

In 2018 the American Joint Committee on Cancer (AJCC) released the eighth edition of the AJCC Cancer Staging Manual with several major changes to Head & Neck staging⁶. These changes included the introduction of a new N3b stage for ‘clinically overt’ evidence of ENE in nodal characterization of high-risk Human Papilloma Virus (HPV) -ve and non nasopharyngeal carcinomas^6,7. The AJCC has defined clinical N3b staging as clinically overt signs of gross ENE, such as skin involvement, muscle invasion or dysfunction to surrounding nerves, supported by radiological evidence^6,7. This differs significantly from pathological N3b on the other hand is defined as extension of metastatic carcinoma through the fibrous capsule of a lymph node into the surrounding connective tissue, regardless of the presence of stromal reaction and is further subcategorized as macroscopic and microscopic ENE^6,7.

In the literature, the accuracy of both pathological and radiological assessment of ENE has been previously assessed. It is well documented that early or microscopic ENE can be reliably identified on pathologic examination and given a quantitative value in millimetres⁸. The ability to diagnose ENE in radiology has also been extensively studied in both MRI and CT imaging with varied results^8,9,10.

However, there is no study to date assessing accuracy of clinical examination in reliably detecting ENE and moreover no studies assessing whether clinicians agree on cases ‘clinically overt’ ENE. The terms overt, unequivocal and unambiguous rightly encourage a very high bar for inclusion in this staging category. The stringency of the wording is designed to prevent stage migration and the requirement of supportive radiology further reinforces this. One might suspect that this clarification would provide for unanimous agreement in making this clinical diagnosis, but several questions must be answered to empirically support this assumption. Can surgeons accurately and consistently identify overt ENE on their clinical examination alone? Is there variability between surgeon’s assessments of the same patient and how does this affect the clinical staging in context of the new cN3b AJCC 8th edition staging system? This will ultimately improve our understanding of the AJCC HNSCC staging and give insight into whether cN3b staging is made in a consistent fashion. This remains of critical importance because without clinical suspicion, radiological evidence alone, however clear, does not in itself permit the cN3b diagnosis to be made.⁷ We aimed to investigate the accuracy and inter-observer agreement of Head and Neck surgeons in their clinical examination & assessment of clinically overt ENE and compare surgeons assessment to radiological findings consistent with clinically overt ENE.

Materials and Methods

This was a prospective cohort study conducted with the participation of 12 Head and Neck Surgeons at a weekly Head & Neck Cancer Multidisciplinary Meeting at a high-volume tertiary referral hospital. Institutional approval from a Human Research Ethics Committee was obtained prior to commencing. We included all patients with new diagnosis of Head and Neck SCC (either cutaneous or mucosal) with palpable cervical lymph nodes of any size who had accompanying cross sectional radiology (CT or MRI) (N=64). Palpability of lymph nodes was determined by the senior author (SV). Participating clinicians were consultant Head and Neck Surgical Consultants across the ENT, Plastic Surgery and Oral and Maxillofacial Surgery departments. Prior to the study all surgeons attended a briefing explaining the study, with further clarification of the cN3b stage and the need for overt and unambiguous clinical findings for diagnosis. Data were collected over a five-month period. Head & Neck Surgeons were blinded to all radiological findings prior to clinical assessment and asked to a) assess the presence of palpable cervical lymph nodes and b) identify specific physical examination findings of overt ECE in partivcular unambiguous skin involvement, muscle invasion and/or dysfunction to surrounding nerves. Surgeons completed a study proforma in a dichotomous (‘yes’ or ‘no’) fashion, along with an illustration of the location and number of lymph nodes for that patient (Fig 1). Radiological assessment of each patient was performed by a Head and Neck Radiologist blinded to the clinical findings. Again, the participating specialist documented their findings in a dichotomous fashion (‘yes’ or ‘no’) using the proforma (Fig 1). Statistical Analysis was performed calculating Inter-rater agreement using the overall percentage agreement and Fleiss’ Kappa statistic, which allows for more than two raters to be evaluated. Coefficients were calculated for the presence of palpable lymph nodes, skin involvement, muscle invasion and dysfunction to surrounding nerves. We used STATA version 15.1 (Stata Corporation, College Station, Texas, USA) for the analysis. Guidelines for Reporting Reliability and Agreement Studies (GRRAS) was utilised.

Results

A total of 82 palpable lymph nodes in 64 patients were assessed for clinically overt ENE. On average, three clinicians clinically assessed each patient with a total of 246 lymph node assessments.

A total of 246 lymph node assessments over a five-month period revealed substantial agreement (93%) with regards to the ability of surgeons to detect palpable lymph nodes in the patient sample. Statistical analysis with identification of the clinical signs of overt ENE revealed that there was perfect overall percentage agreement and inter-observer agreement for invasion of skin (100% and Fleiss’ Kappa=1, respectively) and almost perfect overall agreement on interobserver agreement on dysfunction of surrounding nerves (99% and Fleiss’ Kappa=0.91, respectively). Lower overall percentage agreement and inter-observer agreement were demonstrated for assessment of overt muscular infiltration with 81% and Fleiss’ Kappa=0.53, respectively.

Comparison of clinical and radiological assessment was also performed. Where all surgeons agreed there was clinically overt ENE (n=13), there was concordant radiological evidence of features of overt ENE in all cases. Conversely in patients where the surgeons all agreed that clinically overt ENE was not present (n=33) all cases demonstrated no radiological evidence of ENE. All cases where some surgeon disagreement existed as to the presence of overt ENE were based on the presence or absence of obvious muscle invasion or tethering to surrounding structures. (n=18). Each of these patients were assessed by three surgeons giving a majority of agreement or disagreement in each case. Of the ten patients in whom most surgeons agreed on the presence of muscular invasion a total six (6/10) demonstrated clear radiological evidence of muscular or deep tissue invasion. Of the eight patients in whom most surgeons felt that obvious muscular invasion did not exist, only one (1/8) showed radiological evidence of muscular invasion.

Discussion

This novel prospective comparative cohort study is the first of its kind looking at inter-observer agreement between Head and Neck surgeons in the assessment and identification of the clinical components of stage cN3b disease. Consistency in the staging of patients is important for several reasons. Staging allows for accurate prognostication which is valuable both to the patient and the treating clinician. When undertaking research, comparisons between patients are only valid when staging has been performed accurately and uniformly. Additionally, staging can at times aid in decision making with regards to best clinical management of the patients. The new N3b staging in head and neck cancer is a highly valuable addition for patient prognostication. The stringency for making the diagnosis is clear from both the unambiguous nature of the clinical findings required and confirmation with radiology. Our findings demonstrate a high level of consistency in making this diagnosis. This was particularly the case for cutaneous or nerve involvement by a lymph node. Evidence of overt muscular invasion showed high level of agreement but was not unanimous. When all surgeons agreed in their clinical assessment, the radiological findings concurred in every case. This was the case in 46 out of 64 patients. Based on our findings several patients might have been staged differently by different surgeons. These are all, by definition, patients with radiological evidence of ENE (without this evidence the cN3b diagnosis cannot be made). In several cases where there was disagreement, surgeons reported during tumour board discussion that they had a suspicion of muscular invasion but that it was not sufficiently unambiguous to meet the criteria. In the same circumstance’s others felt that there was sufficient tethering or fixation to label it unambiguous. Experience of the surgeon was not assessed.

As a multidisciplinary team it is often possible to get multiple surgeons to examine one patient. If during our study the ‘majority decision’ was to be used in making the cN3b diagnosis only one patient was at risk of mis-staging (1/64).

Conclusion

A total of 246 lymph node assessments were performed in 64 patients, between 12 Head and Neck surgeons. Our study has revealed high surgeon agreement in the ability to detect lymph nodes, the presence of skin involvement and in surrounding nerve involvement. Some variability of surgeon assessment was seen in the assessment of overt muscular involvement. Multiple surgeon assessment of each patient with acceptance of a majority decision on clinical findings increased concordance with radiological findings.

References

1. Myers JN, Greenberg JS, Mo V, et al. Extracapsular spread. A significant predictor of treatment failure in patients with squamous cell carcinoma of the tongue. Cancer. 2001;92:3030-3036.

2. Wreesmann VB, Katabi N, Palmer FL, et al. Influence of extracapsular nodal spread extent on prognosis of oral squamous cell carcinoma. Head Neck. 2016;38(Suppl 1):E1192-E1199.

3. Bernier J, et al. Defining risk levels in locally advanced head and neck cancers: a comparative analysis of concurrent postoperative radiation plus chemotherapy trials of the EORTC (#22931) and RTOG (# 9501) Head Neck. 2005;27:843–850. doi: 10.1002/hed.20279.

4. Cooper JS, et al. Long-term follow-up of the RTOG9501/intergroup phase III trial: postoperative concurrent radiation therapy and chemotherapy in high-risk squamous cell carcinoma of the head and neck. Int. J. Radiat. Oncol. Biol. Phys. 2012;84:1198–1205. doi: 10.1016/j.ijrobp.2012.05.008.

5. Johnson JT, Barnes EL, Myers EN, Schramm VL Jr, Borochovitz D, Sigler BA. The extracapsular spread of tumors in cervical node metastasis. Arch Otolaryngol. 1981;107(12):725-729.

6. Amin MB, Edge SB, Greene FL, et al (eds). AJCC Cancer Staging Manual, 8th ed. New York: Springer; 2017

7. Lydiatt WM, Patel SG, O’Sullivan B, Brandwein MS, Ridge JA, Migliacci JC, Loomis AM, Shah JP. Head and neck cancers—major changes in the American Joint Committee on cancer eighth edition cancer staging manual. CA: a cancer journal for clinicians. 2017 Mar;67(2):122-37.

8. Prabhu RS, Magliocca KR, Hanasoge S, et al. Accuracy of computed tomography for predicting pathologic nodal extracapsular extension in patients with head-and-neck cancer undergoing initial surgical resection. Int J Radiat Oncol Biol Phys. 2014;88:122-129.

9. Carlton JA, et al. Computed tomography detection of extracapsular spread of squamous cell carcinoma of the head and neck in metastatic cervical lymph nodes. Neuroradiol. J. 2017;30:222–229.

10. Chai RL, et al. Accuracy of computed tomography in the prediction of extracapsular spread of lymph node metastases in squamous cell carcinoma of the head and neck. JAMA Otolaryngol.–Head Neck Surg. 2013;139:1187– 1194. doi: 10.1001/jamaoto.2013.4491.