Introduction: The Spanish Society of Immunology recently warned that a history of past COVID-19 could result in a higher incidence of adverse events (AEs) related to vaccination. We set out to analyze whether there were any differences in AEs between healthcare workers vaccinated for COVID-19 (either after the first or second dose) who had had a prior diagnosis SARS-CoV-2 infection compared to those who had not had COVID-19 before vaccination. Methods: This was a retrospective cohort study in a population of healthcare workers. AEs related to the first and second doses of the Pfizer vaccine were recorded. We compared the incidence of AEs and compared individuals with 0–3 different AEs to those with 4 or more AEs. The relative risks (RR) and their 95% confidence intervals were calculated. Results: Past infection was associated with having more AEs after the first dose (p < 0.001), but not the second one (p = 0.476), as well as a higher incidence of AEs (p < 0.001). Common AEs that were statistically associated with past COVID infection included arthralgia, asthenia, fever, chills, headache, and myalgia (p ≤ 0.001). The RR for having an increased absolute number of different AEs was 1.18 (95%CI [1.05, 1.33]) after the first dose and 1.05 (95%CI [0.96, 1.14]) after the second dose. Conclusions: Our results showed that the incidence of AEs was higher in individuals with a history of prior COVID-19 infection.