Background It is recommended by the latest guidelines that common bile duct stones (CBDSs) should be removed, preferentially endoscopically, regardless of the presence of symptoms or complications. However, it may not be feasible due to very-old age and/or co-morbidities or rejected by patients. Aim We aimed to identify factors for development of complications after diagnosis of CBDSs to select vulnerable patients for whom removal of CBDSs should be performed despite of risks. Methods Medical records of patients with radiologically-confirmed CBDSs between October 2005 and September 2019 were retrospectively analyzed. Results Among a total of 634 consecutive patients, complications, defined as obstructive jaundice, biliary pancreatitis, or acute cholangitis, were already present in 416 (65.6%) at the time of diagnosis of CBDSs and developed in 95 (15.0%) afterwards. Forty four (6.9%) patients remained silent with the median follow-up periods of 31.5 months. With multivariate analyses, untreatment within 1 month and the size of CBDSs > 5 mm turned out to be independent risk factors for development of complications. Spontaneous passage of CBDSs was confirmed radiologically in 9 (11.1%) out of 81 patients untreated within 1 month. Conclusions CBDSs should be removed within 1 month from the diagnosis, even in unsuitable patients, especially if the size exceeds 5 mm.