While awaiting the COVID-19 vaccines, researchers have been actively exploring the effectiveness of existing vaccines against the new virus, among which the BCG vaccine (Bacillus Calmette-Guérin) receives the most attention. While many reports suggest a potential role for BCG immunization in ameliorating SARS-CoV-2 infection, these findings remain controversial. With country-level COVID-19 outbreak data from Johns Hopkins University Coronavirus Resource Center, and BCG program data from World Atlas of BCG Policies and Practices and WHO/UNICE, we estimated a dynamic model to investigate the effect of BCG vaccination across time during the pandemic. Our results reconcile these varying reports regarding protection by BCG against COVID-19 in a variety of clinical scenarios and model specifications. We observe a notable protective effect of the BCG vaccine during the early stage of the pandemic. However, we do not see any strong evidence for protection during the later stages. We also see that a higher proportion of vaccinated young population may confer some level of communal protection against the virus in the early pandemic period, even when the proportion of vaccination in the older population is low. Our results highlight that while BCG may offer some protection against COVID-19, we should be cautious in interpreting the estimated effectiveness as it may vary over time and depend on the age structure of the vaccinated population.