OncoNoteBERT: A Foundation Representation Model for Natural Language Processing of Real-World Outpatient Oncology Notes

OncoNoteBERT, a BERT-style encoder trained from scratch on 290,026 UK outpatient oncology notes from 21,564 lung and head-and-neck cancer patients, achieved a masked language modelling perplexity of 2.10 after continued pretraining, outperforming RadBERT and PathologyBERT in zero-shot evaluation.