PURPOSE: Assess whether we can forecast future rapid visual field (VF) worsening using deep learning models (DLMs) trained on early VF, OCT, and clinical data. DESIGN: Retrospective cohort study. SUBJECTS: 4,536 eyes from 2,962 patients. 263 (5.80%) of eyes underwent rapid VF worsening (MD slope <-1dB/yr across all VFs). METHODS: We included eyes that met the following criteria: 1) followed for glaucoma or suspect status 2) had at least five longitudinal reliable VFs (VF1, VF2, VF3, VF4, VF5) 3) had one reliable baseline Optical Coherence Tomography (OCT) scan (OCT1) and one set of baseline clinical measurements (Clinical1) at the time of VF1. We designed a DLM to forecast future rapid VF worsening. The input consisted of spatially oriented total deviation values from VF1 (including or not including VF2 and VF3 in some models) and retinal nerve fiber layer thickness values from the baseline OCT. We passed this VF/OCT stack into a vision transformer feature extractor, the output of which was concatenated with baseline clinical data before putting it through a linear classifier to predict that eye’s risk of rapid VF worsening across the five VFs. We compared the performance of models with differing inputs by computing area under receiver operating curve (AUC) in the test set. Specifically, we trained models with the following inputs: Model V: VF1; VC: VF1+ Clinical1; VO: VF1+ OCT1; VOC: VF1+ Clinical1+ OCT1; V2: VF1 + VF2; V2OC: VF1 + VF2 + Clinical1 + OCT1; V3: VF1 + VF2 + VF3; V3OC: VF1 + VF2 + VF3 + Clinical1 + OCT1. MAIN OUTCOME MEASURES: AUC of DLMs when forecasting rapidly worsening eyes. RESULTS: Model V3OC best forecasted rapid worsening with an AUC (95% CI) of 0.87 (0.77, 0.97). Remaining models in descending order of performance and their respective AUC [95% CI] were: Model V3 (0.84 [0.74 to 0.95]), Model V2OC (0.81 [0.70 to 0.92]), Model V2 (0.81 [0.70 to 0.82]), Model VOC (0.77 [0.65, 0.88]), Model VO [0.75 [0.64, 0.88], Model VC (0.75 [0.63, 0.87]), Model V (0.74 [0.62, 0.86]). CONCLUSION: DLMs can forecast future rapid glaucoma worsening with modest to high performance when trained using data from early in the disease course. Including baseline data from multiple modalities and subsequent visits improves performance beyond using VF data alone.