Durvalumab and Olaparib for the Treatment of Prostate Cancer in Men Predicted to Have a High Neoantigen Load

Inclusion Criteria: * Histologic confirmation of adenocarcinoma of the prostate * The patient must have received definitive local therapy for prostate cancer, consisting of either radiation therapy and/or prostatectomy (salvage or adjuvant radiation post-prostatectomy is not exclusionary) * PSA must be \>= 2 ng/ml if received only prior definitive radiation (no PSA threshold required if prior prostatectomy was performed) with a PSA doubling time (PSADT) =\< 10 months: * PSADT calculation must include all recorded PSA values \> 0.2 ng/ml over the past 6 months prior to randomization, with a minimum of 3 values spaced at least 2 weeks apart, with each included value preferably measured at the same laboratory. PSA values obtained prior to localized therapy will be excluded * The calculation of PSADT is based on the natural log of PSA * Prior salvage radiation or not a candidate for localized salvage radiation due to subject preference or clinical assessment based upon disease characteristics and/or subject co-morbidities * Prior hormonal therapy (i.e. androgen deprivation therapy) when given as neoadjuvant/concurrent/adjuvant therapy along with definitive radiation is allowed, provided this was stopped \>= 6 months prior to starting treatment per protocol AND testosterone is \>= 150 ng/dl * No evidence of metastatic disease on imaging by whole body bone scan and computed tomography (CT) or magnetic resonance imaging (MRI) of the chest/abdomen/pelvis within 6 weeks before s…