Pembrolizumab in Treating Participants With Metastatic, Recurrent or Locally Advanced Cancer and Genomic Instability

Inclusion Criteria: * Be willing and able to provide written informed consent/assent for the trial * Have a diagnosis of a tumor with evidence of genomic instability on Clinical Laboratory Improvement Amendments (CLIA) certified genomic testing, inclusive of mutations in POLE, POLD1 for arm 1 and in BRCA1 and BRCA2 for arm 2; in arm 2, enrollment of breast and ovarian histologies will be limited to a total of 10 patients * Have advanced cancer (metastatic, recurrent or locally advanced) and measurable disease based on Response Evaluation Criteria in Solid Tumors (RECIST) 1.1 * Be willing to provide archived tumor tissue; tissue from the most obtained core or excisional biopsy of a tumor lesion is preferred; 20 unstained slides are preferred but a minimum of 15 slides will be acceptable; if adequate tissue is not present the patient may consent to a newly obtained biopsy * Have a performance status of 0 or 1 on the Eastern Cooperative Oncology Group (ECOG) performance scale * Absolute neutrophil count (ANC) \>= 1,500 /mcL, performed within 10 days of treatment * Platelets \>= 100,000 / mcL, performed within 10 days of treatment * Hemoglobin \>= 9 g/dL or \>= 5.6 mmol/L without transfusion or erythropoietin (EPO) dependency (within 7 days of assessment), performed within 10 days of treatment * Serum creatinine =\< 1.5 X upper limit of normal (ULN) OR Measured or calculated creatinine clearance \>= 60 mL/min for subject with creatinine levels \> 1.5 X institutional ULN (glomeru…