Durvalumab With or Without Tremelimumab in Treating Participants With Stage II-IVA Oropharyngeal Squamous Cell Cancer

Inclusion Criteria: * Written informed consent to participate in the study according to the investigational review board (IRB) * Suspected or histologically/cytologically confirmed oropharyngeal squamous cell carcinoma (OPSCC), stage II, III, or IVA (according to the American Joint Committee on Cancer \[AJCC\] 7th edition), or patients with loco-regional recurrence from an OPSCC primary, if time of recurrence is at least 6 months after completion of initial curative intent treatment (surgery or radiotherapy +/- chemotherapy or cetuximab). Patients with a suspected lesion may be enrolled and a baseline biopsy will be obtained as part of the study. If squamous cell histology is not confirmed, patients will be discontinued from the study * Patients must have surgically resectable disease in the opinion of the treating physician. For patients with a primary OPSCC, patients must be eligible for TORS (transoral robotic surgery) * Available tissue from prior biopsy (minimum of 10 unstained slides), or willing to undergo core biopsy to obtain tumor material. Biopsy will be mandatory for patients with recurrent disease * Eastern Cooperative Oncology Group (ECOG) performance status of 0 or 1 * Hemoglobin \>= 9.0 g/dL * Absolute neutrophil count (ANC) \>= 1.5 x 10\^9/L (\> 1500 per mm\^3) * Platelet count \>= 100 x 10\^9/L (\> 100,000 per mm\^3) * Serum bilirubin =\< 1.5 x institutional upper limit of normal (ULN). This will not apply to subjects with confirmed Gilbert's syndrome (pers…