Durvalumab and Tremelimumab With or Without High or Low-Dose Radiation Therapy in Treating Patients With Metastatic Colorectal or Non-small Cell Lung Cancer

Inclusion Criteria: * Patients must have histologically or cytologically confirmed non-small cell lung cancer (cohort 1) or colorectal cancer (cohort 2) * Patients must have measurable disease, defined as at least one lesion that can be accurately measured in at least one dimension (longest diameter to be recorded for non-nodal lesions and short axis for nodal lesions) as \>= 20 mm (\>= 2 cm) with conventional techniques or as \>= 10 mm (\>= 1.5 cm) with spiral computed tomography (CT) scan, magnetic resonance imaging (MRI), or calipers by clinical exam * Patients in both cohorts must have progressive disease following prior therapy; specifically: * Cohort 1 (NSCLC): Patients must have evidence of radiologic or clinical disease progression during previous treatment with systemic PD-1 directed therapy and/or have been deemed not to derive clinical benefit from PD-1 directed treatment; this includes patients who demonstrated an initial response and subsequent progression; no prior treatment with chemotherapy or targeted agents are required; intervening therapy is allowed between previous PD-1 directed treatment and there is no required interval from prior PD-1 treatment required; PD-1 directed treatment includes treatment with antibodies targeting the PD-1 receptor such as pembrolizumab or nivolumab, as well as PD-L1 targeted antibodies such as MEDI4736 (durvalumab), atezolizumab and avelumab; these agents may have been administered as part of a clinical trial * Cohort 2 …