Pembrolizumab and Stereotactic Body Radiation Therapy or Non-Stereotactic Wide-Field Radiation Therapy in Treating Patients With Non-small Cell Lung Cancer

Inclusion Criteria: * Pathologically confirmed non-small lung cancer; for patients in group 5, any solid tumor histology to be included * Stage IV metastatic disease (only during the phase II) * At least one thoracic or liver lesion amenable to radiation, for group 5 we need one area that can safely receive SBRT or WFRT, not restricted to lung or liver sites * At least one additional non-contiguous lesion to the irradiated lesion amenable to radiographic evaluation * Be willing and able to provide written informed consent/assent for the trial * Have measurable disease based on immune related response criteria (irRC) criteria * Have a performance status of 0 or 1 on the Eastern Cooperative Oncology Group (ECOG) performance scale * Absolute neutrophil count (ANC) \>= 1,500 /mcL (performed within 28 days prior to study registration up to the first dose of study drug) * Platelets \>= 100,000 /mcL (performed within 28 days prior to study registration up to the first dose of study drug) * Hemoglobin \>= 9 g/dL or \>= 5.6 mmol/L (performed within 28 days prior to study registration up to the first dose of study drug) * Serum creatinine =\< 1.5 X upper limit of normal (ULN) or measured or calculated creatinine clearance (glomerular filtration rate \[GFR\] can also be used in place of creatinine or calculated creatinine clearance \[CrCl\]) or \>= 60 mL/min for subject with creatinine levels \> 1.5 X institutional ULN (creatinine clearance should be calculated per institutional standa…