TAS-102 and Radiation Therapy in Treating Patients With Rectal Cancer That Is Locally Recurrent, Metastatic, or Cannot Be Removed by Surgery

Inclusion Criteria: * Histological or cytological confirmation of locally recurrent or metastatic rectal adenocarcinoma * Note: Patients with locally recurrent/persistent disease within the pelvis after primary therapy (chemotherapy, surgery, and/or radiotherapy) are eligible * Note: Patients who have had prior pelvic radiotherapy with a total dose of =\< 54 Gy are eligible * Note: Patients with or without metastatic disease (excluding untreated central nervous system \[CNS\] metastasis), with primary pelvic disease or pelvic recurrence are eligible * Note: Patients with pelvic disease that is potentially resectable or unresectable are eligible * Measurable disease * Eastern Cooperative Oncology Group (ECOG) performance status (PS) 0, 1 or 2 * Expected life expectancy \>= 12 weeks * Obtained =\< 14 days prior to registration: Absolute neutrophil count (ANC) \>= 1500/mm\^3 * Obtained =\< 14 days prior to registration: Platelet count \>= 100,000/mm\^3 * Obtained =\< 14 days prior to registration: Hemoglobin \>= 9.0 g/dL * Obtained =\< 14 days prior to registration: Total bilirubin =\< 1.5 x upper limit of normal (ULN) (in patients with well-documented Gilbert's syndrome and the total bilirubin is grade 1, then direct bilirubin value must be =\< 1.0 mg/dL) * Obtained =\< 14 days prior to registration: Aspartate transaminase (AST) =\< 2 x ULN (=\< 5 x ULN for patients with liver involvement) * Obtained =\< 14 days prior to registration: Alanine aminotransferase (ALT) =\…