Regorafenib and Pembrolizumab in Treating Participants With Advanced or Metastatic Colorectal Cancer

Inclusion Criteria: * Patients who provided written informed consent to be subjects in this trial * Patients with histologically or cytologically confirmed advanced or metastatic colorectal cancer who had failed or are intolerant of oxaliplatin, irinotecan, and fluorouracil (5-FU) * Patients with an Eastern Cooperative Oncology Group (ECOG) performance status (PS) of 0 or 1 * Patients capable of taking oral medication * Patients with evaluable or measurable lesions as per Response Evaluation Criteria in Solid Tumors (RECIST) version 1.1 * Neutrophil count \>= 200/mm\^3 * Platelet count \>= 7.5 x 10\^4/mm\^3 (transfusion \> 2 weeks before testing permitted) * Aspartate transaminase (AST), alanine transaminase (ALT) =\< 2.5-times the upper limit of normal (=\< 5-times in patients with liver metastasis) * Total bilirubin =\< 1.5-times the upper limit of normal * Creatinine =\< 1.5-times the upper limit of normal * Lipase =\< 1.5 x the upper limit of normal (ULN) * International normalized ratio (INR) =\< 1.5 x ULN and partial thromboplastin time (PTT) or activated partial thromboplastin time (aPTT) =\< 1.5 x ULN unless receiving treatment with therapeutic anticoagulation. Patients being treated with anticoagulant, e.g. heparin, will be allowed to participate provided no prior evidence of an underlying abnormality in these parameters exists. Close monitoring of at least weekly evaluations will be performed until INR and PTT are stable based on a pre-dose measurement as defined b…