Tremelimumab and Durvalumab in Treating Patients With Colorectal Cancer With Liver Metastases That Can Be Removed by Surgery

Inclusion Criteria: * Patients must have histologically or cytologically confirmed colorectal cancer with liver metastases deemed resectable by a general or liver surgeon (resectability may involve the use of ablative techniques to some but not all liver metastases); those patients with known disease outside of the liver are not eligible (except for patients with primary lesions in place that are planned for resection or nonspecific lung metastases \< 1 cm) * Patients must have measurable disease, defined as at least one lesion that can be accurately measured in at least one dimension (longest diameter to be recorded) as \>= 10 mm with spiral computed tomography (CT) scan * All lines of prior therapy accepted; subjects with prior hepatic or extra-hepatic resections of metastatic disease will be included * Life expectancy of greater than 6 months * Eastern Cooperative Oncology Group (ECOG) performance status =\< 1 (Karnofsky \>= 70%) * Leukocytes \>= 3,000/mcL * Absolute neutrophil count \>= 1,500/mcL * Platelets \>= 100,000/mcL * Total bilirubin \< 1.5 X institutional normal limits (subjects with known Gilbert syndrome are eligible with total bilirubin \< 3.0 mg/dL) * Aspartate aminotransferase (AST) (serum glutamic oxaloacetic transaminase \[SGOT\])/alanine aminotransferase (ALT) (serum glutamic-pyruvic transaminase \[SGPT\]) =\< 3 X institutional upper limit of normal * Creatinine within normal institutional limits OR creatinine clearance \>= 60 mL/min/1.73 m\^2 for patien…