Nivolumab in Treating Patients With Metastatic Adrenocortical Cancer

Inclusion Criteria: * Patients must have a histologically confirmed stage IV or unresectable locally advanced adrenocortical carcinoma * Patients must have disease progressing after treatment with at least one line of therapy including mitotane and/or chemotherapy; Note: patients declining first line treatment with mitotane and/or chemotherapy based on limited efficacy are also eligible for this study * Patients must have measurable disease according to the standard RECIST version 1.1; CT scans or magnetic resonance imaging (MRIs) used to assess the measurable disease must have been completed within 28 days prior to registration * Patients must exhibit an Eastern Cooperative Oncology Group (ECOG) performance status of 0-3 * Leukocytes \>= 2,000/mcL * Absolute neutrophil count \>= 1,500/mcL * Hemoglobin \>= 9 g/dL * Platelets \>= 100,000/mcL * Total bilirubin =\< 1.5 × institutional upper limit of normal (ULN) (except patients with Gilbert Syndrome, who can have total bilirubin \< 3.0 mg/dL) * Aspartate transaminase (AST) (serum glutamic-oxaloacetic transaminase \[SGOT\])/alanine transferase (ALT)(serum glutamate pyruvate transaminase \[SGPT\]) =\< 2.5 x ULN * Serum creatinine of \< 3.0 x ULN (upper limit of normal) or creatinine clearance (CrCl) \> 30 mL/minute (using Cockcroft/Gault formula below) * Patients with history of central nervous system (CNS) metastases are eligible if CNS disease has been radiographically and neurologically stable for at least 6 weeks prior to st…