Nivolumab With or Without Ipilimumab in Treating Patients With Gastrointestinal Stromal Tumor That Is Metastatic or Cannot Be Removed by Surgery

Inclusion Criteria: * Written informed consent must be obtained from the subject/legal representative prior to performing any protocol-related procedures, including screening evaluations * Eastern Cooperative Oncology Group (ECOG) performance status (PS) 0-1 * Life expectancy \> 12 weeks * Histological confirmation of GIST * Mutational testing of patient samples for KIT and platelet-derived growth factor receptor (PDGFR) mutations; (this will not hold up starting therapy, but will be done for all patients lacking up front mutational testing) * Patients must have refused or have evidence of intolerance to or progression on imatinib * This study permits the re-enrollment of a subject that has discontinued the study as a pre-treatment failure (i.e., subject has not been randomized / has not been treated); if re-enrolled, the subject must be re-consented * Adequate archival tissue must be available from the prior 3 months to signing consent; if not, an adequate tumor specimen obtained by either excisional biopsy, incisional biopsy or core needle biopsy must be sent to the central pathology lab for evaluation; the material must measure at least 0.8 × 0.1 cm in size or contain at least 100 tumor cells * Measurable tumor lesions according to Response Evaluation Criteria in Solid Tumors (RECIST) 1.1 criteria * Hemoglobin \>= 9 g/dL * Absolute neutrophil count \>= 1,500/mm\^3 * Platelet count \>= 100,000/mm\^3 * Aspartate aminotransferase (AST) and alanine aminotransferase (ALT) =\< …