Ipilimumab and Nivolumab in Treating Patients With Recurrent Stage IV HER2 Negative Inflammatory Breast Cancer

Inclusion Criteria: * Patients must have histologically or cytologically confirmed stage IV breast carcinoma with a previous clinical diagnosis of IBC based on the presence of inflammatory changes in the involved breast, such as diffuse erythema and edema (peau d'orange), with or without an underlying palpable mass involving the majority of the skin of the breast; pathological evidence of dermal lymphatic invasion should be noted but is not required for diagnosis * Patients must have local or metastatic recurrence of IBC after prior surgery * Patients must have a metastatic tumor negative for HER2; the lack of HER2 overexpression by immunohistochemistry (IHC), is defined as 0 or 1+ where as hyperexpression is defined as 3+; if equivocal IHC, 2+, the tumor must be non-gene amplified by fluorescence in situ hybridization (FISH) performed upon the primary tumor or metastatic lesion (ratio \< 2 and HER2 copy number \< 4) * Patients may have measurable disease, defined as at least one lesion that can be accurately measured in at least one dimension in accordance with RECIST criteria v. 1.1 OR non-measurable tumors; NOTE: Non-measurable tumors are small lesions (longest diameter \< 10mm or pathological lymph nodes with \>= 10 to \< 15 mm short axis); bone lesions, leptomeningeal disease, ascites, pleural/pericardial effusions, lymphangitis cutis/pulmonitis, inflammatory breast disease, and abdominal masses (not followed by computed tomography \[CT\] or magnetic resonance imaging \…