Nivolumab and Multi-fraction Stereotactic Radiosurgery With or Without Ipilimumab in Treating Patients With Recurrent Grade II-III Meningioma

Inclusion Criteria: * Patients must have histologically confirmed World Health Organization (WHO) grade II-III meningioma which has relapsed after prior radiation therapy with radiologically progressive or recurrent disease * Patients must have measurable disease, defined as at least 1 lesion that can be accurately measured in at least one dimension as \>= 1 cm on brain magnetic resonance imaging (MRI) but with the maximum dimension =\< 5 cm OR gross tumor volume \< 20 cm\^3. All the relapsed disease would need to be eligible to be treated with reirradiation * Patients must have at least one prior surgery with available archival formalin-fixed paraffin-embedded (FFPE) tumor blocks of the initial or recurrent meningioma. If there are multiple tumor blocks from multiple surgeries, the most recent tumor block (and ideally of the relapsed tumor after initial radiation therapy) should be submitted. If a tumor block is not available, an alternative of 20-30 unstained slides may be submitted instead. Annotation regarding whether the tumor block is before or after initial radiation therapy should be provided * Prior initial radiation therapy may include external beam radiation or radiosurgery, or combination of both. However, the total dose of prior radiation exposure to the site of recurrent tumor (for consideration of re-irradiation) cannot be more than 70 Gy. The duration since the previous radiation exposure to the site of reirradiation need to be at least 6 months * Age \>= 18 …