Pomalidomide in Treating Younger Patients With Recurrent, Progressive, or Refractory Central Nervous System Tumors

Inclusion Criteria: * Patients must have received standard therapy (or generally accepted upfront therapy if no standard exists) and have no known curative therapy * Patients with a histologically confirmed diagnosis of a primary CNS tumor that is recurrent, progressive or refractory to standard therapy; refractory disease will be defined as the presence of persistent abnormality on conventional magnetic resonance imaging (MRI) imaging that is further distinguished by histology (biopsy or sample of lesion) or advanced imaging, OR as determined by the treating physician and discussed with the primary investigator prior to enrollment; all tumors must have histological verification at either the time of diagnosis or recurrence except for patients with diffuse intrinsic brain stem tumors or optic pathway gliomas; patients with neurofibromatosis type-I (NF-1) associated CNS tumors are eligible if they meet all other eligibility criteria * Patients must have evaluable disease on MRI imaging * Patients must have body surface area (BSA) \> 0.55 m\^2 at the time of enrollment * In the event of de-escalation from dose level 1 to dose level 0, patients with BSAs \< 0.67 m\^2 are not eligible * Patients must have recovered from clinically significant, acute, treatment-related toxicities of prior therapies; for those acute baseline adverse events attributable to prior therapy, recovery is defined as a toxicity grade =\< 2, using Common Terminology Criteria for Adverse Events (CTCAE) v…