Low-Dose or High-Dose Lenalidomide in Treating Younger Patients With Recurrent, Refractory, or Progressive Pilocytic Astrocytoma or Optic Pathway Glioma

Inclusion Criteria: * Patients must have a body surface area (BSA) \>= 0.4 m\^2 at the time of study enrollment * Patients must have a pilocytic astrocytoma or optic pathway glioma that has relapsed, progressed, or become refractory to conventional therapy; patients with neurofibromatosis (NF-1) are eligible * Patients must have histologic verification of malignancy; histologic confirmation for patients with optic pathway gliomas will not be required * Patients must have measurable residual disease, defined as tumor that is measurable in two perpendicular diameters on magnetic resonance imaging (MRI); for a lesion to be considered measurable, it must be at least twice the slice thickness on MRI (i.e. visible on more than one slice) * To document the degree of residual tumor, the following must be obtained: * All patients must have a brain MRI with and without contrast (gadolinium) within 1 week prior to study enrollment; for patients on steroids, baseline MRI scans must be performed after at least 1 week at a stable or decreasing dose of steroids * All patients with a history of spinal or leptomeningeal disease, and those patients with symptoms suspicious of spinal disease, must have a spine MRI with and without contrast (gadolinium) performed within 2 weeks prior to study enrollment * Patients must have a Lansky or Karnofsky performance status score of \>= 60%; use Karnofsky for patients \> 16 years of age and Lansky for patients =\< 16 years of age * Patients must hav…