Nivolumab and Pomalidomide in Treating Patients With Relapsed or Refractory Central Nervous System Diffuse Large B Cell Lymphoma or Primary Vitreoretinal Diffuse Large B Cell Lymphoma

Inclusion Criteria: * Patient must have one of the following: * Relapsed or refractory primary central nervous system (CNS) diffuse large B cell lymphoma (PCNSDLBCL) with a brain lesion \>= 1 cm, or with cerebrospinal fluid (CSF) relapse with positive CSF cytology, or with ocular relapse with positive ocular tissue biopsy * NOTE: Tissue biopsy is not absolutely necessary for brain tumor unless clinical and radiologic findings strongly suggest other etiologies as per treating physician. Initial diagnosis must be made by tissue biopsy OR * Relapsed/refractory primary vitreoretinal diffuse large B cell lymphoma (DLBCL) with a CNS lesion \>= 1 cm, or with CSF relapse with positive CSF cytology, or with ocular relapse with positive ocular tissue biopsy. Relapsed PVRL must have progressed or failed at least one systemic regimen * NOTE: Intraocular treatments are not regarded as systemic therapy * NOTE: If recurrence in ocular or leptomeningeal space, the patient will need a positive ocular tissue biopsy and CSF biopsy. Tissue biopsy requirement of the CNS lesion is as outlined in bullet above * Patient progressed after or did not respond to at least 1 line of systemic therapy (e.g., high-dose methotrexate, high-dose methotrexate based regimen, high dose cytarabine, etc) * Eastern Cooperative Oncology Group (ECOG) performance status (PS) 0, 1, 2 or 3 * Absolute neutrophil count (ANC) \>= 1500/mm\^3 (within =\< 14 days prior to registration) * Platelet count \>= 1…