Durvalumab With or Without Lenalidomide in Treating Patients With Relapsed or Refractory Cutaneous or Peripheral T Cell Lymphoma

Inclusion Criteria: * Documented informed consent of the participant and/or legally authorized representative * Registered into Revlimid REMS program * Eastern Cooperative Oncology Group (ECOG) performance status of 0-2 * Fully recovered from acute toxicities (except alopecia) of all prior therapies to Common Terminology Criteria for Adverse Events (CTCAE) =\< grade 1 * Relapsed/refractory disease * Failed \>= 2 prior systemic therapies \*NOTE: For systemic ALCL prior systemic therapy must also include progression on brentuximab vedotin CUTANEOUS T-CELL LYMPHOMA (CTCL) ONLY * Histologically confirmed mycosis fungoides (MF) or Sezary syndrome (SS); Phase 1: \>= stage IIB OR \>= stage IB-IIA folliculotropic/transformed MF; Phase 2: \>= stage IB * Stage of disease according to TNMB classification * Pathology report must be diagnostic or be consistent with MF/SS criteria * SS is defined as meeting T4 plus B2 criteria; where the biopsy of erythrodermic skin may only reveal suggestive but not diagnostic histopathological features, the diagnosis may be based on either node biopsy or fulfillment of B2 criteria * For MF where the histological diagnosis by light microscopic examination is not confirmed, diagnostic criteria that has been recommended by the International Society of Cutaneous Lymphomas (ISCL) should be used * Measurable disease per modified severity weighted assessment tool (mSWAT) and/or Sezary count * Baseline skin biopsy taken within 6 months available for …