Localized Radiation Therapy or Recombinant Interferon Beta and Avelumab With or Without Cellular Adoptive Immunotherapy in Treating Patients With Metastatic Merkel Cell Carcinoma

Inclusion Criteria: * Signed written informed consent * Confirmation of MCC by internal pathology review of initial or subsequent biopsy or other pathologic material * If an accessible lesion is present, a biopsy will be performed within 6 weeks of the start of study intervention; the results of the biopsy must be obtained prior to initiation of study intervention * Evidence of MCPyV TAg tumor expression by immunohistochemistry on any prior or current tumor specimen or viral oncoprotein antibody confirmation within 6 weeks of the start of study intervention * Eastern Cooperative Oncology Group (ECOG) performance status (PS) =\< 2 at trial entry * Patients must have at least one bi-dimensionally measurable lesion by palpation, clinical exam, or radiographic imaging within 6 weeks of the start of study intervention (X-ray, computed tomography \[CT\] scan, positron emission tomography \[PET\] scan, magnetic resonance imaging \[MRI\], or ultrasound) * For patients designated to be treated on Group 2: cardiac ejection fraction \>= 35%; for patients with significant risk factors for coronary artery disease (Framingham risk score \> 15%), a cardiac stress test is recommended * At least 3 weeks must have passed since any of the following: systemic corticosteroids, immunotherapy (for example, T-cell infusions, immunomodulatory agents, interleukins, MCC vaccines, intravenous immunoglobulin, expanded polyclonal tumor infiltrating lymphocytes \[TIL\] or lymphokine-activated killer \[LAK…