Axitinib and Avelumab in Treating Patients With Recurrent or Metastatic Adenoid Cystic Carcinoma

Inclusion Criteria: * Eastern Cooperative Oncology Group ECOG performance status 0 or 1 * Histologically confirmed recurrent or metastatic adenoid cystic carcinoma not amenable to curative intent surgery or radiotherapy * Measurable disease per RECIST 1.1 * Evidence of disease progression within 6 months of study enrollment or worsening disease-related symptoms * Previously untreated subjects and subject treated with any number of prior lines of therapy are eligible * Absolute neutrophil count (ANC) \>= 1.5 x 10\^9/L * Platelet count \>= 100 x 10\^9/L * Hemoglobin \>= 9 g/dL (may have been transfused) * Total bilirubin level =\< 1.5 x the upper limit of normal (ULN) range * Aspartate aminotransferase (AST) and alanine aminotransferase (ALT) levels =\< 2.5 x ULN or AST and ALT levels =\< 5 x ULN (for subjects with documented metastatic disease to the liver) * Estimated creatinine clearance \>= 30 mL/min according to the Cockcroft-Gault formula (or local institutional standard method) * Must have archival tissue (formalin-fixed, paraffin-embedded \[FFPE\] tissue available-minimum of 15 unstained slides) or be willing to undergo a biopsy * For patients receiving anti-therapeutic coagulation, patients must be on stable anticoagulant regimen and international normalized ratio (INR) or activated partial thromboplastin time (aPTT) must be =\< 1.5 upper limit of normal * Females of childbearing potential must not be breast feeding and must have a negative serum or urine pregnancy te…