Testing the PD-1 Antibody, MK3475, Given With Ziv-aflibercept in Patients With Advanced Cancer

Inclusion Criteria: * In dose escalation, patients must have histologically or cytologically confirmed metastatic disease from any solid tumor (NOS 10029000). In dose expansion part 1, patients must have histologically or cytologically confirmed metastatic melanoma (10053571), renal cell carcinoma (NOS 10038485), ovarian cancer (NOS 10033272), or colorectal cancer (10009951,10038045). In dose expansion part 2, patients must have PD-1 resistant melanoma (10053571), PD-1 resistant renal cancer (NOS 10038485), or sarcoma (10061271) * Renal cell patients must have had at least one prior vascular endothelial growth factor (VEGF) tyrosine kinase inhibitor (TKI) * Ovarian cancer patients must be resistant to platinum therapy; therapy (i.e. within 6 months of last platinum therapy); patients who received greater than two prior platinum containing regimens will not be eligible * Patients with colorectal cancer should have failed at least one oxaliplatin-containing regimen * No more than two prior therapies for metastatic disease * Age \>= 18 years. Because no dosing or adverse event data are currently available on the use of MK-3475 in combination with ziv-aflibercept in patients \< 18 years of age, children are excluded from this study, but will be eligible for future pediatric trials * Eastern Cooperative Oncology Group (ECOG) performance status =\< 1 (Karnofsky \>= 70%) * Estimated life expectancy of greater than 6 months * Leukocytes \>= 2,000/mcL (within 10 days of treatment ini…