Liposomal Irinotecan, Fluorouracil, Leucovorin Calcium, and Rucaparib in Treating Patients With Metastatic Pancreatic, Colorectal, Gastroesophageal, or Biliary Cancer

Inclusion Criteria: * Phase I only: Histologic confirmation of pancreatic, colorectal, gastroesophageal or biliary adenocarcinoma, as follows: * Patients with metastatic disease from pancreatic cancer who received no more than 2 lines of prior therapy in the metastatic setting * Patients with metastatic disease from colorectal cancer who received no more than 3 lines of prior therapy in the metastatic setting * Patients with metastatic disease from gastroesophageal cancer who received no more than 1 line of prior therapy in the metastatic setting * Patients with metastatic disease from biliary tract cancer who received no more than 1 line of prior therapy in the metastatic setting * NOTE: No prior exposure to irinotecan in the metastatic setting will be allowed except in the phase I dose escalation portion and in colon cancer patients only; in pancreas cancer, exposure to irinotecan is only allowed in the neoadjuvant setting and no progressive disease \< 3 months from last dose of irinotecan * Phase Ib only: Patients with metastatic adenocarcinoma of the pancreas who have who received no more than 1 line of prior therapy in the metastatic setting * NOTE: Exposure to irinotecan is only allowed in the neoadjuvant setting and no progressive disease \< 3 months from last dose of irinotecan * Phase II only: Patients with metastatic adenocarcinoma of the pancreas with genomic markers (signature) of homologous recombination deficiency (HRD) or BRCA1 or BRCA2 or PALB2 …