BMS-986205 and Nivolumab as First or Second Line Therapy in Treating Patients With Liver Cancer

Inclusion Criteria: * Willing and able to provide written informed consent for the trial * Life expectancy \> 12 weeks * Histologically or imaging confirmed hepatocellular carcinoma (mixed hepatocellular/cholangiocarcinoma or fibrolamellar subtypes are excluded) * Have disease that is not amenable for curative treatment approach * Have measurable disease based on RECIST v1.1 * \>= 1 liver lesions accessible for core biopsy that was either not previously treated by liver-directed therapy or progressed following liver-directed therapy * Child-Pugh score of A * Eastern Cooperative Oncology Group (ECOG) performance status of 0 or 1 * Absolute neutrophil count (ANC) \>= 1000 cell/mm\^3 * Platelet count \>= 50,000/mm\^3 * Hemoglobin (Hgb) \>= 8 g/dL * Aspartate aminotransferase (AST)/serum glutamic oxaloacetic transaminase (SGOT), alanine aminotransferase (ALT)/serum glutamic pyruvic transaminase (SGPT) =\< 5 x upper limit of normal (ULN) * Total bilirubin =\< 2 ULN * Creatinine =\< 2 x ULN * Subjects with active hepatitis B virus (hep B) are allowed if antiviral therapy for hepatitis B has been given for \> 8 weeks and viral load is \< 100 IU/ml prior to first dose of trial treatment. Subjects with untreated hepatitis C virus (HCV) are allowed * Willingness to undergo mandatory pre-treatment biopsy (unless there is adequate archival tumor specimen available) and mandatory on-treatment biopsy * Female subject of child-bearing potential must have a negative urine pregnancy =\< 24 h…