Z-Endoxifen Hydrochloride in Treating Patients With Metastatic or Locally Recurrent Estrogen Receptor-Positive Breast Cancer

Inclusion Criteria: * Histologically confirmed diagnosis of metastatic or locally recurrent breast cancer * ER positive defined as \> 1% nuclear staining on the biopsy that was obtained at the confirmation of metastatic or locally recurrent disease * Lesion type of either evaluable or measurable disease * Pre- or post-menopausal female * For the expansion cohorts: tumor that is accessible for biopsy * Eastern Cooperative Oncology Group (ECOG) performance status (PS) 0 or 1 * Life expectancy \> 16 weeks * Capable of understanding investigational nature, potential risks and benefits of the study and able to provide written informed consent * Absolute neutrophil count (ANC) \>= 1,000/uL * Platelet count \>= 75,000/uL * Total bilirubin =\< 1.5 x institutional upper limit of normal (ULN) * Aspartate aminotransferase (AST) (serum glutamic oxaloacetic transaminase \[SGOT\]) and alanine aminotransferase (ALT) (serum glutamate pyruvate transaminase \[SGPT\]) =\< 2.5 x institutional ULN (\< 5 x institutional ULN if liver function test \[LFT\] elevations due to liver metastases) * Creatinine =\< 1.5 x institutional ULN * Women with human epidermal growth factor (HER)-2 positive disease must have received and progressed on at least one prior anti-HER-2 directed regimen (trastuzumab, lapatinib) for their metastatic disease * For dose escalation cohort: * Any number of prior systematic therapy regimens is allowed * NOTE: prior systematic therapy in the adjuvant setting is n…