Testing Olaparib Either Alone or in Combination With Atezolizumab in BRCA Mutant Non-HER2-positive Breast Cancer

Inclusion Criteria: * Patients must have histologically documented unresectable locally advanced or metastatic non-HER2-positive breast cancer and a known BRCA 1/2 mutation present; both germline and somatic mutations are acceptable, however somatic mutations must be identified by either tumor sequencing of tumor tissue or ctDNA in plasma; patients with BRCA mutations of unknown significance are not allowed * Patients must have measurable disease, defined as at least one lesion that can be accurately measured in at least one dimension (longest diameter to be recorded for non-nodal lesions and short axis for nodal lesions) as \>= 20 mm (\>= 2 cm) by chest x-ray or as \>= 10 mm (\>= 1 cm) with CT scan, MRI, or calipers by clinical exam * Prior chemotherapy is allowed, including platinum therapy; patients must not have received chemotherapy for 4 weeks prior to the initiation of study treatment and must have recovery =\< grade 1 from any adverse events from any prior chemotherapy (other than alopecia); patients must not have had nitrosoureas or mitomycin C for 6 weeks prior to the initiation of study treatment * Prior radiation therapy is allowed; patients must not have received minimal radiation therapy (=\< 5% of their total marrow volume) within 3 weeks prior to the initiation of study treatment; otherwise, patients must not have received radiation therapy (\> 5% of their total marrow volume) within 4 weeks prior to the initiation of study treatment; patients who have receiv…