Ivy Baker Priest was born September 7, 1905, in Kimberly, Piute County, to Clara Fernley and Orange D. Baker. Her father worked as a gold miner near Kimberly, but when the gold vein played out he moved his young family to Coalville. Following a boiler explosion there in 1912 the Baker family again moved, this time settling at Bingham Canyon where Orange obtained work in the copper mine.

Life in Bingham proved challenging. Ivy's father seemed strangely accident prone and was frequently unable to work due to injury. In an effort to meet the family's financial needs Ivy's mother opened a boarding house for miners, which boosted the family income but also brought 20 to 30 miners tramping through the Baker house each day. The mess their muddy shoes created led, in a roundabout way, to Ivy's lifelong, influential political career.