The story of Washington parallels the history of the Cotton Mission. There was more cotton acreage planted in Washington than the other settlements. The settlers struggled with nature: floods washed out their dams built on quicksand bottoms year after year; and they suffered from chills and fever (malaria spread by the mosquitoes which hatched in the springs and along the edges of creeks), drought, grasshoppers and animals at night. They also soon realized that man does not live by cotton alone. They learned they would have to grow their own food and supply their own goods to survive. Many quit the mission; by June 1861 only twenty families remained in Washington. More calls were made by the church leaders in Salt Lake City to serve in the mission, and by 1964 there were 85 families with 413 people.

The colonists were beset by poverty from the beginning. They succeeded in growing cotton, but being far removed from the market, they could not dispose of their lint. They tried freighting to California and to Omaha, Nebraska. They sold some to Brigham Young in Salt Lake City. These measures were temporary, however, and Brigham Young determined that a factory was needed closer to the produce. He had a factory dismantled in Salt Lake City and shipped to Washington where the supply of water was dependable year round. The colonists provided the labor and material and completed the one-story factory in 1866. They began operations January 1867 and added two more stories in 1868. Labor at that time was two dollars a day. Most employees received pay in factory goods, produce and factory scrip. A store was operated in conjunction with the factory, which served as a clearing house for most of the products of the area.