In 1964, Ivory Mae marries Simon Broom in the backyard of 4803 Wilson Avenue in New Orleans East. This is not the Yellow House, but it is nearby. Simon Broom works in maintenance at NASA. Having already given birth to Carl, her first child with Simon, Ivory Mae is now pregnant with her fifth child Karen. When Simon and Ivory Mae met, he was also married with children. Simon’s daughters Deborah and Valeria move in with Simon and Ivory Mae after Simon’s ex-wife Carrie dies of leukemia in 1963. Deborah is ten and Valeria is eight. 19 years her senior, Simon is tall, dark-skinned, and handsome with an easy, self-possessed manner. He speaks well and dresses well. A hard worker, Simon fought in World War II in the Asiatic-Pacific Theatre. Ivory Mae is attracted to his experience and his age. 

In March 1961, an advertisement for a home at 4121 Wilson Avenue runs in the Times-Picayune newspaper. This is the Yellow House. 1961 is three years before Simon and Ivory Mae merge their families. At this time, New Orleans East is primarily a cypress swamp. The ground is soft, and the area is home to nutria and muskrat. The 40,000-acre tract of swampland goes by many names: Gentilly East, Gentilly, Chantilly, Orangedale, Citrus, Pines Village, Little Woods, Plum Orchard, east of the Industrial Canal, Orleans East, or eastern New Orleans.