Rafford Hall was built as the Pensacola Athletic Club in 1889 on the southwest corner of Baylen and Belmont Streets as the Pensacola Athletic Club. The club was the endeavor of a group of businessmen, led by Louis P. Knowles, whose financial success (and the accompanying combination of rich food and a sedentary lifestyle) had given them corpulent physiques. William D. Chipley, who lived nearby and reportedly favored the idea, sold the land for the club at a reduced price. Membership to the club cost $10 for initiation and $3 per month, and by 1890 there were a hundred members.
The club remained popular until the Pensacola YMCA, with professional trainers and more advanced facilities, opened nearby on Palafox Street in 1904. The Athletic Club closed, and building was sold to the Independent Order of Odd Fellows (IOOF) on January 1, 1906,[date uncertain] renovated over several months and named after benefactor Richard Rafford. The Athletic Club building replaced the IOOF's previous meeting hall on West Garden Street (also named after Rafford) built a few years earlier, which was sold for "a handsome sum" to the Pensacola Investment Company. It was used for decades as a meeting hall and exercise center for the IOOF and other fraternal benevolent organizations, including the Knights of Pythias, the Woodmen of the World, Royal Neighbors of America, the Brotherhood of Railway Trainmen, the International Association of Machinists and the Communications Workers of America.
The building was acquired by the North Hill Preservation Society in September 1974 and restored by Theophalis May. Its construction, which inspired the L&N Marine Terminal Building, was two-and-a-half stories of wood frame, plus a wraparound balcony on the second floor, over a brick basement. Its external dimensions were 85'4" by 41' with a 40 by 60 foot meeting room and gymnasium. The 230 foot running track, constructed of wood, was suspended from the second floor and encircled the gymnasium's inner perimeter.
The following plans and elevations were published by the Historic American Buildings Survey.