According to March 20, 1941 correspondence from the Chicago, Milwaukee, St. Paul & Pacific Railroad Company, the line from Albert Lea to St. Clair was built in 1905 and 1906 as the Duluth, St. Cloud, Glencoe & Mankato Railway Company. It was built by the Milwaukee Railroad Company construction crew and nicknamed “The Alphabet Road.” For a time it was operated under the St. Paul company as a branch line, but under the name of the original company. The line was taken over by the St. Paul company in 1910. The first tracks reached Plum Valley in the latter part of 1906, and the first regular scheduled train arrived in 1907. A depot was also built in 1907.
There had been much controversy over the name of Plum Valley from 1905 to 1908, some residents feeling it should be renamed “Hodgkins” after Samuel Hodgkins. The name was changed to Hodgkins by the railroad when they came in 1907. Soon after, in October of 1907, the Federal Post Office came. The government took matters into their own hands and placed a plaque on the depot calling the village by its present name of “Waldorf.” Another story is that the daughter of E.L. Tobie, the railroad President, named the town after the Waldorf Astoria in New York City. And yet another story is that E.L. Tobie himself selected names for three new townsites on this “Alphabet” railroad line, Matawan, Waldorf and Pemberton. Other stories were told, as well. No one seems to know which story is true, but the town was officially dedicated as Waldorf on July 9, 1908. In 1908 the businesses in Plum Valley (except the Plum Valley Creamery) moved their buildings a very short distance south to be closer to the new Depot.
The Chicago, Milwaukee, St. Paul & Pacific railroad from Albert Lea to St. Clair was abandoned in 1978. In 1978 the tracks and ties were removed and the railroad company’s land parcels were sold individually. The Depot was taken down in approximately 1985.