Three Union gunboats operated from its wharves on patrol and blockade duty, joined by a constant stream of transport ships and supply vessels.
Sutlers' stores sold all kinds of wares to the soldiers inside the fort, while outside their sprawling tent cities and the town's well-armed earthen walls the landscape was covered with the shanties of thousands of contraband slaves.
Quaker missionaries came at Wistar's invitation beginning in July 1863, building and operating the schools that taught the runaway blacks to read and write by the hundreds.
So bustling a place did the fort and harbor become that the enterprising troops opened their own photography studio. They also used confiscated presses from Williamsburg to publish "The Cavalier" newspaper.
"Wistar was the one who really beefed things up," Colonial National Historical Park historian Diane Depew says, describing how the energetic general transformed his command.
"Every building in the town was being used by the army — and outside the fort he ordered the construction of hundreds of contraband cabins in Slabtown."
By some accounts the number of refugee slaves grew to as many as 10,000 during the war, depriving the Confederacy of one of its chief assets.
Some fled from as far as away as Richmond, dodging Southern troops and slave hunting parties in a desperate attempt to reach the historic Revolutionary War battle ground that many identified with freedom.
That symbolic association only grew with the October 1863 arrival of the 4th and 6th United States Colored Troops infantry regiments, who were among the first black soldiers seen in Virginia.
Not only did they serve as models of resistance for the incoming contrabands but several men from the 4th also went on to win the Medal of Honor.
"By the summer of 1863, the Union had some real obligations to the huge number of contrabands gathered here — and this becomes an important recruiting area," Depew says.
"The competition between recruiting officers became so great that they started to fight over who had a right to be here."
Erickson can be reached at 757-247-4783. Find more stories on Hampton Roads history at dailypress.com/history and Facebook.com/hrhistory.
Want to go?
Fort Yorktown Sesquicentennial Weekend
What: Living history programs featuring nearly 100 re-enactors presenting Union infantry and cavalry encampments and drills, a Civil War hospital, courts martial, mail calls, refugee slaves and Quaker missionary work.
Where: Along Main and Nelson streets in the Yorktown historic area
When: 10 a.m.-4:30 p.m. Saturday and Sunday
Information: 757-898-2410 or http://www.nps.gov/colo