The idea behind the army's formation was to put more than thirty thousand troops serving on garrison duty to good use by attacking Richmond and bringing an end to the war ahead of the 1864 congressional and presidential elections. (Republicans in the North were concerned that, absent some military success, U.S. president Abraham Lincoln would lose the November election to his Democratic challenger, the former Army of the Potomac commander George B. McClellan.) The Union Tenth and Eighteenth corps were assigned to Butler and his Department of Virginia and North Carolina, which had been created the previous July. The Tenth Corps included the famed 54th Massachusetts, the regiment of black troops that had led the assault on Fort Wagner, South Carolina, in July 1863. In December 1864, the two corps' white troops were merged into the Twenty-fourth Corps, their black troops into the Twenty-fifth Corps.

The army's first mission was to act as the southern pincer in Grant's Overland Campaign against Richmond. While the Army of the Potomac—commanded by Meade but under Grant's personal supervision—advanced on the capital from the north, the Army of the James would advance from the south, up the James River. Most importantly, Butler was charged with cutting Confederate general Robert E. Lee's supply line, the Richmond and Petersburg Railroad, forcing Lee to move troops to the south and thereby weakening his defense against Grant and Meade. Despite high hopes for the plan's success, by June 16, 1864, Confederate troops under Pierre G. T. Beauregard and George E. Pickett managed to pen up the superior Union force on the Bermuda Hundred Peninsula, near the confluence of the James and Appomattox rivers. This led some, including Grant, to quip that Butler's situation was similar to being inside a tightly corked bottle.

Politicians larded the ranks of the Army of the James, and Butler was especially notorious. A droopy-eyed, slightly overweight Democrat who had served in the Massachusetts legislature before the war, Butler briefly had supported Jefferson Davis for the Democratic presidential nomination in 1860. During the war, however, he was anything but sympathetic to the Confederacy. In particular, he angered Southerners by confiscating their runaway slaves and, in 1862, charging with prostitution any New Orleans, Louisiana, woman who spoke ill of the Union. By 1864, Butler was known as a "beast" in the South, and his lack of military training contributed to his unsuccessful attempts, during the Petersburg Campaign, to capture Petersburg despite outnumbering the Confederates. Still, Butler was not entirely to blame. Historians also have found fault with the often vague orders that he received from Grant.

Despite its failures on the battlefield, the Army of the James became a technological testing ground. For instance, Butler tested at least one early model of the Gatling gun. (Patented in 1862 by North Carolina native Dr. Richard J. Gatling, the Gatling gun was a continuously firing weapon that resembled a modern machine gun.) Beyond championing technology, though, Butler used his command to embrace social innovation, employing large numbers of U.S. Colored Troops. Unlike many other Union officers, Butler harbored little prejudice toward African American soldiers and insisted that they be treated equally.

When Lee evacuated Petersburg on April 2, 1865, elements of the Army of the James pursued the fleeing Confederate army, while others, including the black troops of the Twenty-fifth Corps, proceeded north and were among the first Union soldiers to enter Richmond on the morning of April 3. The Twenty-fourth Corps, meanwhile, is said to have fired the final volley of the Appomattox Campaign, which ended in Lee's surrender of the Army of Northern Virginia on April 9.

Following the war, units from the Army of the James would serve as an occupying force in Virginia, with the last troops mustered out in February 1866. The Twenty-fifth Corps served on border duty in Texas that was designed to dissuade the French from meddling in Mexico. It mustered out of service in January 1866.