Born in England, Philip Ludwell was the son of Thomas and Jane Cottington Ludwell. He immigrated to the colony of Virginia in about 1660. As with many colonists, Ludwell was eager to rise socially and financially. He married a woman named Lucy Higginson Burwell Bernard in c. 1667. She had been twice-widowed since arriving in Virginia. This, too, was typical of life in seventeenth-century Virginia.

Life was hard, and lifespans were frequently shortened by disease, change in environment, Indian attack, and in the case of women, death in childbirth. The couple had two children together before Mrs. Burwell died in 1675. Five years later Ludwell made a most advantageous marriage by wedding the widow of Albemarle Governor Samuel Stephens and Virginia Governor William Berkeley. Her own name was Lady Frances Culpepper Stephens Berkeley, and even after her marriage to Ludwell she maintained her title of Lady Berkeley.

Ludwell moved into Green Spring plantation, the home Lady Berkeley and the governor had shared. The couple had no children. Ludwell's new, powerful connections enabled him to continue his colonial rise at a much swifter pace. He soon acquired additional plantations and became involved with Virginia politics. He was a member of Governor Berkeley's council and served briefly as secretary of state. He was involved in Bacon's Rebellion, siding loyally with Berkeley.

The fact that he was such a firm supporter of Berkeley is likely the reason Lady Berkeley agreed to marry him later. As did most wealthy planters, Ludwell spend time in London, serving as his own agent for tobacco and other products from his plantations that were exported to England. While he was in London during 1689 Ludwell was commissioned as governor of the northern portion of the Carolina province. Upon his return to Virginia he went to North Carolina and established a government. However, both he and Lady Berkeley generally continued to reside in Virginia.

Disagreements between those who had established a government there earlier created difficulties for Ludwell. Known for his very hot temper, such stresses did not encourage him to continue in Virginia politics, despite his reappointment as governor of Carolina. Once more, Ludwell followed the pattern of a number of wealthy Virginia colonials and decided to return to live in England. He retired there in 1700 and died around 1723. He was buried in a family vault in Middlesex, England.