Sheppard was born in Owings Mills, Maryland. He attended the Maryland Institute (now Maryland Institute, College of Art, known as MICA), from 1948 to 1952, where he studied with Jacques Maroger. From 1956 to 1957, he was artist-in-residence at Dickinson College, Carlisle, Pennsylvania. In 1957, Sheppard was awarded a Guggenheim Fellowship to study in Florence, Italy. From 1960 to 1975 he taught at MICA.
In addition to painted portraits, Sheppard has been commissioned to do full-length bronzes, murals, and other public works. Most notably, in Baltimore alone, he is responsible for the bronze of former major league baseball player Brooks Robinson (2011); the Pope John Paul II Monument (2008) at the Basilica of the National Shrine of the Assumption of the Blessed Virgin Mary; and the Holocaust Memorial sculpture (1988). Other portraits include George H.W. Bush, U.S. Senator Barbara Mikulski, and Archbishop Edwin F. O’Brien.
In 2011, the University of Maryland University College, opened its Leroy Merritt Center for the Art of Joseph Sheppard. The center houses a semi-permanent installation of Sheppard’s sculpture, his art history library, and a selection of drawings.