The answer seems to be that there looks to be a positive but small benefit to holding the bonds, though there is high variance in the outcomes. Lindert and Morton (1989) look at 1,522 bonds over the course of 150 years and find a 0.42% premium of their portfolio against a portfolio of British and U.S. bonds. Eichengreen and Portes find that U.S. bonds beat default-prone sovereign bonds, but default-prone sovereign bonds bean U.K. bonds during the 1930's round of defaults. Klingen, Weder and Zettelmeyer (2004) estimate long run premia of between -0.17% and 0.46%, using various methodologies.
Below, from the book Debt Defaults and Lessons from a Decade of Crises (Sturzenegger and Zettelmeyer), Corrections depicts the degree to which a portfolio of sovereign bonds beats a portfolio of "safe" (U.S. or U.K.) bonds over long time periods for 32 country-time periods (click to enlarge).