Bring 2 quarts of water to boil over medium-high heat in a large pot. Add in the tea bags, rosemary, orange, and vanilla bean. Cover and let infuse for 10 minutes. Remove tea bags and bring again to a boil, stir in honey, place in peaches, and boil, stirring occasionally, 10 to 15 minutes.
Remove peaches and let cool slightly. Meanwhile, remove all solids from the poaching liquid, and simmer liquid until it reduces by at least half, about 20 minutes.
When peaches are cool to the touch and easy to handle, gentle slip the skins off and discard.
To serve, place a skinned poached peach in a bowl and ladle reduced liquid over peach. Serve with creme fraiche and pistachios, if desired.