Grate the zest of one orange and use a vegetable peeler to peel big strips of orange peel off the second. Juice both oranges.
In a 9x13 baking dish, stir together the olives, olive oil, 2 sprigs of rosemary (and thyme if using), the larger pieces of orange zest you removed with a peeler, orange juice, garlic and one pinch of crushed red peppers.
Bake about 30 minutes, stirring and tossing the olives every ten minutes.
Remove the olives from the oven and add the fresh rosemary leaves off the third sprig. Sprinkle with the finely grated orange zest, (fennel powder if using) and another pinch of red pepper flakes. Toss and serve.