The VNT turbos on the 6.0L Fords are notorious for failing. There are several reasons they can go out. More than likely when your EGR cracked the last time you ran some coolant through the turbo, this can heavily score the bearing because the coolant takes all the lubricity out of the oil. Once the bearings in the turbo are scored it is only a matter of time before it dies completely.
The other thing that may be causing the oil to come out of the Down Pipe is the crank case breather design on those trucks. So for example on a Cummins the crank case breather is a tube running out of the top of the valve cover and down the side of the block venting to atmosphere. On the Powerstrokes and the Duramaxes the crank case vent is plumbed into the intake manifold, the problem with this is it puts the vent under constant vacuum and over time you suck oil from the heads into the intake filling the intercooler pipes, intercooler, and eventually the turbo with oil. The turbo should never have oil in the housings, only in the center section. So the oil begines building up on the turbine fins and can cause clipping of the fins on the turbo housing eventually destroying the turbo completely.
I have replaced more stock turbos than I care to remember on those trucks. My personal advice would be to delete the EGR completely and do a crank case reroute which will vent the crank case to a catch can instead of your intake. Here are a few helpful links.
BD BDD1032175 6.0L PowerStroke Crank Case Filter Kit
Sinister MFG. EGR Delete Kit for 03-07 6.0L Ford Power Stroke
Best of luck and I hate to hear that you are going through this. But you are not the only one to have these problems. But there is a fix.