Off throttle surge isn't an instant turbo killer. You'll find plenty of tuners who intentionally tune adjustable BOVs to not open at lower vacuum levels (so it doesn't open when shifting after a low boost level) in order to improve throttle response. Surge like you hear when you're actually on the throttle is HORRIBLE and usually very damaging as both the compressor and exducer are both under a serious load and then start getting pummeled with severe oscillations axially and radially, thrust washers decay as the oil film is compromised and the same with the bearings. On the other hand a BOV which isn't closing all the way can over work the turbo and cause excessive wear if it's not caught but it's usually not an overnight kind of thing.
The back firing doesn't tell me much other than your fueling may need adjusted, you're air metering system isn't setup properly or there's a significant timing problem.
It is possible for a bad BOV to cause damage but it's unlikely unless there were other critical issues with the setup. Plenty of cars came without them ever installed, manufacturers still make some without them and a lot of race vehicles don't use them. I haven't run a BOV on my MR-2 in years, pushing 16psi on the stock turbo... It's still not showing much more wear than when I bought it. I don't recommend everyone start running out and removing theirs but doing so is seldom a car killer.
A defective wastegate or inadequate discharge aperture will often be evident as your boost creeps beyond the set boost level while you're on the throttle. That's if the system isn't able to expel the excess exhaust charge. This isn't necessarily going to kill the turbo unless it really gets up there but may cause significant engine damage if your system isn't setup for the kind of load and fueling requirements of said boost. Of course if you don't notice as your boost climbs up to and beyond 25psi then the problem is even more with the driver than the car.
If the compressor is working too hard because of a poorly matched compressor and exducer you'll get the really nasty kind of compressor surge and are very likely to cause significant damage throughout the system. There are a couple other causes for this problem but I'm not going to get into all of it. The quickest fix is to turn the boost down, in all but the most extreme cases this will alleviate the problem. In really bad cases you may even see the intake charge meeting or exceeding the ambient air pressure (ie. it's making boost) at low RPMs or even at idle...
The BOV has nothing to do with this situation. Sure the BOV may be responsible for an off throttle flutter but the WOT surging isn't related and that's the one that's a real killer.
Without seeing the turbo and dismantling it I can't offer more than ignorant guesses as to what your problem was.