As it turns out, one should replace the knitted seal ring that sits between turbo and exhaust pipe when replacing one or the other as it will not seal properly. I have not done so when turbo was still on the table and thus had some quality time removing the old ring in situ. Had to hurt it a lot before it agreed to part with turbo. Below are a couple of pictures comparing new and old.
Exhaust knitted seal ring
The black stuff on the right is actually graphite or other soft and heat resistant material fill. It is covered with a layer of knitted stainless steel wire. All this is necessary to ensure a tight seal between motor and tailpipe as there is no way to connect these two without a flexible joint. You can read more about the problem and possible solutions in this patent.