Check out the paper Model Explanations with Differential Privacy - N. Patel, R. Shokri, and Y. Zick. 2022 Check out the slides PDF here.