Less global iv-counsel

I would like to use iv-counsel only with certain keybindings, like accessing my org-roam files and when invoking M-x. However, it seems to take over everything. I especially don't want it for invoking find-file. Can someone maybe help me with a hook that could achieve this?

> I especially don't want it for invoking find-file why?

Not activate ivy-mode and assign key bindings to counsel functions instead? E.g. M-x to counsel-m-x (or what it is called …)