If your mdbook is using mathjax v2.x (which is the case when downloading mac binary), you can override its font and expression to trigger mathjax.

In book.toml, add the followings

[output.html]
mathjax-support = true
additional-js = ["mathjax-config.js"]

And in mathjax-config.js (which should be at the top of your project directory and not under src):

MathJax.Hub.Config({
    "HTML-CSS": { scale: 90, availableFonts: ["TeX"] },
    tex2jax: {
        inlineMath: [
            ["$", "$"],
            ["\\(", "\\)"],
        ],
    },
});

Change ‘scale’ to adjust font size. Also, see the $…$ in inlineMath. Now you can inline using $equation$ which is much easier than \\(…\\)