Even after caching sometimes when you run an audit in Google Lighthouse, you will basically encounter fonts not visible issues. This is because some browsers wait for the fonts to get loaded and don't show texts in the browser until they load the font.

The major impact for SEO is when user may come from www or non-www link of your website. And if your web server is not capable to handle those and start serving on the same requested links, then your SEO score is totally divided between your 2 domains links.