Yes, there is quite a significant impact on first-load performance. The browser has to pull down all of the linked scripts before it can display any content—if you’re using a library like jquery that’s a sizeable chuck of code that has to be loaded and executed just on its own.
Modern browsers may be screamingly fast at executing JS, but modern devices aren’t: mobile phones are significantly slower than desktop devices for this kind of thing.
There’s a trade-off in the other direction as well of course—if you are loading subsequent content via ajax performance on further clicks will be better than if you were doing full page refreshes.
To summarise: yes it absolutely affects performance, but that doesn’t mean you should avoid it.