Outgoing calling via Hangouts Dialer app uses 3G/4G LTE/Wi-Fi data, and utilizes high-quality voice technology, depending on your connection (it will sound better on LTE vs. 3G.) It's very similar to how other VoIP solutions work (Viber, Line, Skype, etc.) with the added bonus of being able to call domestic landlines and cellular providers at no extra cost, aside from higher data usage of course.
Calling via the built-in Phone app on the other hand uses regular voice service, where your calls are routed over the cellular voice network of whichever provider Project Fi is currently connected to (T-Mobile, Sprint, or US Cellular currently.) The "HD Voice" feature is not available on Project Fi yet, and therefore the voice quality will be comparable to that of regular telephone service.
Unfortunately there is no way to use the Hangouts app for incoming voice calls for accounts that are activated on Project Fi. It's only possible for regular Google Voice accounts by enabling the "Incoming phone calls" option in Hangouts -> Settings -> your GV account. This is a known limitation of Project Fi.