The phrase “everything should work as fast as possible” describes the wrong way for developing software. Software is a means to an aim. The aim is a functionality of a sort, just like a car or a house. When constructing a solution, one has to take performance in the context of the solution and not as an ideology.
Imagine how a house would look like if the drive way was paved four lanes wide just for the sake of performance.
The prioritization of performance is determined by the business requirements. A thorough analysis of the required throughput in each of the application flows reveals by itself the highways and the sideways.
Marking the road map in such a way is not premature optimization, but the proper use of the information that the architect has from day one. The final tuning can be done at the later stages when performance issues come up. However, developers need to know how wide the road they are paving should be.
See also “Data Buckets”.