PS: Who picks the Prime Minister?

I am listening to Tony Abbott, on the radio, repeating the mantra - 'Let the Australian people elect the Prime Minister not the faceless men in the Labor Party'- not quite a direct quote but nearly.
Last time I looked at a book on politics it clearly states, the people elect the party and the Party elects the Leader to become Prime Minister.
We don't yet have a directly elected Prime Minister!
Mis-information and populist quoting again!