Decide on what you want. Understand that buying a horse is a big investment. Good quality ponies usually start at a $1,000 price range and can go higher.
There are also various types and breeds of horses for different purposes. You must consider your purpose. Do you need a horse for competitions, for work, or for leisure?