Configuration weaknesses in VoIP devices and underlying operating systems can enable denial of service attacks, eavesdropping, voice alteration (hijacking) and toll fraud (theft of service), all of which can result in the loss of privacy and integrity. To perform well in VoIP environments, security appliances must both protect the VoIP infrastructure and maintain the voice quality, availability and reliability of the connection.
Establishing a secure VoIP and data network is a complex process that requires greater effort than that required for data-only networks.
VoIP systems can be expected to be more vulnerable than conventional telephone systems, in part because they are tied into the data network, resulting in additional security weaknesses and avenues of attack. Confidentiality and privacy may be at greater risk in VoIP systems unless strong controls are implemented and maintained.
Use strong authentication and access controls on the voice gateway system. Since some VoIP telephones are not powerful enough to perform encryption, placing this burden at a central point ensures all VoIP traffic emanating from the enterprise network will be encrypted. Financial institutions should enable, use and routinely test the security features included in VoIP systems.
http://www.bankinfosecurity.com/articles.php?art_id=207
Security Considerations for Voice Over IP Systems – NIST Special Publication 800-58