Virtual Choreographer is a free and open source real time 3D engine that relies on an XML-based definition of 3D scenes with graphic and sonic components.
Virtual Choreographer is well-documented and offers facilities such as high dimension particle systems through GPGPU, MPEG4 facial and skeletal character animation, audio source localization, collision detection and vertex and pixel shader support.
Installation instructions for Mac OS X can be found here.