以下は 11/9 時点の古い内容。 11/12 の homebrew の更新で、インストールの問題は解決されたようです。

$ brew install mplayer
Warning: It appears you have MacPorts or Fink installed.
Software installed with other package managers causes known problems for
Homebrew. If a formula fails to build, uninstall MacPorts/Fink and try again.
==> Downloading http://www.mplayerhq.hu/MPlayer/releases/MPlayer-1.1.tar.xz
Already downloaded: /Library/Caches/Homebrew/mplayer-1.1.tar.xz
==> ./configure --prefix=/usr/local/Cellar/mplayer/1.1 --cc=clang --host-cc=clan
==> make
                glBegin(GL_LINES);
                        ^
9 errors generated.
make: *** [libvo/vo_corevideo.o] Error 1
make: *** Waiting for unfinished jobs....

READ THIS: https://github.com/mxcl/homebrew/wiki/troubleshooting

These open issues may also help:
    https://github.com/mxcl/homebrew/pull/23809
    https://github.com/mxcl/homebrew/issues/23503

こんなエラーが出た。最後に関係ありそうな pull request と issue のリンクが表示されていて親切。

で、色々試した結果、これだけでいいみたい。

$ wget -O /usr/local/Library/Formula/mplayer.rb https://raw.github.com/i8degrees/homebrew/d5687ef38dfe5bcaabf181d6bc8eedfbe85b3be1/Library/Formula/mplayer.rb
$ brew install mplayer

参考URL