After trying it out I see that youtube-dl will actually do the extraction
to mp3 for you (provided that ffmpeg and probably lame are installed):

youtube-dl --audio-format=mp3 --extract-audio http://youtube.com/video