One thing it made me wonder was whether the English expression refers to ancient or modern Greek. I guess I've always assumed we were all talking about ancient Greek, which is historically a language required of the highly educated, and of course also a very difficult one to learn. But now looking at this chart I'm not so sure -- there are indeed some examples of academic, historical languages filling this role, but there are also a whole bunch of just plain foreign languages in there.

UPDATE: A commenter on Language Log remembers that "It's Greek to me" appears in Julius Caesar, but it looks like that probably isn't the original source.