Java's best claim to fame is that it is a write once, run everywhere programming language.
And it is *very* good at that.
But that comes with a cost: It means having to cater to the lowest common denominator. Getting a run time engine like Java to do exactly the same thing on every platform that Java runs on, *AND* at the same time, making it work and feel like a native application, not some half-arsed Java app is an extremely difficult task to pull off.
Java has gotten much better at this over the last 10 years. It used to be that if you ran a Java app that displayed a File Open dialog box, that dialog box was so *radically* different (and far worser) than every other File Open dialog box that native applications used, that you would just say...YUCK...JAVA APPS SUCK. (They did). Writing an OS agnostic GUI is hard...really, really friggin hard.
One area where Java fares well is middleware. Code that has no UI. Code that might do things like shuffle data between point A and point B (A might be a TCP/IP socket or a web service. B might be a file or a database). Java shines here, because OS differences can be abstracted fairly easily. Sockets are sockets. Databases are databases. People don't ***** if things are a different shade of gray or the tab key doesn't work the same.
Writing an OS in Java would be tough. Java does OK at abstracting the differences between different OS'es, but that doesn't mean it can do well at *BEING* the OS. It wasn't designed to do that.