GUI = Graphical User Interface as opposed to old-fashion CLI = Command Line Interface
All Windows-programs (and X programs under Unix/Linux) are GUI-programs. GUI-programs are programs where you use a pointer (ie. the mouse) to give commands to the program, by clicking buttons, menus, check-boxes and drop-down lists and so on.
This is different from a "terminal" (CLI) program, where you typically gives orders to the program either at start-up or by typing keys at the keyboard. It may be you get a menu and type in a number or letter based on that menu (which may bring forth another menu of choices), or it may be you write more complex commands that the program parses.
Java is a programming-language, so it's neither. You may write both GUI and CLI programs in Java. Odds are the the editor or IDE you write your programs in are GUI. (The actual Java-compiler is CLI, as it just takes options and arguments on the command-line and goes to work without more interacting.)
In Java you can write both simple CLI-programs that are run in the command-prompt (you'll probably do this for your first simple programs)... but you can also write fully featured GUI-programs, as Java includes widgets (thats buttons, check-boxes, radio-buttons and so on) you can use in your programs.
A huge difference between a CLI and GUI is what it does when it waits for a user command. CLI-programs will usually just stop and wait -- pause -- until the user type a key or press enter.
GUI-programs on the other hand, usually runs an "idle loop". During this loop, it constantly checks if an event -- like a key being pressed, the mouse being moved or the mouse being clicked -- has occured. If it has, it checks if it applies to it (the program), and if it does, it calls a program corresponding to the that particular event for that particular widget (e.g. ButtonStart was clicked or CheckBoxRandom was checked).
Because of the difference between just waiting (CLI) and having to constantly check if something has happened (GUI), GUI-programs uses more resources even when they appear to do nothing. On top of that, GUI-programs must redraw -- and occationally reporocess -- parts of themselves, if things happens, like another window being moved over it.