JPEGs are your "normal" type of file format for anything, but mainly photos. But should only be used under certain circumstances. They can be huge files, with high quality for printing, or very small files with low quality for the web. They are better when used with images that have lots of different colored pixels, like a photo. It shouldn't be used for a small logo or basic shapes with color (use a gif). They can't be edited after saving to that filetype, especially if you need to change a word, which is why you want to keep the original handy (Photoshop file, illustrator file, etc.) Jpegs can't support transparency, meaning, you can't see through to whats behind it. The shape of an jpeg can only and always be a square. If your jpeg is a red circle, than the rest of it will be white to fill in the square (or rectangle).
PNG - is almost exactly like a jpeg, same specifications, except it will support transparency if you tell it too. This includes opacity (seeing through it if you want it semi-transparent), or an odd shape. The png will still be a square shaped file, but if you have a circle, you have the option of making the edges of it clear. PNGs aren't used as often as other formats, but they should.
TIFF - mainly exists only for print and friendly transfer of files from one systems to another. The cool thing about TIFFs is that they can keep certain elements applied to it from photoshop: Transparency, type styles, layers... If you have a photoshop file that you know you need to see on a computer that doesn't have photoshop, but you don't want to lose quality by saving it as a Jpeg, use a TIFF. If you use the correct settings, you can open the TIFF file in Photoshop and edit it as if it were a PSD (photoshop document). Very useful if you are sending a photoshop layout to a printer and you aren't sure if they have photoshop. If you send a TIFF, they'll be able to open it to print it.
GIFs - are only used for small web images like avatars, logos, or thumbnails. They have a tremendous loss of quality which is why you can see the pixel dots in the images. GIFs are best used for small files that aren't complex. If it's a logo, an image with only a few colors like a computer drawing, or a cartoon, use GIF. If its more complex or you need a better quality, larger sized file, use JPEG. A few cool things about GIFs is they can support transparency like PNGs, but aren't nearly as huge of files, but again, huge loss of quality there. They are also the ONLY kind of file format of its kind (except for SWF in some circumstances), that are able to be animated. The animated images you see people posting on your myspace are animated GIFs. Again, since GIFs are very low quality, the animations are never that smooth.
So:
JPEGs: good for more complex images, small or large
PNG: same as JPEGs but can have transparent items
TIFF: used for print or if you want to see a photoshop layout on other computers
GIF: small files used best for small images. Can be animated and transparent. Don't use it for complex images.