Most languages start as a form of another language. For example, Java is considered C-like, HTML inherits from SGML, and so forth. So, first you want to explore what the basic form of your language is.
Once you have determined the basic form, you need to codify its grammar. The standard way of doing this is to document it in Backus-Naur form (or simply BNF). BNF is a language that describes grammars and the "official definition" of a programming language is nearly always described in BNF.
http://en.wikipedia.org/wiki/Backus-Naur_form
Once you have your language defined in BNF, you need to create a parser for your language.
What's nice about having BNF is that there are tools to generate a parser for you from your BNF. One such example of this, used for compiled programming languages like C, is a program called YACC (Yet Another Compiler Compiler)
http://en.wikipedia.org/wiki/Parser_generator
http://en.wikipedia.org/wiki/Yacc
Once your parser is complete, you need to turn the source code of your language into "tokens" for your parser to chew on. This is where a lexical analyzer comes into play.
http://en.wikipedia.org/wiki/Lexical_analyzer
Just like YACC, there's a number of tools available to assist with building these. The classic tool for this is called "lex", but there's more current variants such as Flex.
http://en.wikipedia.org/wiki/Flex_lexical_analyser
Granted, what you do with the output of these programs depends on what kind of language you are writing. You'd handle the output for an interpreted language much differently than a compiled language.
If your language is going to be compiled you'll need to write an assembler and linker for these outputs.
http://en.wikipedia.org/wiki/Assembly_language
http://en.wikipedia.org/wiki/Linker
If your language is going to be interpreted, you'll need to write an interpretive runtime.
http://en.wikipedia.org/wiki/Interpreter_(computing)