There are several ways. I'll describe the two most typical.
1) fuzzing the source code.
It is easy to automate this process with open source software where source code is widely available. Let's use an easy example: We can use the language of our choice to scan a PHP script's source code and look for fileupload() that does no checking on what type of file was uploaded. If our fuzzer finds no checking of filetype, it is probably vulnerable to a fileupload attack to execute remote code as the process ID on the web-server running PHP.
Next lets look for data input with no bounds checking and see if we can input more data into a buffer than is expected. Well, if it can we might have a buffer overflow that will allow remote code execution.
2) fuzzing a running service.
This involves sending random data to a port and see what is returned. For example if append a single quote to the query string of a web application, and get an SQL error back, we know it is vulnerable to SQL injection. From there it is just a matter of manipulating the SQL query in a way that will send it from the web-server to the SQL server without any error.
The same can be done for buffer overflows, integer overflows. and pretty much everything.....
To dipshit above, not it is not. You have no clue of the steam you're blowing out your rear! Fuzzing is how UNKNOWN EXPLOITS are found.
What should you learn:
1) programming. I'd start with PERL or Python. Both are widely adopted in the security community and easy to learn. Metasploit framework is designed for Python.
2) how clients communicate with daemon processes
3) How the memory stack works in different operating systems and CPU architectures.
4) learning assembly language would not hurt, but