Saiful,
There are a few aspects to getting your MEX-file running in 64-bit MATLAB. Once you've done that, you can also enhance it to handle very large arrays.
You can find most of this information in the MEX-files section of the MATLAB documentation:
http://www.mathworks.com/access/helpdesk/help/techdoc/matlab_external/
1. You'll need a supported 64-bit compiler:
http://www.mathworks.com/support/compilers/current_release/
Many compilers (including LCC, which is shipped with MATLAB on 32-bit Windows) can't create 64-bit binaries.
Starting with a simple MEX-file such as the documentation example "yprime.c", rather than your full code, makes it easier to make sure you have the compiler configured properly.
2. Compile, debug, and test your MEX-file on your 64-bit system.
You can find instructions for attaching your debugger here:
http://www.mathworks.com/support/tech-notes/1600/1605.html
If you encounter compiler errors, post a more specific question (with the full error message) in one of the forums listed below.
3. (C/C++ handles pointer sizes automatically. For Fortran users with similar questions --- you'll need to use the mwPointer data type for pointers. See "doc mwPointer" for more help.)
Those steps will get your existing code working on your 64-bit system.
4. If you are moving to 64-bit in order to work with very large arrays of data, the next step is to update your MEX-file to support those:
"How do I update MEX-files to use the large array handling API (-largeArrayDims)?"
http://www.mathworks.com/support/solutions/en/data/1-5C27B9/?solution=1-5C27B9
Besides Yahoo Answers, there are several other helpful forums for MATLAB questions:
5. The MATLAB newsgroup, comp.soft-sys.matlab . One way to access this group is:
http://www.mathworks.com/matlabcentral/newsreader/
You can find many similar questions (and their answers) in this newsgroup.
6. MathWorks Technical Support,
http://www.mathworks.com/support/contact_us/index.html
Good luck and happy coding.
Sincerely,
Ken Mitton
Quality Engineer
The MathWorks, Inc.