The easiest way to ensure you don't get duplicates, is to check each time for duplicates, and get a new random number if one crops up. If you're doing it this way, it may be easier to store an index giving you a quick way of looking up a number which has already been used.
However, if you want to ensure that you never duplicate a day/period, it may be easier to walk through the days and periods and randomly pick the subject (rather than randomly pick all the day, period and subject).
If you want to limit the number of occurrences of a subject, then keep a tally of subjects and get a fresh number if you don't like the one you've been given. Each time you get an acceptable subject, reduce the tally. If the tally reaches 0, the subject isn't acceptable again.
For example:
/* define the maximum number or times we are prepared to loop to find an availiable subject before giving up */
#define NUMBER_OF_ATTEMPTS 10
/* define the number of times each subject may be picked, always make the 0th item more than the number of periods in a week, to be sure there is something that can always be picked. Each number corresponds to the number of times you'll allow a particular subject (ie 8 for subject 1 and 6 for subject 3 */
int subject_tally[]={9999, 8,8,6,6,4};
/* The array containing your final data*/
int picks[5][8];
/* The loop counter variables */
int day, period, subject, attempts;
/* initialise random seed */
srand (time(NULL));
/* loop through each period of each day*/
for (day=0; day<5; day++)
{
for (period=0; period<8; period++)
{
attempts=0;
/*attempts loop prevents run away */
while (attempts++
{
subject = rand() %6 ; /* gives a random number between 0 and 5 */
/* only consider the subject if it still has an excess of available periods*/
if (subject_tally[subject]>0)
{
picks[day][period]=subject;
subject_tally[subject]--;
/* Leave this while loop if a subject was successfully picked*/
break;
}
}
/* if we ran out of attempts, then make it a free period */
if (attempts > NUMBER_OF_ATTEMPTS)
{
picks[day][period]=0;
}
}
}