There are some logic issues which I will get to but to start with there are some donts as far as javascript goes as the code you are working with looks very dated and may not work across different browsers. In order of occurrence:
The language attribute in the script tag is deprecated, use type instead.
Get into the habit of closing statements with a semi-colon as not only does it allow for cleaner code and more statements per line - it's much easier to debug when errors start occurring in more complex code.
You shouldn't just access a DOM object by it's ID like you have as this could quite easily cause errors or fail completely in some browsers. You should fetch an object reference first to avoid ambiguity.
setTimout works but for animation it makes more sense to use setInterval as it will be more accurate as code gets more complicated and eliminates any confusing problems that can crop up due to the recursive nature of setTimeout. Also note you should reference the function directly in the parameters rather than giving it a string value like you have, as A) it then won't need parsing and B) is more secure.
You should set up an event listener for onload too as it is a good idea to keep javascript and html entirely separate for usability - it is more versatile that way too.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
To your logic:
The reason your ball "jerks back" is because of this line: ball.style.pixelLeft =-3
you probably meant to say: -= instead of =- as you are setting pixelLeft to -3
Once you have sorted that though you get the next error, that is once you reach 400 pixels you get to a point where you are both adding and subtracting 3 in the same operation thus sending the ball nowhere.
There's also an incorrect logical assumption of deducting 3 from pixelLeft after 400 pixels (thus changing the direction) but if you imagine the steps for example:
pL=398 (add 3),
p=401 (sub 3),
p=398 (add 3)
... you are going to get to a point of jumping back and forth which isn't what you want - this is why your dir variable was declared global.
I've altered your code, the logic should be easy to follow but give me a shout if there is anything specific you don't get.
Simple Animation