I want to write a script to download data from a website?

Question

I want to write a script that will go and grab data from a website. Specifically, the weatherunderground site. I need it to be able to grab a date from a list, go to weatherunderground, and pull up a comma delimited file and save it to a directory.

I'm not sure how to go about doing this. My computer will compile python and perl, but I only program in IDL (similar to FORTRAN). If I could get perl/python to do this one task, I can put it in an IDL program that will read in the data for me. I know this is possible, but I don't know how to go about doing it. Here's the site. http://www.wunderground.com/history/index.html Do I pass this site to the script and somehow tell it what to enter, or do I pass it the full address of the file (just the dates change in the address)? How do I get it to go 'grab' the data and save it for me?

Answer 1

You know what I would do Eri? Pull down the html source from the website and parse out the tags to find what you are looking for... This way, you can write the parse routine using IDL instead of having to futz with Perl/Python.



Let me go take a look at the source of such a feed and get back to you in a little while.



UPDATE: Ok, here you go.



To get directly to a comma delimited output file, use the following URL:



http://www.wunderground.com/history/airport/KSUS/2008/5/29/DailyHistory.html??format=1&format=1



As far as I can tell, it uses the airport code (in this case KSUS) to find the right weather data. The other information is for display only, but I changed it just in case.



So, you need to write a script that will send the URL string above with the correct Airport Code, Date and other information. What comes back to you (HTML already removed) is this.



TimeCDT,TemperatureF,Dew PointF,Humidity,Sea Level PressureIn,VisibilityMPH,Wind Direction,Wind SpeedMPH,Gust SpeedMPH,PrecipitationIn,Events,Conditions

12:54 AM,54.0,50.0,86,30.19,10.0,Calm,Calm,-,N/A,,Clear

1:54 AM,53.1,48.9,86,30.18,10.0,Calm,Calm,-,N/A,,Clear

2:54 AM,52.0,48.9,89,30.18,10.0,Calm,Calm,-,N/A,,Clear

3:54 AM,52.0,48.0,86,30.18,10.0,Calm,Calm,-,N/A,,Clear

4:54 AM,51.1,48.0,89,30.18,10.0,Calm,Calm,-,N/A,,Clear

5:54 AM,50.0,46.9,89,30.18,10.0,Calm,Calm,-,N/A,,Clear

6:54 AM,54.0,48.9,83,30.18,10.0,Calm,Calm,-,N/A,,Clear

7:54 AM,59.0,51.1,75,30.19,10.0,Calm,Calm,-,N/A,,Clear

8:54 AM,66.0,54.0,65,30.19,10.0,Calm,Calm,-,N/A,,Clear

9:54 AM,71.1,48.0,44,30.18,10.0,Calm,Calm,-,N/A,,Clear



Obviously, the first line is the heading line, and the rest is the data that is available. Of course, this is comma delimited and can be read directly into anything that supports that format.



So then your script just needs to run until it has pulled the date ranges that you want.



If need help with writing the actual script, let me know and I'll post a stub. I just don't have alot of time to day or I would have written it for you.



Al Moskowitz (aka, Achmed)



UPDATE: I momentarily forgot what you do for a living!! :) Go to the CS department and they can write you a scirpt to do what I've described in about 30 minutes fully tested.

Answer 2

Well, if you look at the website address it spits out, you'll get something that looks like this:



http://www.wunderground.com

/history/airport/KNYC

/2008/5/29/DailyHistory.html

?req_city=Jersey+City

&req_state=NJ

&req_statename=New+Jersey



I put in my zipcode and this was my output. Yours will differ of course, but you can set a script to write anything in those places. You'll probably do best to hardcode your city and state and stuff, and make the date variable.



Do you know bash? Because that's what I would use. Its for Unix and its really easy.

Anyway, what you want is to get the html file (wget in Unix and bash).

Then remove all the html formatting (sed -e :a -e 's/<[^>]*>//g;/
Good luck!

Answer 3

There is a whole book written on how to scrape the web using Perl. Follow the link below...

Answer 4

Try DAP.... Its a software... After installing the software right click on the web page from where u want to download a set of files.... Select Download all with DAP.... It will grab all the downloadable files and show u.... U can select from the list of downloadable files which u want to download... I hope this will help u out...

Answer 5

so if i understand you correctly you need a form that uploads files? 1. html script:

2. php script: " if (move_uploaded_file($FILES['file']['tmp_... { echo "File uploaded" else echo "File failed to upload" } echo "" ?> save this php code as upload.php and put it in the same directory as the html script. hope that helps, if you have more problems feel free to contact me.