Importing data to an SQL database with Python?

Question

Hello Yahoo!,

I have no experience with SQL (all I know is that it is used to create/modify/search databases), but I was wondering if it would be the best choice for my application.

I have a Python script that loads/creates a lot of data. Currently, I store this data in a CSV text file, but this makes it very hard to search through the data quickly and there is more and more data every day. Instead of these CSV files, I would like to create a database that python is able to access/add to.

Is this possible? And are there any links/things to search for to get me started?

Thank you so much!

Answer 1

Relational databases are usually the best choice for any complex data structure and SQL is the defacto method for dealing with them. If you have the data in csv format, most relational database software should be able to import the data into tables that way (XML is the usual other alternative.)

Answer 2

Python comes with a built-in database: sqlite3



I Googled "python sqlite3 tutorial" and got a lot of helpful results. I'm only going to recommend one source because I haven't read through most of the tutorials and the few that I glanced at, there was something about the authors that rubbed me the wrong way (their code style just didn't make me think of them as great teachers).



The source I'd like to recommend is the official Python sqlite3 module documentation: http://docs.python.org/library/sqlite3.html

Feel free, however, to perform the same search and look through the results. They might convey their message to you and your learning style better than they did me and mine.



If you want to work with MySQL specifically, there's a module for that: http://sourceforge.net/projects/mysql-python/



If you want to work with PostgreSQL, there are several modules for that: http://wiki.postgresql.org/wiki/Python



Whatever you choose, I agree that databasing is the way to go if you're working with as large a data set as you say. Good luck!

Answer 3

SQL is really easy to learn, and there are free database programs out there for you to use, such as MySQL or MS SQL Server. I would highly suggest learning at least basic SQL.

Answer 4

you are able to hyperlink to an exterior records source interior of Excel. Its under kit. you will choose an ODBC definition lower back to the sq. Server with the ideal credentials. From there, you are able to create a pivot table or in certainty do in spite of you like. beware of the 65k row cut back in older variations of Excel. Exporting to MS get admission to is likewise an concept. in basic terms hyperlink the tables in get admission to lower back on your sq. Server db and carry out the ideal make table queries in get admission to. beware of the 2Gb. report length cut back for MS get admission to. good success.