如何合并许多 SQLite 数据库?

如果我有大量的 SQLite 数据库,它们都具有相同的模式,那么为了对所有数据库执行查询,将它们合并在一起的最佳方法是什么?

我知道使用 附件来做这件事是可能的,但是它有32个和64个数据库的 一个极限,这取决于机器上的内存系统。

99415 次浏览

If you only need to do this merge operation once (to create a new bigger database), you could create a script/program that will loop all your sqlite databases and then insert the data into your main (big) database.

With no offense, just as one developer to another, I'm afraid that your idea seems terribly inefficient. It seems to me that instead of uniting SQLite databases you should probably be storing several tables within the same Database file.

However if I'm mistaken I guess you could ATTACH the databases and then use a VIEW to simplify your queries. Or make an in-memory table and copy over all the data (but that's even worse performance wise, especially if you have large databases)

To summarize from the Nabble post in DavidM's answer:

attach 'c:\test\b.db3' as toMerge;
BEGIN;
insert into AuditRecords select * from toMerge.AuditRecords;
COMMIT;
detach toMerge;

Repeat as needed.

Note: added detach toMerge; as per mike's comment.

Although a very old thread, this is still a relevant question in today's programming needs. I am posting this here because none of the answers provided yet is concise, easy, and straight-to-point. This is for sake of Googlers that end up on this page. GUI we go:

  1. Download Sqlitestudio
  2. Add all your database files by using the Ctrl + O keyboard shortcut
  3. Double-click each now-loaded db file to open/activate/expand them all
  4. Fun part: simply right-click on each of the tables and click on Copy, and then go to the target database in the list of the loaded database files (or create new one if required) and right-click on the target db and click on Paste

I was wowed to realize that such a daunting task can be solved using the ancient programming skill called: copy-and-paste :)

Late answer, but you can use:

#!/usr/bin/python


import sys, sqlite3


class sqlMerge(object):
"""Basic python script to merge data of 2 !!!IDENTICAL!!!! SQL tables"""


def __init__(self, parent=None):
super(sqlMerge, self).__init__()


self.db_a = None
self.db_b = None


def loadTables(self, file_a, file_b):
self.db_a = sqlite3.connect(file_a)
self.db_b = sqlite3.connect(file_b)


cursor_a = self.db_a.cursor()
cursor_a.execute("SELECT name FROM sqlite_master WHERE type='table';")


table_counter = 0
print("SQL Tables available: \n===================================================\n")
for table_item in cursor_a.fetchall():
current_table = table_item[0]
table_counter += 1
print("-> " + current_table)
print("\n===================================================\n")


if table_counter == 1:
table_to_merge = current_table
else:
table_to_merge = input("Table to Merge: ")


return table_to_merge


def merge(self, table_name):
cursor_a = self.db_a.cursor()
cursor_b = self.db_b.cursor()


new_table_name = table_name + "_new"


try:
cursor_a.execute("CREATE TABLE IF NOT EXISTS " + new_table_name + " AS SELECT * FROM " + table_name)
for row in cursor_b.execute("SELECT * FROM " + table_name):
print(row)
cursor_a.execute("INSERT INTO " + new_table_name + " VALUES" + str(row) +";")


cursor_a.execute("DROP TABLE IF EXISTS " + table_name);
cursor_a.execute("ALTER TABLE " + new_table_name + " RENAME TO " + table_name);
self.db_a.commit()


print("\n\nMerge Successful!\n")


except sqlite3.OperationalError:
print("ERROR!: Merge Failed")
cursor_a.execute("DROP TABLE IF EXISTS " + new_table_name);


finally:
self.db_a.close()
self.db_b.close()


return


def main(self):
print("Please enter name of db file")
file_name_a = input("File Name A:")
file_name_b = input("File Name B:")


table_name = self.loadTables(file_name_a, file_name_b)
self.merge(table_name)


return


if __name__ == '__main__':
app = sqlMerge()
app.main()

SRC : Tool to merge identical SQLite3 databases

If you have reached the bottom of this feed and yet didn't find your solution, here is also a way to merge the tables of 2 or more sqlite databases.

First try to download and install DB browser for sqlite database. Then try to open your databases in 2 windows and try merging them by simply drag and drop tables from one to another. But the problem is that you can just drag and drop only one table at a time and therefore its not really a solution for this answer specifically but yet it can used to save some time from further searches if your database is small.

Here is a simple python code to either merge two database files or scan a directory to find all database files and merge them all together (by simply inserting all data in other files to the first database file found).Note that this code just attaches the databases with the same schema.

import sqlite3
import os




def merge_databases(db1, db2):
con3 = sqlite3.connect(db1)


con3.execute("ATTACH '" + db2 +  "' as dba")


con3.execute("BEGIN")
for row in con3.execute("SELECT * FROM dba.sqlite_master WHERE type='table'"):
combine = "INSERT OR IGNORE INTO "+ row[1] + " SELECT * FROM dba." + row[1]
print(combine)
con3.execute(combine)
con3.commit()
con3.execute("detach database dba")




def read_files(directory):
fname = []
for root,d_names,f_names in os.walk(directory):
for f in f_names:
c_name = os.path.join(root, f)
filename, file_extension = os.path.splitext(c_name)
if (file_extension == '.sqlitedb'):
fname.append(c_name)


return fname


def batch_merge(directory):
db_files = read_files(directory)
for db_file in db_files[1:]:
merge_databases(db_files[0], db_file)


if __name__ == '__main__':
batch_merge('/directory/to/database/files')