I have to design a table where records will be inserted in two ways.
- Bulk insert from a batch process from a delimited text file between 2 – 3 million rows in a short burst. Later the same records will also need an bulk update for the status. e.g. from ‘Pending’ to ‘Completed’.
- Single row insert from a LOB application, via stored procedure, say one thousand rows per day. Some of these records will not require the status update and some will.
Process 1 is already in place (The text file is used as an external table. insert/update via a stored procedure) but we are willing to re-design this as currently we have about 2 million rows but we are expecting major volume growth of say 30 million records per year from the batch plus say 3.5 million from the application. Data for 7 years needs to be stored. Data will be queried using a Id, type and date range (say current year) with pagination.
The preferred DB is Oracle 11G. But I am also open to SQL Server 2012 enterprise edition. In this case process 1 will perhaps be replaced by SSIS.
My questions are:
- What are the considerations required for designing a table like this? Should the bulk and single row insert be in different tables with identical structure? I wouldn’t like the bulk insert to affect the performance of single row insert and query when happening at the same time.
- Is there any significant difference between using Oracle vs SQL Server for this scenario specially for performance?
- Is SSIS appropriate to use for process 1 if SQL Server is chosen?