Bulk load into empty dimension uses too much memory (outofmemory errors)

Topics: Possible Bugs/Missing Features?
Jan 8, 2013 at 9:00 AM

Hi all,

I've used the v1.6 task for a few smaller dims but i've got one that's an initial size of 1.2mil where the row size has an avg value of 400bytes (only half a gig)

but the task fails with out of memory errors as it gets close to 1million

This is the 32bit version

I notice that the memory commit size for the process increases as the source rows are passing in, and are only going out at 1/2 to 1/4 the speed with which they come in (im using a junk destination). I've reduced the number of rows to get it to work and i can see that the commit hits a peak when the input from the source is finished and just sits there! It doesn't decrease as the rows get slowly fed through to the destination

my scd settings are

every field monitored for type2

effectivefrom/to dates and an iscurrent bit field

surrogate keys handled by the database

unchanged, deleted, scd type1, invalid, statistics, auditing are turned off (all fields excluded)

new scd type 2 redirected to new

 

I set up a quick demo at home on my 64bit machine and the behavior was the same

a single data flow with just the scd task and related oledb sources

I used information_schema.columns + row_number() as the source (repeated to 10mil rows). The memory usage grew to about 6gig and sat there until the package finished

 

Is there any setting i can turn on/off that'll cause the 'new' rows to not be buffered?