Component seems to hang

Topics: Possible Bugs/Missing Features?
Aug 22, 2011 at 1:31 PM
Edited Aug 23, 2011 at 1:01 AM



First - I love this component, thank you!

I have used it many times but I am now getting an error that I am not sure how to troubleshoot.

This is the initial load for this table so there are no records coming in through the Existing Dimension input (it is hooked up though).  The package runs fine until it gets to 162,800 of 162,875 source rows.  At that point I get the error below and the package seems to "hang" - Visual Studio is not frozen but the Dimension Merge component is yellow showing 162,875 rows on the source input processed, 0 on the existing, and 162,800 on the output of "New" - and nothing happens from there.  

Here is the error I see:

Error: 0x0 at Load Vibe Ship Tos, Dimension Merge Slowly Changing Dimension: Internal error (Error building work units: Error getting next work unit - marking all keys matched: Internal error in RuntimeInputCache retrieving current key list.) in ProcessCache_Thread_MatchKeys.

  • The Existing Dimension has 0 rows (this is the first load)
  • The Source Dimension has 162,875 rows and is sorted
  • SQL 2008 R2, I am running the package from within Visual Studio in 64bit mode
  • The 162,800 that go to "New" are actually inserted in the DB.
  • I thought this may be a memory problem, initially the machine had only 4GB of RAM - I bumped it to 10GB and verified that 8.5GB were free before starting but I get the exact same result, I would expect if it were memory related it would fail a little further down the line now that it had access to more memory.  I notice the package gets up to about 5GB at it's peak.

As a note I did try running it a second time after the 162,800 rows had been committed so that the existing dimension now had 162,800 rows - it still "hangs" at 162,875 on the source input and shows nothing on any of the outputs.  In this case no error is thrown so the package continues to "run" but nothing is happening.  I have let it sit for 12 hours and nothing happens, I just see "Sleeping - no work" in the output window. 

I continually get this message in the output while it "hangs"

Information: 0x0 at Load Vibe Ship Tos, Dimension Merge Slowly Changing Dimension: 0 II, 162669 UC, 215 NW, 0 DL, 9 S1, 10 S2E, 10 S2N

Information: 0x0 at Load Vibe Ship Tos, Dimension Merge Slowly Changing Dimension: ED 0/162688(comp), SS 1/162904(comp) - 0 matched, 0 work units ready.

Information: 0x0 at Load Vibe Ship Tos, Dimension Merge Slowly Changing Dimension: 0 II, 162669 UC, 215 NW, 0 DL, 9 S1, 10 S2E, 10 S2N

Information: 0x0 at Load Vibe Ship Tos, Dimension Merge Slowly Changing Dimension: ED 0/162688(comp), SS 1/162904(comp) - 0 matched, 0 work units ready.

Information: 0x0 at Load Vibe Ship Tos, Dimension Merge Slowly Changing Dimension: 0 II, 162669 UC, 215 NW, 0 DL, 9 S1, 10 S2E, 10 S2N

 

 

I'm not sure where to look to troubleshoot this, here is what I have tried:

  • I have tried sorted inputs on the existing and source inputs and I have tried removing the sort component (I get the performance warning at that point)
  • I have tried disabling threading
  • I have tried all three hash methods
  • I have tried turning on all debugging (as in this post: http://dimensionmergescd.codeplex.com/workitem/5400 )
  • Running as Administrator
  • Limiting the source input (using "TOP xxx") --> The cutoff seems to be somewhere around 105,000 - below that it finishes and above that it "hangs"

 

Thank you - 

Jason

Mar 20, 2012 at 1:10 AM

Hi,

I have the same problem.
I've been able to track it down a business key problem.
The existing dimension contains a lot of duplicate keys.
The business key is a DT_WSTR field and most of the duplicate keys has a length of 3 characters.

A soon as I add a single character to the business key so they are at least 4 characters, the dimension merge component works just fine.

Know that the above sounds kind of crazy but this is what I have found when testing why the package "hangs" without freezing BIDS.

Hope this will help someone.

Thanks -

Gunnar

Mar 20, 2012 at 12:25 PM

Thank you - we "fixed" this by putting in a Sort and checking to remove duplicates so it definately aligns with what you are seeing. 

Mar 20, 2012 at 1:34 PM

Thank you palmersu,

There are more oddities in this case.
My suggested solution to add a single character did only work when adding the character 'a'.
Adding character 'b' did not solve anything, the good old "hang" was back...

So, the now 100% working solution is to Base64 encode the business key in both source dimension and exsisting dimension.
This will assure 4 things:
1. Business key will be at least 4 characters
2. No illegal characters (is there any??)
3. Business key is of type DT_STR, not unicode.
4. The Dimension Merge component can handle the duplicates as expected.

Have tried to recreate the problem reading the same data from a flat file but no luck.
The component works as expected in that case.

Specs:
- Sql Server 2008 R2 x64
- Ole Db source

Thanks

Gunnar