SSIS Dimension Merge SCD Component

Commented Issue: Duplicates in Dimension tables when many SCDC controls run in Parallel [10368]

jantje_20 — Mon, 20 Feb 2012 07:02:07 GMT

We have six packages that all have SCDC controls running in parallel they create duplicate row(s) in dimension tables, When i create a unique index on the Business Key columns the SCDC control fails with unique key constraint violation.

However when i change the packages to run in sequence this issue is resolved.
We are using "Dimensional Merge Slowly Changing Dimension Transformation V1.6 for SSIS 2008 X86"

did anyone face a similar situation.
Comments: ** Comment from web user: jantje_20 **

we run into the same problem since a couple of weeks ago. The months before the 3 packages were running fine in parallel. Last two weeks i've been looking to solve this problem, but I can't find a solution, except running the packages in parallel.
It's odd though, it's always the same package that keeps failing, and it's only failing in 20% of the time.
We're using version 1.5 of the component on a 64bit windows/sql 2008 server

This is the error i get:

SSIS Error Code DTS_E_OLEDBERROR. An OLE DB error has occurred. Error code: 0x80004005.
An OLE DB record is available. Source: "Microsoft SQL Server Native Client 10.0" Hresult: 0x80004005 Description: "The statement has been terminated.".
An OLE DB record is available. Source: "Microsoft SQL Server Native Client 10.0" Hresult: 0x80004005 Description: "Violation of PRIMARY KEY constraint 'PK_Dim_Table1'. Cannot insert duplicate key in object 'dbo.Dim_Table1'.".

SSIS Error Code DTS_E_INDUCEDTRANSFORMFAILUREONERROR. The "input "OLE DB Destination Input" (12478)" failed because error code 0xC020907B occurred, and the error row disposition on "input "OLE DB Destination Input" (12478)" specifies failure on error. An error occurred on the specified object of the specified component. There may be error messages posted before this with more information about the failure.

SSIS Error Code DTS_E_PROCESSINPUTFAILED. The ProcessInput method on component "Insert Destination" (12465) failed with error code 0xC0209029 while processing input "OLE DB Destination Input" (12478). The identified component returned an error from the ProcessInput method. The error is specific to the component, but the error is fatal and will cause the Data Flow task to stop running. There may be error messages posted before this with more information about the failure.

Created Issue: Duplicates in Dimension tables when many SCDC controls run in Parallel [10368]

chandrasc — Mon, 13 Feb 2012 14:56:31 GMT

New Post: SCD Wizard not holding values

adrianturcu — Thu, 02 Feb 2012 08:48:36 GMT

Hello,

I'm using v1.6 on Sql Server 2008r2 sp1 64bit on WS2008R2 64, and I'm experiencing the following behavior:

Tab - Auditing - New SCD2 Output Counter - Keeps resetting to <not used>

As a workaround i put a RowCount at the output of "New SCD2" pipeline

Thanks,

Adrian

New Post: Permission error while executing from SQL Server Agent job

akhakhkhaR — Fri, 27 Jan 2012 16:43:05 GMT

I am using v1.6 of the Dimenstion Merge SCD component (Dimension Merge SCD Setup 2008 x64.msi)and I am suddenly getting this error when running my package from a SQL Server Agent job:

A problem occurred reading the metadata for the Slowly Changing Dimension component: Access to the temp directory is denied. Identity 'HQ\bi_ssis_job_prod' under which XmlSerializer is running does not have sufficient permission to access the temp directory. CodeDom will use the user account the process is using to do the compilation, so if the user doesnt have access to system temp directory, you will not be able to compile. Use Path.GetTempPath() API to find out the temp directory location. Metadata will be regenerated, but some settings may be lost. If you also receive an error message stating the component is 'Unable to generate a temporary class', then you may need to grant the account executing the package rights to the Windows Temporary folder.

I am not aware of any permissions that have changed in my production environment so I am wondering what could have provoked this failure. After granting the user id full administrative privileges, the job runs successfully. Any insight into the cause for the above error would helpful.

Thanks,

Ami

Created Issue: SCD2 Inserting New Records even though nothing has changed [10242]

gbuschmann — Thu, 26 Jan 2012 20:30:32 GMT

I'm havine an issue with the SCD2. I have a dimensions where I want to use SCD2 for any attribute field that changes. In the set up I've identified a surrogate key, business key, SCD2 Effective Date, SCD2 Expiry Date, and SCD2 Current Record. I've used the system container start time for the "now" value, granulatiry of a minute, and the last record expiry date is a varialbe stating 12/31/9999. I have used the 0 true current member, and 1 false current member. Everything else is pretty straight forward and I have used most of the defaults. The New output includes all recommended fields.

On the first run (table starts empty) all expected records load. On a second load with no changes to either the dimension or source data,all the records load again. The minute time has passed and the new records inserted are exactly the same as first record. I would expect the reocrds would go to unchanged on the second run. Any input as to why they would go to New output again?

Thanks in advance for any help.

New Post: Update history for SCD Type 1 Changes

WeeeBob — Wed, 25 Jan 2012 10:03:13 GMT

Ahh a little clearer now.

I tend to just go for option 2 for Type 1 changes, if your not doing this then its not a Kimball type 1 change? as you are now tracking change in time for this column. I'll assume you have a valid reason though (maybe late arriving dimension attributes?), and by the sounds of it a conditional split (rather than multicast) on the SCD 1 output would be your best bet.

New Post: Update history for SCD Type 1 Changes

howetj — Wed, 25 Jan 2012 05:08:37 GMT

WeeeBob - The intro was an attempt on some light humour - sorry you missed it.

I am fully aware of how the DM SCD component works and that it does not create downstream components but rather different output paths to deal with the different types of SCD changes - so no I am not getting confused.

The major piece of the puzzle that you might be missing is that I am asking whether there might be some kind of workaround to deal with Type 1 changes that:

1) update the value in the current record for some columns (i.e we are not interested in fixing expired records) or

2) update the value in all expired rows for other columns (i.e. we want to fix expired records)

I was thinking of maybe adding a multicast component to split the Update SCD 1 output into 2 different streams and doing a update on the surrogate key for 1 above and an update using the business key(s) for type 2

Still thinking about this - comments welcome

New Post: Update history for SCD Type 1 Changes

WeeeBob — Tue, 24 Jan 2012 16:37:53 GMT

what discussion? this is a new discussion? a tad confusing Trevor.

The SSIS Dimension Merge SCD component doesn't create your down stream components or alter them when you make changes to the component, One of the advantages to using it compared to the standard Slowly Changing Dimension component. So no wizard? Are you getting confused between the components?

or am I missing some major piece of the puzzle here?

New Post: Update history for SCD Type 1 Changes

howetj — Tue, 24 Jan 2012 13:56:10 GMT

Before closing this discussion thinking - this dude does not know what he is talking about - SCD1 Changes do not have history - let me explain.

We have a dimension that has a mix of SCD type 1 and SCD type 2 changes. Lets use a Client dimension as an example. One of the attributes, example Geographical location has been identified as Type 2 and this will create new records and expire old when the Geographic location changes. Another attribute is Date of Birth which is type 1 - a clients date of birth does not change so why track history.

Over time Client with a business key of 'ABC' had 3 different Geographic locations and therefore has two expired records and one current one. Next we discover that the Date of Birth had always been incorrect and the new one is run in the ETL process. This correctly updates the current record with the correct date of birth.

But now we want to update the 2 expired records as well - and we want to do it via the Dimension Merge SCD wizard.

Currently our Update SCD 1 leg does an update statement like:

Update DimClient Set DateOfBirth = ? WHERE ClientKey = ?

but for the history update this must be something like:

Update DimClient Set DateOfBirth = ? WHERE ClientCode = ? (i.e. business key in where clause will update all history and not just the individual row)

Anyone have any ideas of how to do this via the wizard. Can it be done.

Thanks

Trevor

Commented Issue: Unpredictable sorting of "New" output [10174]

Sm1th — Mon, 23 Jan 2012 07:06:34 GMT

Hi! I have 2 inputs - source data and existing dimension data. Source data is sorted by Order by clause. Starting with 1 existing record in dimension (unknown member), so it's first time filling. When new records going out of DMSC they are sorted in a strange manner - by blocks or something like this, but not like intput records. This is not the order I'd like to see in my dimension table.
I tried to set IsSorted property with sortKeyPosition on business key field, but it didn't help. Tried do that on source, on ed, on both of them and by apart - no use.
Of course Sort component come to the rescue, but asynchronous component is not desirable.
I found a DMSC property - Number of Processing Threads with value 0, changed that to 1 and got what I want - sorted new records. Tried to change to 2 and returned to the problem.
So what to prefer - left 1 thread (on multicore system), use Sort component or do someting else? In terms of performance and best practice.
Thank you.
Comments: ** Comment from web user: Sm1th **

A remark:
setting Number of Processing Threads = 1 doesn't help with dimension tables size of about 10k rows+. It works just fine with small dimensions, but for a bit larger ones need use Sort component for New records output. Sad that the component messing up the input stream

Created Issue: Unpredictable sorting of "New" output [10174]

Sm1th — Thu, 19 Jan 2012 06:33:33 GMT

New Post: Dimension merge SCD stop running when data reach 9000 records

WeeeBob — Fri, 06 Jan 2012 10:03:23 GMT

Couple things I've noted about the image you supplied.

1. Your existing dimension hasn't supplied any rows yet to the dimension merge (EXIST_PROM_X_ITM OLE DB Source) and this is why the package is waiting. Could you have a lock problem? Try adding WITH (NOLOCK) hint into the source query.

2. You have a 'fully blocking' component in there, a row count. Try removing this for troubleshooting purposes.

Also make sure your new data and existing dimension sources are sorted in the OLE DB source, and don't forget to tell the component which fields your sorting on. The Natural Key/Business Key is what you should be sorting on.

Looks to me like you have problems with the data sources rather than the SCD component. You can also try hooking your existing dimension source to a trash destination for troubleshooting to ensure the component is supplying data.

New Post: Dimension merge SCD stop running when data reach 9000 records

areraizard — Fri, 06 Jan 2012 09:11:26 GMT

Hi everyone,

I have a problem with Dimension merge SCD in my SSIS package, my problem is about performance when I try to run data around 10,000 records.

So I test my package by a small group of data such 10, 100, 1,000 or 6,000 records and finished in a few seconds but when I try 9,000 records and full data(around 12,000 records) the package stop when running at 9,000 records and not doing anything even though I try to wait for it more than half a hour.

Please see screenshot of my package for more information.

http://img684.imageshack.us/img684/967/scdssis.jpg

Thank you for your help :)

New Comment on "FAQ"

bcruz03 — Wed, 28 Dec 2011 15:56:16 GMT

I installed the component on a VSS Team Sytem 2008 and when selecting to add custom toolbox items the VSS closes. Tried re-intsalling the product with same result. Do you have a solution to the problem? Thanks

Commented Issue: v1.6 2005 x86 Install wants DimensionMergeSCD100, not 90 [9031]

dave_lumley — Sun, 25 Dec 2011 01:15:42 GMT

Installed v1.6 using "Dimension Merge SCD Setup 2005 x86.msi". Always get the error message pasted below when trying to work with the transformation. Note that it is referencing DimensionMergeSCD100, not DimensionMergeSCD90.

Could not load type: MouldingAndMillwork.SSIS.DimensionMergeSCDEditorInterface,DimensionMergeSCD100,Version=1.0.0.0,Culture=neutral,PublicKeyToken=8b0551303405e96c. Verify that the component editor is installed properly.

------------------------------
ADDITIONAL INFORMATION:

Could not load file or assembly 'DimensionMergeSCD100, Version=1.0.0.0, Culture=neutral, PublicKeyToken=8b0551303405e96c' or one of its dependencies. The system cannot find the file specified. (mscorlib)
Comments: ** Comment from web user: dave_lumley **

2
votes
voted
vote v1.6 2005 x86 Install wants DimensionMergeSCD100, not 90
Add Comment

DescriptionInstalled v1.6 using "Dimension Merge SCD Setup 2005 x86.msi". Always get the error message pasted below when trying to work with the transformation. Note that it is referencing DimensionMergeSCD100, not DimensionMergeSCD90.

Could not load type: MouldingAndMillwork.SSIS.DimensionMergeSCDEditorInterface,DimensionMergeSCD100,Version=1.0.0.0,Culture=neutral,PublicKeyToken=8b0551303405e96c. Verify that the component editor is installed properly.

------------------------------
ADDITIONAL INFORMATION:

Could not load file or assembly 'DimensionMergeSCD100, Version=1.0.0.0, Culture=neutral, PublicKeyToken=8b0551303405e96c' or one of its dependencies. The system cannot find the file specified. (mscorlib)

File Attachments
2011-06-29 16h39_47.png
kimball-scd-error.png

Closed by

Comments
tomchen22 wrote Today at 5:37 AM
attached is my error screen.

tomchen22 wrote Today at 5:22 AM
Hi Dave or anybody who can help:

can I get a working version v1.6 for SQL 2008 per your post at
http://dimensionmergescd.codeplex.com/workitem/9031?FocusElement=CommentTextBox

I tweaked source code and changed version to be 1.0.0.1 but when doing choose item from toolbox in data flow to point to updated version of assembly in pipeline components, it still complains about the above error with reference to 1.0.0.0. I wonder where in the SSIS project / BIDS I can remove that hard code reference so that it will update to use 1.0.0.1 of my tweaked assembly?

dave_lumley wrote Jul 6 at 9:35 AM
Hi everyone,

A number of people have been asking me for the working SQL2005 v1.6 files - so, heres a link for downloading the file with the working dll's and how to apply them:

https://skydrive.live.com/?cid=f3f613fedae3e017&sc=documents&Bsrc=EMSHOO&Bpub=SN.Notifications&id=F3F613FEDAE3E017%21128#

Commented Issue: [KSCDv1.5 SQL2005 x86] Upgrade Validation errors occur when opening a saved SSIS Package with an incomplete KSCD Transformation inside [6222]

tomchen22 — Sat, 24 Dec 2011 20:37:25 GMT

Environment:
Win2K8 SP2,
SQL 2005 (SP3) x86,
Visual Studio 2005 Team Suite sp1
KSCD 1.5

Steps to repro:
1) Create new package
2) Add Dataflow
3) Add KSCD to Dataflow
4) Save Package
5) Close Package
6) Open Package (should now have 3 errors that aren't the ones you'd expect to see)
7) Open Data flow
8) Click on KSCD (error box)

Results from Step 6 - 3 Errors in package
Error 1 Validation error. Data Flow Task: DTS.Pipeline: The component metadata for "component "Kimball Method Slowly Changing Dimension" (1)" could not be upgraded to the newer version of the component. The PerformUpgrade method failed. Package1.dtsx 0 0
Error 2 Error loading Package1.dtsx: System.ApplicationException: Unable to upgrade Existing Dimension column metadata for <?xml version="1.0" encoding="utf-16"?><ArrayOfInputColumnDefinition /> column: Index was outside the bounds of the array. ---> System.IndexOutOfRangeException: Index was outside the bounds of the array. at MouldingAndMillwork.SSIS.KimballMethodSCDConfiguration.PerformUpgrade(Int32 fromVersion, IDTSVariableDispenser90 variableDispenser) --- End of inner exception stack trace --- at MouldingAndMillwork.SSIS.KimballMethodSCDConfiguration.PerformUpgrade(Int32 fromVersion, IDTSVariableDispenser90 variableDispenser) at MouldingAndMillwork.SSIS.KimballMethodSCD.PerformUpgrade(Int32 pipelineVersion) at Microsoft.SqlServer.Dts.Pipeline.ManagedComponentHost.HostCheckAndPerformUpgrade(IDTSManagedComponentWrapper90 wrapper, Int32 lPipelineVersion) E:\Projects\TemplateDesign\TemplateDesign\Package1.dtsx 1 1
Error 3 Error loading Package1.dtsx: The component metadata for "component "Kimball Method Slowly Changing Dimension" (1)" could not be upgraded to the newer version of the component. The PerformUpgrade method failed. E:\Projects\TemplateDesign\TemplateDesign\Package1.dtsx 1 1

Results from Step 8 - Error Message Box
KSCD fails validation due to an upgrade error :

TITLE: Microsoft Visual Studio
------------------------------

Component with name "Kimball Method Slowly Changing Dimension" and class ID "{BF01D463-7089-41EE-8F05-0A6DC17CE633}" could not be created because an error was encountered during its upgrade to the current version.
Contact information:

For help, click: http://go.microsoft.com/fwlink?ProdName=Microsoft%u00ae+Visual+Studio%u00ae+2005&ProdVer=8.0.50727.867&EvtSrc=Microsoft.DataTransformationServices.Design.SR&EvtID=ComponentUpgradeFailed&LinkId=20476

------------------------------
BUTTONS:

OK
------------------------------
Comments: ** Comment from web user: tomchen22 **

I wonder if Todd or anyone could me what exactly to be done as to how to reuse the codes to upgrade or open in a newer version of SQL say SQL 2008 or Visual Studio 2008, in addition to changing the UITypeName in AssemblyInfo.cs. in Todd's original codes it's hard coded as "... DimensionMergeSCD100,Version=1.0.0.0,Culture=neutral,PublicKeyToken=8b0551303405e96c", I guess that's why I can change something like GACUtil in post buildevents to make it buildable but still encounter 'could not load type... the system cannot find the file specified' while dragging it from toolbox to data flow design surface. I know I need to rewrite PerformUpgrade method but don't know exactly where are these locations, something like whats mentioned at http://agilebi.com/jwelch/2009/09/28/implementing-performupgrade-in-custom-components/ .

Commented Issue: v1.6 2005 x86 Install wants DimensionMergeSCD100, not 90 [9031]

tomchen22 — Sat, 24 Dec 2011 05:37:05 GMT

attached is my error screen.

Commented Issue: v1.6 2005 x86 Install wants DimensionMergeSCD100, not 90 [9031]

tomchen22 — Sat, 24 Dec 2011 05:22:41 GMT

Hi Dave or anybody who can help:

can I get a working version v1.6 for SQL 2008 per your post at
http://dimensionmergescd.codeplex.com/workitem/9031?FocusElement=CommentTextBox

New Post: SSIS out of memory exception

ashishbhushan2 — Tue, 20 Dec 2011 05:51:15 GMT

Hi, thanks for your reply. I've tried to load the file to direct table and its working fine. Then I tried to add a sort in between file and target table, its running since last 2 hours but it didn't fail. However while running the original package (with Kimball transformation), its still showing the below message:

[Kimball Method Slowly Changing Dimension] Error: Internal Error (Exception of type 'System.OutOfMemory was thrown.') in ProcessInput sending row to Cache.

Also just to inform, I'm using "Kimball SCD v1.5" and not the latest "Dimension Merge" transformation.

Do you have any suggestions?

Regards,

Ashish

New Post: SSIS out of memory exception

WeeeBob — Mon, 19 Dec 2011 13:23:46 GMT

Hi Ashish,

For troubleshooting I'd remove the Dimension merge SCD component, If you do a straight read from the file and write to a temp table do you still get memory issues? Try then adding a 'sort' component between the source and destination as well and see if that causes memory issue. I've used the SCD component to read much more than 430MB of data at a time with no memory issues, so I suspect you have an SSIS issue more than a component issue.