SSIS Dimension Merge SCD Component

Reopened Feature: Support Multiple Intra-day SCD Processing [2318]

toddmcdermid — Fri, 14 Jun 2013 07:26:40 GMT

Currently, the component permits usage of a date/time typed variable as the effective/expiry time cutoff. This doesn't preclude using the component to load/process the same table multiple times in a day. However, the choices for applying this variable's value to the effective and expiry columns does. The choices effectively limit the component to correct operation only for daily loads.

In order to properly support multiple intra-day processing, a smaller "gap" between the expiry date/time of the "old" version and effective date/time of the "new" version must be allowed.

Possible solutions:
A) Add more choices to the "date interpretation" dropdowns - such as "effective is now, expiry is one millisecond ago".
B) Discard "choices" structure, and simply require two variables be specified that contain the effective and expiry date/times.
C) Change "choices" from a predefined list to a set of arguments to "dateadd"

Reopened Issue: Divide by Zero Error Creating Auditing Output When No Rows Are Processed [4900]

toddmcdermid — Fri, 14 Jun 2013 07:26:40 GMT

When I run a KimballSCD package where there are no rows in the source and no rows in the existing dimension (no rows in any input) SSIS crashes and I get this error:

Error: 0x0 at DF (Risk), SCD: Internal error (Attempted to divide by zero.) in CreateAuditingRow.

This is not a major problem since this is not really a real production scenario. We just hadn't inserted any rows into the production source table for one of our dimensions. Easy to work around, but probably worth fixing, that being said.

I'm using version 1.4.

Reopened Issue: Large String Business Keys [5110]

toddmcdermid — Fri, 14 Jun 2013 07:26:40 GMT

Let me start by saying thank you for such a great component. I'm still in the evaluation/learning stages of using this tool, but I really like what I've used so far.

I'm having difficulty with having more than 2 business keys that are varchar(50). In my test data I started out with 4 varchar(50) fields as my business keys and the component hung in BIDS when I ran it. When I reduced the number of Business Keys to 2, everything worked perfectly. If I added in a third key, things broke again. If I substituted the keys around, they all work just fine as long as I'm only using 2. (Since I'm using test data I'm able to have my keys be consistant to test the scenarios) Is this by design? or have I stumbled on to a legitimate issue? I really appreciate the feedback. In my current DW I have a composite key of 3 fields where this may come up.

In my setup I'm only making 2 SCD1 changes, I have an integer serrogate key, 4 varchar(50) business keys, an SCD2 Effective Date and an SCD2 Expiry Date. Very simple examples for learning. I'm using v1.4 Studio 2005. Thanks in advance for any feedback.

Reopened Issue: Deleted Output Not Updating SCD2 Expiry Column [5196]

toddmcdermid — Fri, 14 Jun 2013 07:26:40 GMT

The Deleted output is not marking the SCD2 Expiry and SCD2 Current record markers with appropriate information, if they exist on the output.

Reopened Issue: Inferred Dimension Members Being Deleted [5282]

toddmcdermid — Fri, 14 Jun 2013 07:26:40 GMT

I have a situation that causes some dimension row to arrive quite late. My fact table load handles these situations by loading an inferred member dimension. I have an inferred member identifier that is indicated in the kscd. The problem I am having is that if the dimension load runs after the inferred member row is created before the actual dimension member arrives, the kscd sends the row to the deleted output because there is no match from the source system. Any ideas on how to handle this?

Reopened Issue: New SCD2 (v1.5) upgrading from 1.4 [5304]

amrit — Fri, 14 Jun 2013 07:26:40 GMT

You seem to always need to use the New SCD2 (v1.5) for updates that are occurring on a Dimension that has both a SCD1 and SCD2 update, otherwise the SCD2 New records do not come through - this was a fix in v1.5.

The NEW SCD2 - Route rows to NEW (behave as 1.4) checkbox - isn't working for SCD1 and SCD2 updates at the same time, or at least this is the bug that is in v1.4 (so still behaving this way) and now fixed in v1.5.

Reopened Feature: Enable SCD 2 Date Handling Effective and Expiry Dates Set to SSIS Variable Values [5322]

luisefigueroa — Fri, 14 Jun 2013 07:26:39 GMT

Current support for explicit date is nice but prefer not hard coding in component.
Propose adding option to set these values from SSIS variables to enable these values to be changed via any SSIS supported way to change variable values such as using variable value property expressions or package configuration. (Example: designer creates RowDefaultStartDate and RowDefaultEndDate variables to use within the component.)

Reopened Issue: V1.5 Editor Hanging on "Finish" [5396]

toddmcdermid — Fri, 14 Jun 2013 07:26:39 GMT

Hi,

Hope this is the right place to raise this.

I've been working with 1.5 (32bit) in SSIS2k8 quite a bit in the last few days, and I am finding it is hanging sporadically on clicking the 'finish' button. Only option is to kill and restart visual studio. I can't determine any particular circumstances as to why it is handing - going through the same steps again works fine.

Reopened Issue: Runtime Hang, Resolved by Enabling Debug Options [5400]

TrevorU — Fri, 14 Jun 2013 07:26:39 GMT

Thanks for this great component. I really love it in V1.5. Great work!

Using V1.5 on a 8-way 64-bit system with SQL Server 2008. The component hangs after processing 0-40000 rows. Turning on all logging options made it stable, and completed my 70K record dimension load.
Playing with the logging options the component seems more stable, and I think I have found a logging configuration that made it stable.

For the component to be stable I have to turn on the logging options:
-Show detailed BK matching information in output window
-Show detailed output information in the output window

There may be other combinations that work too.

Having other logging options turned on, I get this log that indicates to mee there's an infinite loop occuring. Also the 8-way system runs at 50% CPU during this infinite loop state.

Information: 0x0 at Load LGD, Kimball Method Slowly Changing Dimension: ED 0/0(comp), SS 1/9376 - 0 matched
Information: 0x0 at Load LGD, Kimball Method Slowly Changing Dimension: 0 II, 0 UC, 9375 NW, 0 DL, 0 S1, 0 S2E, 0 S2N
Information: 0x0 at Load LGD, Kimball Method Slowly Changing Dimension: Thread 'ProcessKey (16)' (in ProcessKey): Sleeping - no work.
Information: 0x0 at Load LGD, Kimball Method Slowly Changing Dimension: Thread 'ProcessKey (19)' (in ProcessKey): Sleeping - no work.
Information: 0x0 at Load LGD, Kimball Method Slowly Changing Dimension: Thread '' (ProcessInput): Sleeping, waiting for workunits to be consumed.
Information: 0x0 at Load LGD, Kimball Method Slowly Changing Dimension: Thread 'ProcessKey (17)' (in ProcessKey): Sleeping - no work.
Information: 0x0 at Load LGD, Kimball Method Slowly Changing Dimension: Thread 'ProcessKey (23)' (in ProcessKey): Sleeping - no work.
Information: 0x0 at Load LGD, Kimball Method Slowly Changing Dimension: Thread '' (ProcessInput): Sleeping, waiting for workunits to be consumed.
Information: 0x0 at Load LGD, Kimball Method Slowly Changing Dimension: Thread 'ProcessKey (18)' (in ProcessKey): Sleeping - no work.
Information: 0x0 at Load LGD, Kimball Method Slowly Changing Dimension: Thread 'ProcessKey (20)' (in ProcessKey): Sleeping - no work.
Information: 0x0 at Load LGD, Kimball Method Slowly Changing Dimension: Thread 'ProcessKey (22)' (in ProcessKey): Sleeping - no work.
Information: 0x0 at Load LGD, Kimball Method Slowly Changing Dimension: Thread 'ProcessKey (21)' (in ProcessKey): Sleeping - no work.
Information: 0x0 at Load LGD, Kimball Method Slowly Changing Dimension: Thread '' (ProcessInput): Sleeping, waiting for workunits to be consumed.
Information: 0x0 at Load LGD, Kimball Method Slowly Changing Dimension: Thread 'ProcessKey (16)' (in ProcessKey): Sleeping - no work.
Information: 0x0 at Load LGD, Kimball Method Slowly Changing Dimension: Thread 'ProcessKey (19)' (in ProcessKey): Sleeping - no work.
Information: 0x0 at Load LGD, Kimball Method Slowly Changing Dimension: Thread '' (ProcessInput): Sleeping, waiting for workunits to be consumed.
Information: 0x0 at Load LGD, Kimball Method Slowly Changing Dimension: Thread 'ProcessKey (23)' (in ProcessKey): Sleeping - no work.
Information: 0x0 at Load LGD, Kimball Method Slowly Changing Dimension: Thread 'ProcessKey (17)' (in ProcessKey): Sleeping - no work.
Information: 0x0 at Load LGD, Kimball Method Slowly Changing Dimension: Thread 'ProcessKey (18)' (in ProcessKey): Sleeping - no work.
Information: 0x0 at Load LGD, Kimball Method Slowly Changing Dimension: Thread 'ProcessKey (20)' (in ProcessKey): Sleeping - no work.
Information: 0x0 at Load LGD, Kimball Method Slowly Changing Dimension: ED 0/0(comp), SS 1/9376 - 0 matched
Information: 0x0 at Load LGD, Kimball Method Slowly Changing Dimension: 0 II, 0 UC, 9375 NW, 0 DL, 0 S1, 0 S2E, 0 S2N
Information: 0x0 at Load LGD, Kimball Method Slowly Changing Dimension: Thread 'ProcessKey (22)' (in ProcessKey): Sleeping - no work.
Information: 0x0 at Load LGD, Kimball Method Slowly Changing Dimension: Thread '' (ProcessInput): Sleeping, waiting for workunits to be consumed.
Information: 0x0 at Load LGD, Kimball Method Slowly Changing Dimension: Thread 'ProcessKey (21)' (in ProcessKey): Sleeping - no work.
Information: 0x0 at Load LGD, Kimball Method Slowly Changing Dimension: Thread 'ProcessKey (16)' (in ProcessKey): Sleeping - no work.
Information: 0x0 at Load LGD, Kimball Method Slowly Changing Dimension: Thread 'ProcessKey (19)' (in ProcessKey): Sleeping - no work.
Information: 0x0 at Load LGD, Kimball Method Slowly Changing Dimension: Thread '' (ProcessInput): Sleeping, waiting for workunits to be consumed.
Information: 0x0 at Load LGD, Kimball Method Slowly Changing Dimension: Thread 'ProcessKey (23)' (in ProcessKey): Sleeping - no work.
Information: 0x0 at Load LGD, Kimball Method Slowly Changing Dimension: Thread 'ProcessKey (17)' (in ProcessKey): Sleeping - no work.
Information: 0x0 at Load LGD, Kimball Method Slowly Changing Dimension: Thread '' (ProcessInput): Sleeping, waiting for workunits to be consumed.
Information: 0x0 at Load LGD, Kimball Method Slowly Changing Dimension: Thread 'ProcessKey (18)' (in ProcessKey): Sleeping - no work.
Information: 0x0 at Load LGD, Kimball Method Slowly Changing Dimension: Thread 'ProcessKey (20)' (in ProcessKey): Sleeping - no work.
Information: 0x0 at Load LGD, Kimball Method Slowly Changing Dimension: Thread 'ProcessKey (22)' (in ProcessKey): Sleeping - no work.
Information: 0x0 at Load LGD, Kimball Method Slowly Changing Dimension: Thread 'ProcessKey (21)' (in ProcessKey): Sleeping - no work.
Information: 0x0 at Load LGD, Kimball Method Slowly Changing Dimension: Thread 'ProcessKey (16)' (in ProcessKey): Sleeping - no work.
Information: 0x0 at Load LGD, Kimball Method Slowly Changing Dimension: Thread 'ProcessKey (19)' (in ProcessKey): Sleeping - no work.
Information: 0x0 at Load LGD, Kimball Method Slowly Changing Dimension: Thread '' (ProcessInput): Sleeping, waiting for workunits to be consumed.
Information: 0x0 at Load LGD, Kimball Method Slowly Changing Dimension: Thread 'ProcessKey (23)' (in ProcessKey): Sleeping - no work.
Information: 0x0 at Load LGD, Kimball Method Slowly Changing Dimension: Thread '' (ProcessInput): Sleeping, waiting for workunits to be consumed.
Information: 0x0 at Load LGD, Kimball Method Slowly Changing Dimension: Thread 'ProcessKey (17)' (in ProcessKey): Sleeping - no work.

Reopened Feature: Improve Message Regarding Improper Column Mapping [5717]

toddmcdermid — Fri, 14 Jun 2013 07:26:39 GMT

When the mapping between Source System columns and Existing Dimension columns is not completely or correctly configured, the component will report a message similar to: "Source System columns are not completely mapped to Existing Dimension columns."
It would assist troubleshooting if the error message could list the columns that are not mapped, as it's not always immediately obvious which columns the error is referring to.

Reopened Issue: Deleted Output vs. Expired SCD2 Output [6032]

toddmcdermid — Fri, 14 Jun 2013 07:26:39 GMT

Hi Todd!Many thanks for your reply. I tried various ways to get rows in DELETED output definitely. Actually I use you component in the same way as tcash wrote you before http://kimballscd.codeplex.com/Thread/View.aspx?ThreadId=208962 just working with smaller fact tables. It's really extremly fast component!I've two databases named Source and Destination and one table in each database. In Source database

CREATE TABLE [DimensionSrc1](
[ID] [bigint] NOT NULL,
[Field1_varchar] [varchar](20) NULL,
[Field2_money] [money] NULL,
[Field3_forkey_bigint] [bigint] NULL,
CONSTRAINT [PK_Dimention1] PRIMARY KEY CLUSTERED
(
[ID] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY]

and in Destination database

CREATE TABLE [DimensionDW1](
[DimentionKey] [bigint] NOT NULL,
[AlternateKey] [bigint] NULL,
[DateEffective] [datetime] NULL,
[DateExpires] [datetime] NULL,
[Field1_varchar] [varchar](20) NULL,
[Field2_money] [money] NULL,
[Field3_forkey_bigint] [bigint] NULL,
[IsCurrent] [bit] NULL
) ON [PRIMARY]

To fill tables with sample data you can use the following scripts

INSERT INTO [DimensionSrc1]
VALUES (200001,'somevalue1', 10000, 100001),
(200002,'somevalue2', 20000, 100002),
(200003,'somevalue3', 30000, 100003),
(200004,'somevalue4', 40000, 100004),
(200005,'somevalue5', 50000, 100005),
(200007,'somevalue7', 70000, 100007),
(200008,'somevalue8', 80000, 100008),
(200009,'somevalue9', 90000, 100009),
(200010,'somevalue10', 100000, 100010)

INSERT INTO [DimensionDW1](
[DimentionKey],
[AlternateKey],
[Field1_varchar],
[Field2_money],
[Field3_forkey_bigint]
)
VALUES (1,200008,'somevalue8',80000,100008),
(2,200004,'somevalue4',40000,100004),
(3,200010,'somevalue10',100000,100010),
(4,200007,'somevalue7',70000,100007),
(5,200002,'somevalue2',20000,100002),
(6,200006,'somevalue6',60000,100006),
(7,200009,'somevalue9',90000,100009),
(8,200001,'somevalue1',10000,100001),
(9,200003,'somevalue3',30000,100003),
(10,200005,'somevalue5',50000,100005)

As you can see there are 9 rows in source dimension table and 10 rows in destination dimension table. As you wrote above I marked housekeeping fields as "not used". After execution the package I have all 10 rows in UNCHANGED output and no rows in DELETED one.

Thanks,

Alex

Reopened Issue: [KSCDv1.5 SQL2005 x86] Upgrade Validation errors occur when opening a saved SSIS Package with an incomplete KSCD Transformation inside [6222]

tomchen22 — Fri, 14 Jun 2013 07:26:39 GMT

Environment:
Win2K8 SP2,
SQL 2005 (SP3) x86,
Visual Studio 2005 Team Suite sp1
KSCD 1.5

Steps to repro:
1) Create new package
2) Add Dataflow
3) Add KSCD to Dataflow
4) Save Package
5) Close Package
6) Open Package (should now have 3 errors that aren't the ones you'd expect to see)
7) Open Data flow
8) Click on KSCD (error box)

Results from Step 6 - 3 Errors in package
Error 1 Validation error. Data Flow Task: DTS.Pipeline: The component metadata for "component "Kimball Method Slowly Changing Dimension" (1)" could not be upgraded to the newer version of the component. The PerformUpgrade method failed. Package1.dtsx 0 0
Error 2 Error loading Package1.dtsx: System.ApplicationException: Unable to upgrade Existing Dimension column metadata for <?xml version="1.0" encoding="utf-16"?><ArrayOfInputColumnDefinition /> column: Index was outside the bounds of the array. ---> System.IndexOutOfRangeException: Index was outside the bounds of the array. at MouldingAndMillwork.SSIS.KimballMethodSCDConfiguration.PerformUpgrade(Int32 fromVersion, IDTSVariableDispenser90 variableDispenser) --- End of inner exception stack trace --- at MouldingAndMillwork.SSIS.KimballMethodSCDConfiguration.PerformUpgrade(Int32 fromVersion, IDTSVariableDispenser90 variableDispenser) at MouldingAndMillwork.SSIS.KimballMethodSCD.PerformUpgrade(Int32 pipelineVersion) at Microsoft.SqlServer.Dts.Pipeline.ManagedComponentHost.HostCheckAndPerformUpgrade(IDTSManagedComponentWrapper90 wrapper, Int32 lPipelineVersion) E:\Projects\TemplateDesign\TemplateDesign\Package1.dtsx 1 1
Error 3 Error loading Package1.dtsx: The component metadata for "component "Kimball Method Slowly Changing Dimension" (1)" could not be upgraded to the newer version of the component. The PerformUpgrade method failed. E:\Projects\TemplateDesign\TemplateDesign\Package1.dtsx 1 1

Results from Step 8 - Error Message Box
KSCD fails validation due to an upgrade error :

TITLE: Microsoft Visual Studio
------------------------------

Component with name "Kimball Method Slowly Changing Dimension" and class ID "{BF01D463-7089-41EE-8F05-0A6DC17CE633}" could not be created because an error was encountered during its upgrade to the current version.
Contact information:

For help, click: http://go.microsoft.com/fwlink?ProdName=Microsoft%u00ae+Visual+Studio%u00ae+2005&ProdVer=8.0.50727.867&EvtSrc=Microsoft.DataTransformationServices.Design.SR&EvtID=ComponentUpgradeFailed&LinkId=20476

------------------------------
BUTTONS:

OK
------------------------------

Reopened Issue: Status Output is Incorrect [6318]

toddmcdermid — Fri, 14 Jun 2013 07:26:38 GMT

The output status message is not putting the correct value for deleted records. It is using the counter for new records.

Line 4994 in the PostExecute Method:
SSISUtility.Messaging.PostInformation(ComponentMetaData, "Input rows processed: "
+ this._existingDimensionInputCache.CountRows.ToString() + " Existing Dimension, "
+ this._specialMembersInputCache.CountRows.ToString() + " Special Members, "
+ this._sourceSystemInputCache.CountRows.ToString() + " Source System");
SSISUtility.Messaging.PostInformation(ComponentMetaData, "Rows output: "
+ this._counterOutputUnchanged.ToString() + " Unchanged, "
+ this._counterOutputNew.ToString() + " New, "
+ this._counterOutputNew.ToString() + " Deleted, "
+ this._counterOutputSCD1Update.ToString() + " SCD1, "
+ this._counterOutputSCD2Expired.ToString() + " SCD2 Expired, "
+ this._counterOutputSCD2New.ToString() + " SCD2 New, "
+ this._counterOutputInvalid.ToString() + " Invalid Input");

Reopened Issue: DT_DBTIME2 support [6621]

toddmcdermid — Fri, 14 Jun 2013 07:26:38 GMT

The KSCD seems to support the new time data type (in sql server 2008) when in design mode, however when you run the package the component errors.

[MSC_SCD ClaimDetail] Error: Internal error (Unsupported Data Type for comparison: DT_DBTIME2) in ProcessKey, detecting SCD1 and 2 changes in column TimeOfLoss, key: ffffffffffffffff.

Looking at your code the method CompareFields doesn't cater for DT_DBTIME2. Is this by design?

Thanks,

Alex

Reopened Issue: KimballSCD 1.5 component hangs when using a derived column in the update scd1 output [6745]

toddmcdermid — Fri, 14 Jun 2013 07:26:38 GMT

I have a relative simple configuration. There are two inputs and two outputs, one for new rows and one for updated SCD1 rows. The component hangs after adding a derived column (for additional auditing) to the SCD1 update output. After adding the derived column, reattach the dataflows and open the kimballscd component and close it (without making a change) the component hangs and i have to kill the devenv.exe process

I am using the x86 SQL2008 component on WinXpSP3

Ps: I need to use the derived column because of lack of the feature described in workitem "Add Support for SCD Type 0 (Fixed) Attributes

Reopened Issue: Audit Tab Not Saving -1 As Default Value for New Rows [7030]

toddmcdermid — Fri, 14 Jun 2013 07:26:38 GMT

See discussion: http://kimballscd.codeplex.com/Thread/View.aspx?ThreadId=225047

New Post: New Data cannot Insert

AnakSimple — Wed, 22 May 2013 01:55:28 GMT

I have A as source data and B as destination data. I already setting New output to OLEDB Destination. When I running that package i got A have 10 rows and B have 8. Unfortunately on 'New' Line to OLEDB Destination I didn't see that 2 new rows and B didn't get that new rows.
I didn't found any error. I set Id to be Business Key and already sort by Id.
What I am missing ?

Please your advice

Thanks

Closed Feature: Support Multiple Intra-day SCD Processing [2318]

toddmcdermid — Thu, 16 May 2013 10:52:00 GMT

Closed Issue: Divide by Zero Error Creating Auditing Output When No Rows Are Processed [4900]

toddmcdermid — Thu, 16 May 2013 10:51:55 GMT

Closed Issue: Large String Business Keys [5110]

toddmcdermid — Thu, 16 May 2013 10:51:55 GMT