DimensionMergeSCD Issue Tracker Rss Feed

Closed Feature: Support Multiple Intra-day SCD Processing [2318]

toddmcdermid — Thu, 16 May 2013 10:52:00 GMT

Currently, the component permits usage of a date/time typed variable as the effective/expiry time cutoff. This doesn't preclude using the component to load/process the same table multiple times in a day. However, the choices for applying this variable's value to the effective and expiry columns does. The choices effectively limit the component to correct operation only for daily loads.

In order to properly support multiple intra-day processing, a smaller "gap" between the expiry date/time of the "old" version and effective date/time of the "new" version must be allowed.

Possible solutions:
A) Add more choices to the "date interpretation" dropdowns - such as "effective is now, expiry is one millisecond ago".
B) Discard "choices" structure, and simply require two variables be specified that contain the effective and expiry date/times.
C) Change "choices" from a predefined list to a set of arguments to "dateadd"

Closed Issue: Divide by Zero Error Creating Auditing Output When No Rows Are Processed [4900]

toddmcdermid — Thu, 16 May 2013 10:51:55 GMT

When I run a KimballSCD package where there are no rows in the source and no rows in the existing dimension (no rows in any input) SSIS crashes and I get this error:

Error: 0x0 at DF (Risk), SCD: Internal error (Attempted to divide by zero.) in CreateAuditingRow.

This is not a major problem since this is not really a real production scenario. We just hadn't inserted any rows into the production source table for one of our dimensions. Easy to work around, but probably worth fixing, that being said.

I'm using version 1.4.

Closed Issue: Large String Business Keys [5110]

toddmcdermid — Thu, 16 May 2013 10:51:55 GMT

Let me start by saying thank you for such a great component. I'm still in the evaluation/learning stages of using this tool, but I really like what I've used so far.

I'm having difficulty with having more than 2 business keys that are varchar(50). In my test data I started out with 4 varchar(50) fields as my business keys and the component hung in BIDS when I ran it. When I reduced the number of Business Keys to 2, everything worked perfectly. If I added in a third key, things broke again. If I substituted the keys around, they all work just fine as long as I'm only using 2. (Since I'm using test data I'm able to have my keys be consistant to test the scenarios) Is this by design? or have I stumbled on to a legitimate issue? I really appreciate the feedback. In my current DW I have a composite key of 3 fields where this may come up.

In my setup I'm only making 2 SCD1 changes, I have an integer serrogate key, 4 varchar(50) business keys, an SCD2 Effective Date and an SCD2 Expiry Date. Very simple examples for learning. I'm using v1.4 Studio 2005. Thanks in advance for any feedback.

Closed Issue: Deleted Output Not Updating SCD2 Expiry Column [5196]

toddmcdermid — Thu, 16 May 2013 10:51:55 GMT

The Deleted output is not marking the SCD2 Expiry and SCD2 Current record markers with appropriate information, if they exist on the output.

Closed Issue: Inferred Dimension Members Being Deleted [5282]

toddmcdermid — Thu, 16 May 2013 10:51:54 GMT

I have a situation that causes some dimension row to arrive quite late. My fact table load handles these situations by loading an inferred member dimension. I have an inferred member identifier that is indicated in the kscd. The problem I am having is that if the dimension load runs after the inferred member row is created before the actual dimension member arrives, the kscd sends the row to the deleted output because there is no match from the source system. Any ideas on how to handle this?

Closed Issue: New SCD2 (v1.5) upgrading from 1.4 [5304]

amrit — Thu, 16 May 2013 10:51:54 GMT

You seem to always need to use the New SCD2 (v1.5) for updates that are occurring on a Dimension that has both a SCD1 and SCD2 update, otherwise the SCD2 New records do not come through - this was a fix in v1.5.

The NEW SCD2 - Route rows to NEW (behave as 1.4) checkbox - isn't working for SCD1 and SCD2 updates at the same time, or at least this is the bug that is in v1.4 (so still behaving this way) and now fixed in v1.5.

Closed Feature: Enable SCD 2 Date Handling Effective and Expiry Dates Set to SSIS Variable Values [5322]

luisefigueroa — Thu, 16 May 2013 10:51:54 GMT

Current support for explicit date is nice but prefer not hard coding in component.
Propose adding option to set these values from SSIS variables to enable these values to be changed via any SSIS supported way to change variable values such as using variable value property expressions or package configuration. (Example: designer creates RowDefaultStartDate and RowDefaultEndDate variables to use within the component.)

Closed Issue: V1.5 Editor Hanging on "Finish" [5396]

toddmcdermid — Thu, 16 May 2013 10:51:54 GMT

Hi,

Hope this is the right place to raise this.

I've been working with 1.5 (32bit) in SSIS2k8 quite a bit in the last few days, and I am finding it is hanging sporadically on clicking the 'finish' button. Only option is to kill and restart visual studio. I can't determine any particular circumstances as to why it is handing - going through the same steps again works fine.

Closed Issue: Runtime Hang, Resolved by Enabling Debug Options [5400]

TrevorU — Thu, 16 May 2013 10:51:53 GMT

Thanks for this great component. I really love it in V1.5. Great work!

Using V1.5 on a 8-way 64-bit system with SQL Server 2008. The component hangs after processing 0-40000 rows. Turning on all logging options made it stable, and completed my 70K record dimension load.
Playing with the logging options the component seems more stable, and I think I have found a logging configuration that made it stable.

For the component to be stable I have to turn on the logging options:
-Show detailed BK matching information in output window
-Show detailed output information in the output window

There may be other combinations that work too.

Having other logging options turned on, I get this log that indicates to mee there's an infinite loop occuring. Also the 8-way system runs at 50% CPU during this infinite loop state.

Information: 0x0 at Load LGD, Kimball Method Slowly Changing Dimension: ED 0/0(comp), SS 1/9376 - 0 matched
Information: 0x0 at Load LGD, Kimball Method Slowly Changing Dimension: 0 II, 0 UC, 9375 NW, 0 DL, 0 S1, 0 S2E, 0 S2N
Information: 0x0 at Load LGD, Kimball Method Slowly Changing Dimension: Thread 'ProcessKey (16)' (in ProcessKey): Sleeping - no work.
Information: 0x0 at Load LGD, Kimball Method Slowly Changing Dimension: Thread 'ProcessKey (19)' (in ProcessKey): Sleeping - no work.
Information: 0x0 at Load LGD, Kimball Method Slowly Changing Dimension: Thread '' (ProcessInput): Sleeping, waiting for workunits to be consumed.
Information: 0x0 at Load LGD, Kimball Method Slowly Changing Dimension: Thread 'ProcessKey (17)' (in ProcessKey): Sleeping - no work.
Information: 0x0 at Load LGD, Kimball Method Slowly Changing Dimension: Thread 'ProcessKey (23)' (in ProcessKey): Sleeping - no work.
Information: 0x0 at Load LGD, Kimball Method Slowly Changing Dimension: Thread '' (ProcessInput): Sleeping, waiting for workunits to be consumed.
Information: 0x0 at Load LGD, Kimball Method Slowly Changing Dimension: Thread 'ProcessKey (18)' (in ProcessKey): Sleeping - no work.
Information: 0x0 at Load LGD, Kimball Method Slowly Changing Dimension: Thread 'ProcessKey (20)' (in ProcessKey): Sleeping - no work.
Information: 0x0 at Load LGD, Kimball Method Slowly Changing Dimension: Thread 'ProcessKey (22)' (in ProcessKey): Sleeping - no work.
Information: 0x0 at Load LGD, Kimball Method Slowly Changing Dimension: Thread 'ProcessKey (21)' (in ProcessKey): Sleeping - no work.
Information: 0x0 at Load LGD, Kimball Method Slowly Changing Dimension: Thread '' (ProcessInput): Sleeping, waiting for workunits to be consumed.
Information: 0x0 at Load LGD, Kimball Method Slowly Changing Dimension: Thread 'ProcessKey (16)' (in ProcessKey): Sleeping - no work.
Information: 0x0 at Load LGD, Kimball Method Slowly Changing Dimension: Thread 'ProcessKey (19)' (in ProcessKey): Sleeping - no work.
Information: 0x0 at Load LGD, Kimball Method Slowly Changing Dimension: Thread '' (ProcessInput): Sleeping, waiting for workunits to be consumed.
Information: 0x0 at Load LGD, Kimball Method Slowly Changing Dimension: Thread 'ProcessKey (23)' (in ProcessKey): Sleeping - no work.
Information: 0x0 at Load LGD, Kimball Method Slowly Changing Dimension: Thread 'ProcessKey (17)' (in ProcessKey): Sleeping - no work.
Information: 0x0 at Load LGD, Kimball Method Slowly Changing Dimension: Thread 'ProcessKey (18)' (in ProcessKey): Sleeping - no work.
Information: 0x0 at Load LGD, Kimball Method Slowly Changing Dimension: Thread 'ProcessKey (20)' (in ProcessKey): Sleeping - no work.
Information: 0x0 at Load LGD, Kimball Method Slowly Changing Dimension: ED 0/0(comp), SS 1/9376 - 0 matched
Information: 0x0 at Load LGD, Kimball Method Slowly Changing Dimension: 0 II, 0 UC, 9375 NW, 0 DL, 0 S1, 0 S2E, 0 S2N
Information: 0x0 at Load LGD, Kimball Method Slowly Changing Dimension: Thread 'ProcessKey (22)' (in ProcessKey): Sleeping - no work.
Information: 0x0 at Load LGD, Kimball Method Slowly Changing Dimension: Thread '' (ProcessInput): Sleeping, waiting for workunits to be consumed.
Information: 0x0 at Load LGD, Kimball Method Slowly Changing Dimension: Thread 'ProcessKey (21)' (in ProcessKey): Sleeping - no work.
Information: 0x0 at Load LGD, Kimball Method Slowly Changing Dimension: Thread 'ProcessKey (16)' (in ProcessKey): Sleeping - no work.
Information: 0x0 at Load LGD, Kimball Method Slowly Changing Dimension: Thread 'ProcessKey (19)' (in ProcessKey): Sleeping - no work.
Information: 0x0 at Load LGD, Kimball Method Slowly Changing Dimension: Thread '' (ProcessInput): Sleeping, waiting for workunits to be consumed.
Information: 0x0 at Load LGD, Kimball Method Slowly Changing Dimension: Thread 'ProcessKey (23)' (in ProcessKey): Sleeping - no work.
Information: 0x0 at Load LGD, Kimball Method Slowly Changing Dimension: Thread 'ProcessKey (17)' (in ProcessKey): Sleeping - no work.
Information: 0x0 at Load LGD, Kimball Method Slowly Changing Dimension: Thread '' (ProcessInput): Sleeping, waiting for workunits to be consumed.
Information: 0x0 at Load LGD, Kimball Method Slowly Changing Dimension: Thread 'ProcessKey (18)' (in ProcessKey): Sleeping - no work.
Information: 0x0 at Load LGD, Kimball Method Slowly Changing Dimension: Thread 'ProcessKey (20)' (in ProcessKey): Sleeping - no work.
Information: 0x0 at Load LGD, Kimball Method Slowly Changing Dimension: Thread 'ProcessKey (22)' (in ProcessKey): Sleeping - no work.
Information: 0x0 at Load LGD, Kimball Method Slowly Changing Dimension: Thread 'ProcessKey (21)' (in ProcessKey): Sleeping - no work.
Information: 0x0 at Load LGD, Kimball Method Slowly Changing Dimension: Thread 'ProcessKey (16)' (in ProcessKey): Sleeping - no work.
Information: 0x0 at Load LGD, Kimball Method Slowly Changing Dimension: Thread 'ProcessKey (19)' (in ProcessKey): Sleeping - no work.
Information: 0x0 at Load LGD, Kimball Method Slowly Changing Dimension: Thread '' (ProcessInput): Sleeping, waiting for workunits to be consumed.
Information: 0x0 at Load LGD, Kimball Method Slowly Changing Dimension: Thread 'ProcessKey (23)' (in ProcessKey): Sleeping - no work.
Information: 0x0 at Load LGD, Kimball Method Slowly Changing Dimension: Thread '' (ProcessInput): Sleeping, waiting for workunits to be consumed.
Information: 0x0 at Load LGD, Kimball Method Slowly Changing Dimension: Thread 'ProcessKey (17)' (in ProcessKey): Sleeping - no work.

Closed Feature: Improve Message Regarding Improper Column Mapping [5717]

toddmcdermid — Thu, 16 May 2013 10:51:53 GMT

When the mapping between Source System columns and Existing Dimension columns is not completely or correctly configured, the component will report a message similar to: "Source System columns are not completely mapped to Existing Dimension columns."
It would assist troubleshooting if the error message could list the columns that are not mapped, as it's not always immediately obvious which columns the error is referring to.

Closed Issue: Deleted Output vs. Expired SCD2 Output [6032]

toddmcdermid — Thu, 16 May 2013 10:51:53 GMT

Hi Todd!Many thanks for your reply. I tried various ways to get rows in DELETED output definitely. Actually I use you component in the same way as tcash wrote you before http://kimballscd.codeplex.com/Thread/View.aspx?ThreadId=208962 just working with smaller fact tables. It's really extremly fast component!I've two databases named Source and Destination and one table in each database. In Source database

CREATE TABLE [DimensionSrc1](
[ID] [bigint] NOT NULL,
[Field1_varchar] [varchar](20) NULL,
[Field2_money] [money] NULL,
[Field3_forkey_bigint] [bigint] NULL,
CONSTRAINT [PK_Dimention1] PRIMARY KEY CLUSTERED
(
[ID] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY]

and in Destination database

CREATE TABLE [DimensionDW1](
[DimentionKey] [bigint] NOT NULL,
[AlternateKey] [bigint] NULL,
[DateEffective] [datetime] NULL,
[DateExpires] [datetime] NULL,
[Field1_varchar] [varchar](20) NULL,
[Field2_money] [money] NULL,
[Field3_forkey_bigint] [bigint] NULL,
[IsCurrent] [bit] NULL
) ON [PRIMARY]

To fill tables with sample data you can use the following scripts

INSERT INTO [DimensionSrc1]
VALUES (200001,'somevalue1', 10000, 100001),
(200002,'somevalue2', 20000, 100002),
(200003,'somevalue3', 30000, 100003),
(200004,'somevalue4', 40000, 100004),
(200005,'somevalue5', 50000, 100005),
(200007,'somevalue7', 70000, 100007),
(200008,'somevalue8', 80000, 100008),
(200009,'somevalue9', 90000, 100009),
(200010,'somevalue10', 100000, 100010)

INSERT INTO [DimensionDW1](
[DimentionKey],
[AlternateKey],
[Field1_varchar],
[Field2_money],
[Field3_forkey_bigint]
)
VALUES (1,200008,'somevalue8',80000,100008),
(2,200004,'somevalue4',40000,100004),
(3,200010,'somevalue10',100000,100010),
(4,200007,'somevalue7',70000,100007),
(5,200002,'somevalue2',20000,100002),
(6,200006,'somevalue6',60000,100006),
(7,200009,'somevalue9',90000,100009),
(8,200001,'somevalue1',10000,100001),
(9,200003,'somevalue3',30000,100003),
(10,200005,'somevalue5',50000,100005)

As you can see there are 9 rows in source dimension table and 10 rows in destination dimension table. As you wrote above I marked housekeeping fields as "not used". After execution the package I have all 10 rows in UNCHANGED output and no rows in DELETED one.

Thanks,

Alex

Closed Issue: [KSCDv1.5 SQL2005 x86] Upgrade Validation errors occur when opening a saved SSIS Package with an incomplete KSCD Transformation inside [6222]

tomchen22 — Thu, 16 May 2013 10:51:53 GMT

Environment:
Win2K8 SP2,
SQL 2005 (SP3) x86,
Visual Studio 2005 Team Suite sp1
KSCD 1.5

Steps to repro:
1) Create new package
2) Add Dataflow
3) Add KSCD to Dataflow
4) Save Package
5) Close Package
6) Open Package (should now have 3 errors that aren't the ones you'd expect to see)
7) Open Data flow
8) Click on KSCD (error box)

Results from Step 6 - 3 Errors in package
Error 1 Validation error. Data Flow Task: DTS.Pipeline: The component metadata for "component "Kimball Method Slowly Changing Dimension" (1)" could not be upgraded to the newer version of the component. The PerformUpgrade method failed. Package1.dtsx 0 0
Error 2 Error loading Package1.dtsx: System.ApplicationException: Unable to upgrade Existing Dimension column metadata for <?xml version="1.0" encoding="utf-16"?><ArrayOfInputColumnDefinition /> column: Index was outside the bounds of the array. ---> System.IndexOutOfRangeException: Index was outside the bounds of the array. at MouldingAndMillwork.SSIS.KimballMethodSCDConfiguration.PerformUpgrade(Int32 fromVersion, IDTSVariableDispenser90 variableDispenser) --- End of inner exception stack trace --- at MouldingAndMillwork.SSIS.KimballMethodSCDConfiguration.PerformUpgrade(Int32 fromVersion, IDTSVariableDispenser90 variableDispenser) at MouldingAndMillwork.SSIS.KimballMethodSCD.PerformUpgrade(Int32 pipelineVersion) at Microsoft.SqlServer.Dts.Pipeline.ManagedComponentHost.HostCheckAndPerformUpgrade(IDTSManagedComponentWrapper90 wrapper, Int32 lPipelineVersion) E:\Projects\TemplateDesign\TemplateDesign\Package1.dtsx 1 1
Error 3 Error loading Package1.dtsx: The component metadata for "component "Kimball Method Slowly Changing Dimension" (1)" could not be upgraded to the newer version of the component. The PerformUpgrade method failed. E:\Projects\TemplateDesign\TemplateDesign\Package1.dtsx 1 1

Results from Step 8 - Error Message Box
KSCD fails validation due to an upgrade error :

TITLE: Microsoft Visual Studio
------------------------------

Component with name "Kimball Method Slowly Changing Dimension" and class ID "{BF01D463-7089-41EE-8F05-0A6DC17CE633}" could not be created because an error was encountered during its upgrade to the current version.
Contact information:

For help, click: http://go.microsoft.com/fwlink?ProdName=Microsoft%u00ae+Visual+Studio%u00ae+2005&ProdVer=8.0.50727.867&EvtSrc=Microsoft.DataTransformationServices.Design.SR&EvtID=ComponentUpgradeFailed&LinkId=20476

------------------------------
BUTTONS:

OK
------------------------------

Closed Issue: Status Output is Incorrect [6318]

toddmcdermid — Thu, 16 May 2013 10:51:52 GMT

The output status message is not putting the correct value for deleted records. It is using the counter for new records.

Line 4994 in the PostExecute Method:
SSISUtility.Messaging.PostInformation(ComponentMetaData, "Input rows processed: "
+ this._existingDimensionInputCache.CountRows.ToString() + " Existing Dimension, "
+ this._specialMembersInputCache.CountRows.ToString() + " Special Members, "
+ this._sourceSystemInputCache.CountRows.ToString() + " Source System");
SSISUtility.Messaging.PostInformation(ComponentMetaData, "Rows output: "
+ this._counterOutputUnchanged.ToString() + " Unchanged, "
+ this._counterOutputNew.ToString() + " New, "
+ this._counterOutputNew.ToString() + " Deleted, "
+ this._counterOutputSCD1Update.ToString() + " SCD1, "
+ this._counterOutputSCD2Expired.ToString() + " SCD2 Expired, "
+ this._counterOutputSCD2New.ToString() + " SCD2 New, "
+ this._counterOutputInvalid.ToString() + " Invalid Input");

Closed Issue: DT_DBTIME2 support [6621]

toddmcdermid — Thu, 16 May 2013 10:51:52 GMT

The KSCD seems to support the new time data type (in sql server 2008) when in design mode, however when you run the package the component errors.

[MSC_SCD ClaimDetail] Error: Internal error (Unsupported Data Type for comparison: DT_DBTIME2) in ProcessKey, detecting SCD1 and 2 changes in column TimeOfLoss, key: ffffffffffffffff.

Looking at your code the method CompareFields doesn't cater for DT_DBTIME2. Is this by design?

Thanks,

Alex

Closed Issue: KimballSCD 1.5 component hangs when using a derived column in the update scd1 output [6745]

toddmcdermid — Thu, 16 May 2013 10:51:51 GMT

I have a relative simple configuration. There are two inputs and two outputs, one for new rows and one for updated SCD1 rows. The component hangs after adding a derived column (for additional auditing) to the SCD1 update output. After adding the derived column, reattach the dataflows and open the kimballscd component and close it (without making a change) the component hangs and i have to kill the devenv.exe process

I am using the x86 SQL2008 component on WinXpSP3

Ps: I need to use the derived column because of lack of the feature described in workitem "Add Support for SCD Type 0 (Fixed) Attributes

Closed Issue: Audit Tab Not Saving -1 As Default Value for New Rows [7030]

toddmcdermid — Thu, 16 May 2013 10:51:51 GMT

See discussion: http://kimballscd.codeplex.com/Thread/View.aspx?ThreadId=225047

Commented Issue: Merge Component throwing internal error arithmetic overflow [8803]

jakubk — Fri, 10 May 2013 02:24:42 GMT

Hi,
I've migrated a set of packages to a new server that use v1.5 of the SSIS Dimension Merge SCD Component. However, these packages now throw the following error when executed:

Error: 2011-05-12 08:46:00.37
Code: 0x00000000
Source: Kimball SDC DimFund Kimball Method Slowly Changing Dimension
Description: Internal error (Arithmetic operation resulted in an overflow.) in ProcessInput determining thread count.
End Error

They had previously worked without error on two different environments. Has this issues been raised before or is it a known issue?

I'm running on Windows Server R2 Enterprise with .NET 3.5 and 4 installed if that helps.

Regards,
Kevin
Comments: ** Comment from web user: jakubk **

We just deployed a complex solution that utilised this component onto the production system and were greeted with hanging packages and this error.

Is there still no fix?

I'll manually set everything to 16 I think as an interim solution and then rewrite all the packages to use a merge join + checksum instead

Created Issue: Row deleted from source is identified but not redirected to Deleted output [11673]

SushantA — Fri, 29 Mar 2013 09:55:58 GMT

I am using KSCD version 1.5
__Scenario:__ A row is deleted from source and in logs its identified that this rows exist only in existing dimension but while rows output it redirected that row in unchanged output instead of deleted output.

__Logs:__
[Kimball Method Slowly Changing Dimension] Information: Thread 'MatchKeys (6)' (in MatchKeys): Key '6b250b009e120000' from Existing_Dimension is DEFINITELY NOT in Source_System
[Kimball Method Slowly Changing Dimension] Information: Thread 'MatchKeys (6)' (in MatchKeys): Key '6b250b009e120000' queued up
[Kimball Method Slowly Changing Dimension] Information: Thread 'ProcessKey (7)' (in ProcessKey): Processing key '6b250b009e120000'
[Kimball Method Slowly Changing Dimension] Information: Thread 'ProcessKey (7)' (in ProcessKey): Key '6b250b009e120000' only found on Existing Dimension
[Kimball Method Slowly Changing Dimension] Information: Input rows processed: 2 Existing Dimension, 0 Special Members, 1 Source System
[Kimball Method Slowly Changing Dimension] Information: Rows output: 2 Unchanged, 0 New, 0 Deleted, 0 SCD1, 0 SCD2 Expired, 0 SCD2 New, 0 Invalid Input

Please revert in case you need more info.

Commented Issue: SQL Server 2012 update [10604]

dro456 — Wed, 27 Mar 2013 21:46:34 GMT

I'm looking for the SQL Server 2012 SSIS Dimension Merge SCD Component.

Thanks in advance for any help.
Comments: ** Comment from web user: dro456 **

Commented Issue: SQL Server 2012 update [10604]

dro456 — Wed, 27 Mar 2013 21:46:11 GMT

I'm looking for the SQL Server 2012 SSIS Dimension Merge SCD Component.

Thanks in advance for any help.
Comments: ** Comment from web user: dro456 **

It's almost been a year since the last post... any updates?