Source Columns mapped to existing Dimension Columns

Feb 21, 2010 at 8:12 PM

I get this message "Source System columns are not completely mapped to Existing Dimension columns" when my source columns are not mapped to ALL my existing dimension columns.

In order to get rid of the message I can do one of three things

1. Set the existing dimension columns to "not used" 

2. Not bring those columns into the existing dimension dataset

3. Create the empty columns in the source data set to match

The first two of these choices are problematic because if I create a new SCD2 I need the fields to create a new record and they are not available in the outputs.

Option 3, while it works doesn't seem to be practical.

 

Am I getting something wrong in my thinking or use of the component? 

 

 

Coordinator
Feb 22, 2010 at 10:32 PM

I'm not sure I understand what you're trying to do.  So let me explain what that message means...

The component is basically telling you that the "content" columns (BKs, SCD1 and SCD2 columns) from the Existing Dimension aren't associated with counterparts in the Source System.  This is an essential thing to have set up, because you won't be able to tell if the Source System has changes requiring an SCD1 or 2 type change if you can't compare every column in the Existing Dimension to a value in the Source.

I don't understand what you mean by "a new SCD2"... I can only assume you mean a "new record/row".  In that case - the component does that for you.

I must be missing something...

Feb 22, 2010 at 11:13 PM

Hi Todd, thanks for your time and the great component.

You are understanding the question correctly. 

This does assume that the update for a dimension received from the source system must always have exactly the same columns found in the dimension table. 

In my case the update information for one of my Dimensions arrives in two seperate processes.

1. The first update has the core information that I need to check for SCD2 type changes.

2. The second update only carries SCD1 type information for which I use a simple update and dont need to put it through the process.

 

When I receive the first update I cannot compare ALL columns because I do not have them.

So to make the process work I can select "Not used" on those columns, but then those columns are not available in the output dataset, so when I try and create a new record I do not have the values in those fields.

 

What I am after is to be able to say "Not Used" in the context of "ignore these columns when processing, but make them available in the output data set"

 

If this is still unclear, I can email you the example. 

 

Coordinator
Feb 26, 2010 at 7:30 PM

Interesting - I hadn't thought of that kind of scenario.

I think I understand what you're looking for - and I think I could accomodate that kind of behaviour through workitem 3907: "Type 0 Fixed Attributes".  I'll make a note there to ensure I don't forget about it.

In the meantime... I have a couple (less than great) suggestions...

1. Split up your dimension into two dimensions.  In your dimensional model, it will look like two dimensions, but you can treat it as one conceptually.  This is often done for dimensions that have an excessive amount of columns (purely subjective), or when only a few SCD2 columns cause a lot of version changes.

2. Multicast your dimension flow and join it to the source flow prior to processing to add the SCD1 column values.  If your flows are both sorted by BK (which is a best perf practice with the component) this shouldn't add too much overhead.

Feb 28, 2010 at 7:41 PM

Thanks, I will play around with those suggestions.

 

 

Mar 17, 2010 at 5:09 AM

It would really help troubleshooting if the error message could list the Source System columns that are not mapped, e.g.

Source System columns "Column 1", "Column 2" are not mapped to Existing Dimension columns

as it's not always immediately obvious which columns the error is referring to.

 

Coordinator
Mar 18, 2010 at 9:29 PM
This discussion has been copied to a work item. Click here to go to the work item and continue the discussion.