Pass Through Columns

Topics: Possible Bugs/Missing Features?
Jul 30, 2010 at 7:34 PM

When testing this component, I was not able to let certain columns (related to my own audit architecture) pass through. Am I missing something? Is this supported? If not, are there any plans for this?



Jul 31, 2010 at 3:04 AM

There is not currently any "pass through" plan for the component.  Without knowing how your audit architecture works, I can only speculate that your best alternative at this point is to mark those columns as SCD1 types.  The closest issue to your request is 3907 - similar to what's in the SCD Wizard.  If that isn't what you want, please add an issue and describe how such a "Pass Through" column would work.  Specifically, since this column exists on your existing dimension table and your source system - if you found a match (by BK) on two rows between those two systems, which source value would be output?

Nov 23, 2010 at 1:46 PM

Yep, I can see a use for these - on new rows, the column values come from the source data - on existing rows, the values would come from the target dimension

Nov 23, 2010 at 4:30 PM

And those values would NEVER change?  Can you give me a business scenario where that would occur?  I'd like to understand the business case...

Mar 3, 2011 at 11:10 AM

A business scenario for this:

Our source system is Microsoft Dynamics AX where you pretty often have consistency problems in data. One transactional example could be ledger budget (sourcing from Excel import) with values for accounts which do not exist. One dimensional example could be an item (sourcing from a faulty template) with NULL or invalid value for itemgroup.

The first thing would be to send these rows to invalid input (there is another discussion for this) and the next thing would be to log the invalid rows with as much additional information to pinpoint the problem. In the latter example the source input would contain an ItemGroupNumber which should be a passthrough column (the existing input only contains ID’s/surrogate keys). Before reaching the KSCD step the lookup on ItemGroup would return NULL for ItemGroupID. Since ItemGroupID is not allowed as NULL the row is output as Invalid. If I had the ItemGroupNum come through as passthrough I could easily log this as “Item xxx rejected due to invalid item group usage; value nnn”. Today I can only log “Item xxx rejected due to invalid item group usage.”

Anyhow, I need correct handling of invalid rows a lot more than passthrough, even though they are related in my work.