Status of v1.5

Coordinator
Apr 21, 2009 at 12:47 AM
Good evening,

I have recently completed replacement of my hard drive through an RMA - so the days lost loading the OS, installing and reconfiguring all the requisite junk on my computer are done.

Coding begins...
Coordinator
Jun 6, 2009 at 2:32 PM

I've been exceptionally busy of late (you'd know if you read my blog) but I have found a few minutes to hack away lately.  I was merrily working away on issue #2405 (http://kimballscd.codeplex.com/WorkItem/View.aspx?WorkItemId=2405) when KimYda posted a reply to a thread on the MSDN forums (http://social.msdn.microsoft.com/Forums/en-US/sqlintegrationservices/thread/7eb57ef2-9675-4959-a4dd-f5d0dd7852c7) that pointed me in the right direction towards using the "standard" column mapping controls in SSIS.  Without documentation, it's a mess, obviously - but I couldn't leave it alone.  I believe I've got the hang of using the control - probably in a wrong way, but it appears to work - so issue #188 (http://kimballscd.codeplex.com/WorkItem/View.aspx?WorkItemId=188) is back on the table for v1.5.

So far, I've "fixed up" the first tab to my liking (which was really only stripping out stuff, so that's not saying much), and I'm halfway through the Column Mapping tab...

Coordinator
Jun 16, 2009 at 5:06 PM

Despite spending most of last week at DevTeach in Vancouver (which was great, BTW), I managed to spend some time and finally got the "column mapping" controls to work!  I'll be writing up a blog post on how that's done, and you'll have the code in the next release of this component to play with as well...

Coordinator
Jul 6, 2009 at 5:05 PM

I've knocked off a few of the next items - they were pretty simple (extending types)...

Looks like six things left to mess with, and I'll have code and an alpha verison uploaded for my helpful testers!  I plan to construct a set of regression test packages to run against the component to make sure it's functioning as expected.  I'll likely post those up as some kind of a framework - so that if/when you guys run into problems in the future, you can construct your own "test package" from that template to demonstrate to me either how you want it to operate, and/or how it is broken...

Coordinator
Jul 22, 2009 at 7:20 PM

Getting closer... (yet oh, so slowly...)

I've been testing upgrading from v1.4 packages, and I think that's being handled well now.  I've got a few items left to polish off, of course, but I'll soon be ready to really test the component.  If anyone is willing to put in a little time to help, that would be awesome, and will surely speed things up.  Drop me an email, and I'll send you binaries, source, test packages, and notes before anyone else gets them.

Coordinator
Aug 9, 2009 at 3:13 AM

Okee dokee - the alpha is "up".

I still have a few features to work on - I left the tough ones for last, I think.  I'm hoping that dealing with those "large" types isn't too tough - I'm thinking it'll just be learning how to read them in and out of the buffers.  However, getting side-by-side installation could be tricky... even if it's not very difficult... it will likely mean breaking the "upgrade path" for one set of you - either the 2005 guys or the 2008 guys.  We'll see...

Aug 26, 2009 at 3:54 PM

This tools is exactly what I need.  We had an outsode firm develop our DW, but they did not code for SCDs at all.  They kept historical records but not in a way we can use.  I am in the process of updating the design to track the SCDs.  From looking at the posts on the site and the changes/features in v1.5 I think that is the way I should go.  Do you have a time frame for the beta and/or live release of v1.5?  I am tempted to use the alpha version for testing and development but I am not sure I could go into production with it.

This is a great tool and an asset to the BI/DW community.  Thank you very much!!!

Coordinator
Aug 26, 2009 at 10:36 PM

At this time, I wouldn't recommend using v1.5 for production use - there are definitely stability problems with it.  (Some I've resolved but haven't uploaded yet, some I haven't resolved.)  v1.4 is very stable, however, and unless you need some of the new features in v1.5, I would recommend using that in production instead.

I have no estimate on a beta or "stable" release date yet - I will probably only know that once I see that I've been able to address the feedback I've received so far.  (BTW - thanks to all that have tried v1.5 and provided very valuable feedback on my own sloppy testing.)  One of the main capabilities I'm committing to including is a relatively smooth upgrade experience from v1.4 to v1.5 - there will be some manual adjustments required on upgrade, but very minor.  Unfortunately, this is mostly a "spare time" project for me, and that makes it extremely hard to gauge the time I'll have available...

Hopefully it won't be that long - I'm looking forward to using v1.5 in my environment too!

Coordinator
Aug 29, 2009 at 6:42 PM

I'm getting excited... it's almost ready for beta.  I've just updated the binaries and source for v1.5.

I have a regression testing mechanism mostly in place to make sure that I don't break stuff when adding or altering features.  It should also allow you to provide me with sample data that demonstrates errors or incorrect behaviour - instead of you having to describe them or post up tons of messages.  (I haven't included that test environment in the downloads yet.)  Thank you very much for all the testing support so far!  I'll call out Patrick B., Ronald K., Keith M., Stephen B., Greg G., Joe S., James S., Paul A., Ross S., Sean N., Justin T. for your bug reports and suggestions (last names withheld in case you wanted your privacy!)

I'm hoping it's not too long before I can really finish it off.  Please try out v1.5 and continue the bug reporting...

Coordinator
Sep 1, 2009 at 6:23 AM

Apologies to those who downloaded the binaries... they weren't complete :)

I didn't bundle in both DLLs!  That's been rectified - the ZIPs now include both DLLs, so you should be able to test away.

I have run tests through my regression package, and I've run basic comparative tests through my own production data to ensure similar results to v1.4.  So far, no surprises.

Coordinator
Sep 18, 2009 at 8:40 PM

Almost time for the beta.  I've run through a bunch of static "simple" tests in my regression test harness, and I'm now implementing this in my dev environment with "live" data to spot anything else.  Doing that has brought a few other usability improvements...

It's looking like I won't be able to make the component a side-by-side install unless I break one of the versions (2005 or 2008).  The number of people who've downloaded each is pretty darn even - but 2005 wins.  So if I do decide to go through with it and break 1.4 to 1.5 compatibility - it will probably be for those of us using the 2008 version.  Although I hate to do that (seeing as though I'm also using 2008), doing it now will probably help when SQL 11 comes around... Truth is that I'm still sitting on the fence about doing it at all.

And for those of you that are following this, do me a favour and go vote for a few CodePlex improvements for me.  They'd make my life a little easier, giving me more time to actually do work on the project, rather than on the project site:

* Automatically Annotate Releases Based on Linked Issues: http://codeplex.codeplex.com/WorkItem/View.aspx?WorkItemId=24455

* Add More Properties To Issues - "Applies To Release": http://codeplex.codeplex.com/WorkItem/View.aspx?WorkItemId=24454

I expect to have refreshed the 1.5 release to beta within a week.

Coordinator
Sep 30, 2009 at 4:15 PM

OK - no surprise there - it took longer than a week.  The beta is now up - but continue to be careful.  Please post questions in the Discussions (clearly identifying that you're talking about v1.5beta), create issues for bugs, email those bugs directly to me, or hit me up on Twitter @Todd_McDermid.  I still have work to do verifying the upgrade paths, and side-by-side installation.  I will also be providing a test harness shortly that will allow you to (relatively easily) confirm that the component does what I say it's supposed to do.  It will also be a quick vehicle for you to provide case-specific examples of situations where it's not operating properly, so that I can examine something more concrete than a description of what's going wrong.

I'm still in the process of reworking the Wiki.  The 1.5 Wiki is not accessible from the home page - you can get at it from the v1.5 releases page.  Happy processing!

Coordinator
Nov 16, 2009 at 9:15 PM

Thank you to those who have fed back on v1.5.  There were definitely some issues with the installer - and I'm not a great installer-builder :)

I've posted a refresh to v1.5 where I believe I have everything fixed up with regards to the installer and bugs identified to date.  I would really appreciate those of you bleeding-edge types downloading this new v1.5 and trying it out.  Specifically, if you can provide feedback on these scenarios (after backing up your packages!):

1. Does the installer function correctly?
  a) Did it "replace" your v1.4 installation seamlessly?  Does opening a new project, and dropping your existing toolbox item result in a "working" 1.5 component?
  b) Did the 64-bit installer put the component in BOTH 32- and 64-bit PipelineComponents folders?

2. Does an automatic upgrade of a v1.4 package function?
  a) Describe to me what your packages look like when you open them - i.e. did the automatic upgrade work the way you expected?  What (if anything) was left for you to do manually?
  b) Attempt to run your package with v1.5, and let me know if it provided the same output as a v1.4 run did.

3. Can the component be installed with 2005 and 2008 side-by-side?

4. Have all of the issues you may have reported or experienced with v1.4 or earlier v1.5 builds been addressed?

Please post a new discussion thread with your result(s) - and please specify SQL version and bitness.  Or send an email...

Coordinator
Nov 17, 2009 at 5:02 AM

Just added my test harness solution to the download page.  If you have a scenario that the component seems to be processing incorrectly - mock up some test data to demonstrate the unexpected behaviour and send it to me!  Pictures are worth a thousand words...  See the Test Harness instructions for details.

Coordinator
Jan 16, 2010 at 6:58 AM

I'm still not getting quite the time I'd like to to work on this.  If you've been following my blog, you'll know there is some good news - I just got my DW server provisioned, so I will be spending more time on DW/BI stuff.  Hopefully that rubs off over here.  I believe I've finally put the very intermittent (and infuriating) hanging to bed.  That's why I'm stress testing with a complete DW load with four years of data - that should provide comfort to me that I've finally nailed it.  You'll be the first to know when v1.5 goes gold...

Coordinator
Jan 24, 2010 at 7:50 PM

I've bitten the bullet and released v1.5.  The installer may still be a little rough around the edges - particularly the x64 installer which I haven't yet tested to ensure it places files in exactly the right spots.

Despite that, the stress testing has exposed no issues, and the testing harness has ensured that all the issues identified as fixed are still fixed!

Please watch this space, and my blog for new developments, walkthroughs, and other information about the component.  Enjoy!

Coordinator
Feb 11, 2010 at 5:37 AM

Three flaws have been reported in the component since it's v1.5 release (see discussions and workitems):

* Inferred member handling wasn't perfect

* A messagebox was interfering with full (yet still confusing) error reporting

* The x86 2008 installer wasn't GAC-ing the second DLL properly

Fixes to those have been made (on my system) and will make it up here shortly.

Any MVPs that have used the component and that are attending the Summit next week - please DM me on Twitter or send me an email - I'd like to have a beer with you...

Coordinator
Mar 25, 2010 at 7:16 AM

Fellow SCD processors...

You can't stop progress, and I've been tinkering.  No - nothing you've actually asked for, sorry to say.  I have been messing around with some internals of the component to try and streamline some things, and do finally believe that I've destroyed any possibility of the component hanging - rare as that may have been.  As part of those changes, a new output has made its way on to the component.  That makes it eight outputs now, I think - I hope there's no more coming :).  The new output is a "statistics" output.  At this point, it's a little rough, and I don't expect you to easily understand the crap being pumped out of it unless you've read and understood the code.  But I thought you might want to know it's there so it didn't scare you. 

I'm prepping to upload that very soon - in source and binary forms.  I don't plan to advance the version number... significantly anyway.  Maybe a 1.5.1 or something.  Anyway - I also just discovered a big black hole in the middle of my YouTube overview video.  I'll have to find some time to fix that, and work more on the documentation here.  Don't forget: for short questions - or just to say hello - I'm on Twitter as @Todd_McDermid.  I'd love to hear from you.

Mar 29, 2010 at 2:11 PM

Hi Todd,

I have installed the x86 2005 and x86 2008 installers and they both do not work yet.

The problem occurs in the 7th step of the install instructions:

7. Select the “Data Flow Components” tab, and place a checkmark next to “Kimball Method SCD”.

There is no Kimball Method SCD in the list.

Before I post all the details about the server (version, installed software, files in C:\Program files\Microsoft SQL Server and C:\Windows\Assembly) do you information if and when the flaws as mentioned above in your message of Feb 11 will be fixed in a new release?

Thanks a lot...

Coordinator
Mar 30, 2010 at 6:04 AM

hmi,

The items I identified on Feb 10th were fixed and uploaded to the source and downloads on Feb 11th.  Please post all the details you can about your issue in another discussion.

Coordinator
Mar 30, 2010 at 6:10 AM

I have decided to bite off the intra-day loads issue (2318) - I believe I have that done, but I have to test, test, test.  Thanks to all who sent their own hacks in - mine may or may not work for you :)

Due to that, I'll be starting a new release - v1.6.  Consider this thread closed, and start watching this one if you're interested in v1.6: http://kimballscd.codeplex.com/Thread/View.aspx?ThreadId=207732