Image Image Image Image Image Image Image Image Image

Are we, the Informatica community of Developers, Architects, Planners and Hub Administrators, ready to make the Factory model of Data Integration work?1

02 Nov

By

No Comments

Are we, the Informatica community of Developers, Architects, Planners and Hub Administrators, ready to make the Factory model of Data Integration work?

November 2, 2011 | By | No Comments

John Schmidt and David Lyle have successfully made the case for the Factory method as the key Data Integration model for this decade in their book ‘Lean Integration: An Integrated Factory approach to Business Agility’ and it is clear that Informatica have firmly aligned their methodology with this approach. So what about the wider Informatica community? Are we ready to make the step to the Factory model? Well yes, I believe we are, but first we need to consider how our mindset and working practices need to change. Every one of the nine principles of Toyota’s Lean model, which lies behind the Factory model, challenges the working practices of the average Data Integration team and asks what they can change to make things better. Take #7, for example, the ‘Visual Management’ principle, which states that system operations should be transparent so that everyone can see and understand what is happening. Is this common practice in the average Data Integration environment? Sadly no. Until now much is left un-monitored until system failures or schedule overruns occur.  Lets consider Principle #6, ‘Go See’; this probably falls into the same category for the same underlying reason of poor information flow. Many of the other Principles, such as #3 ‘Value Streams’ demand a constant striving for improvement to existing information flow processes that would challenge any hard-pressed Data Integration team.

 

So how do we move forward? Well, thankfully the world hasn’t stood still. With these new challenges have come new enabling technologies. New tools have emerged that enable the Factory aims of continuous improvement. Informatica’s Application ILM Solution is a major step forward. So too are the new generation of Informatica-specific quality and performance visual monitoring tools that vitalise the publishing of real-time metrics to Informatica development teams, support staff and Hub administrators. By putting the tools and the targets together then we produce the kinds of automated feedback loops that enable information to be shared, problems to be anticipated and process improvements to be achieved.  Rather than wait for failure you can proactively seek the improvements and address the problem before it happens, just as Lean challenges you to.

 

HIPPO is the first tool of this new generation of applications to bring an Informatica focus to the monitoring of entire Data Integration processes, environments and resource usage using data visualisation techniques. More tools are emerging which focus on code quality and testing each sharing the same principle: unless we measure then we cannot improve and unless we measure constantly then continuous improvement is not possible. If you are considering implementing a Factory model then include HIPPO’s performance monitoring and trending strengths to reach your goals of controlling costs and gaining maximum value from your Data Integration environment.

02 Nov

By

No Comments

It’s time to face facts; PowerCenter is now so good that trying to make a mapping go faster by performance tuning the mapping alone isn’t worth it.

November 2, 2011 | By | No Comments

OK, if a mapping is poorly written to begin with then it needs to be fixed but that’s not performance tuning: that’s correcting what should have been right first time around. So since Big Data is bound to mean bigger performance challenges, just what DO you do when all of the obvious gotchas are eliminated and the mapping is still not fast enough?
That’s the issue we faced recently. We had squeezed every last drop of inefficiency out of the code but it still wasn’t enough to meet the SLAs on the biggest CRM implementation, at the time, in Europe. We had to look outside Informatica but where should we begin? Pointing the finger at Network or Database or Storage performance is an unpredictable and dangerous sport, never more so when a huge amount of money has already been spent on the infrastructure, as it had here. And anyway, who’s to say it’s any one thing: it could be a combination of factors? And the infrastructure teams often start from the principle that performance issues are the application’s fault, right? The very least they will do is to question our competence to question them. But there was nowhere else to go: we needed to find a way to present evidence, not opinions, that performance might be improved. Otherwise it was goodbye to our SLAs.

We needed something that spoke ‘both languages’: a trusted performance monitoring tool that could present results based on proven knowledge of every part of the infrastructure and how they interacted with PowerCenter when we ran our schedule. But was there a tool out there that fitted the bill? Yes, if you wanted generic Infrastructure information but no if you wanted to view this information in the context of PowerCenter. There was nothing out there could tell us, for example, exactly which mapping of the four running simultaneously at 2 am was consuming most CPU, how were their parent workflows being distributed across the grid and was network or storage latency varying over time and which workflows suffered most? We needed to be able to speak to the Network, Database, Server and Storage Administrators using statistics that were generated by those parts of the infrastructure that they know and trust. We knew that it wasn’t a dense report we needed but easily communicable graphs of infrastructure performance in the context of the PowerCenter workflows, sessions and mappings that we were running. And we needed them in near real time, not two weeks later.
Does this sound familiar to you? Don’t you agree that it is time to rethink what Informatica Performance Tuning should actually mean? PowerCenter is mature enough to enable us to design and write mappings for optimal performance first time around making performance a given, not something we retrofit. That done, if you have performance challenges then you can start by looking outside of PowerCenter and engaging the other people involved in meeting the challenge: your infrastructure colleagues.
We did just that and we never looked back; we even became a trusted partner of Informatica along the way. So what changed? Well, we never found that elusive tool, it didn’t exist, so we wrote our own and called it HIPPO. Now we could share a view of Informatica performance that the infrastructure folks recognized and accepted. They could help us meet our SLAs because they could literally see our problem. The steps we took from there backed up our conviction that real progress in tuning means finding a common language to share an understanding with infrastructure teams of how PowerCenter interacts with the infrastructure that it sits within. Finding a common viewpoint unlocks their insight and expertise and helps everyone understand the real problem areas.
Big data means bigger performance challenges for everyone across IT’s great divide so if you want to be ready to meet these challenges then get a fresh perspective by test-driving HIPPO. You’ll be amazed what can happen when everyone can see the big picture.