In our first post of this series, we discussed how to identify opportunities where Agile can be used in BI\DW enterprise initiatives and how to begin planning for implementation. So you have an eager agile team, prioritized business functionality, a system design, an invested business owner, and a desire to make each iteration better than the last. Smooth sailing from here on out, right? Unfortunately, that isn’t usually case. Every project, including BI/DW projects, will have pitfalls and challenges. However, you can avoid these pitfalls and gracefully face these challenges with some lessons learned and best practices derived from the experience of teams who have successfully made the leap to agile BI/DW.
Stay True to Agile
It may seem obvious, but the key to a successful BI/DW project is to stay true to agile: start small and deliver often. With large scale corporate data systems, it’s very easy to be tempted to build out data domains in anticipation of future needs. An example would be building out a customer dimension with many more attribute fields than is specified by the feature/card simply because the fields are readily available in the source systems and there is likelihood that those fields will be requested in the future. One philosophy would say that there is a cost savings to be realized by adding in as much as possible up front to reduce the number of times that same code will need to be touched in the future. Although there may be a moderate savings in the development effort, additional fields means incremental data modelling, potentially incremental transformation logic or ETL business rules, and definitely incremental testing; all of which translates to additional cost and longer release cycles. Coaching the business owner on the pros and cons of various approaches to development is always encouraged, but decisions should be left in their hands. Don’t gold plate business features. That will always lead to a team that over promises and under delivers.
Proper Planning is Essential
If source system enhancements are required to maintain additional data needed for the BI/DW platform, then plan accordingly. Features should be oriented to the outcome, which is functionality in the BI/DW application. In order to get to that outcome, there should be cards under that feature that account for development required in all areas of the solution, including source system enhancements. Set dependencies between the cards appropriately, and incorporate those dependencies into Release Planning efforts. Do not assume you can develop all of those layers concurrently with alignment to a single deployment date. There is a natural order of operations to developing a multi-platform solution. It is OK to take multiple iterations working across multiple platforms to plan the release of a new feature. Sure, these features that require that level of effort will be more expensive but that is identifiable upfront and should be considered before the work begins.
Use the Right Mix of Resource Skills
If the BI/DW project happens to include all aspects of the technical stack (DBMS, ETL, reporting, metadata/glossary, data quality), the team may run into skill set issues. Agile often assumes that anyone can jump in on any task. That is not always the case with BI skills. A general guideline is to use a mixed group of resources on BI/DW agile teams where at least 50% or more of each team member’s time can be utilized for all iterations in a planning session. If a particular resource cannot be involved at least 50% of the time, use them as a shared service who participates when they can or when work exists that matches their particular skill set.
Get Support from the Business
Treat the corporate application like a “Product”. Give it the organizational support it needs to be successful. Assign a Business Owner and a Product Owner. Alternatively, for BI/DW applications, assign Business Ownership to your Data Governance organization. There will be many cross functional users of most corporate BI platforms and it will be impossible for the agile team to know the priority of the backlog without a business authority to govern that list. Without that degree of Business backing, the BI platform team will be left to make decisions on priority which will always result in one or more unhappy customer. An organization’s data and analytics capabilities are incredibly valuable. A deliberate approach should be taken to maintaining and extending it.
