Agile methodologies, such as Scrum and Extreme Programming (XP), have grown in popularity in recent years for software development projects. Typically, these projects are thought of as endeavors to build stand-alone, packaged products or applications. Less common is the use of agile approaches to build out complex enterprise scale solutions that involve multiple disparate corporate applications. But, in fact, agile methodologies can be extended beyond the “packaged product”. Prominent examples of this are business intelligence and data warehousing (BI/DW) applications where the central technical objective is centralizing and conforming large sets of data to uniform definitions.
For many, this may seem like an improbable assertion. After all, BI/DW projects often occur as larger scale, enterprise level undertakings. Unlike software development, BI/DW involves configuring various commercial tools or home-grown transactional systems so that they integrate seamlessly. Customization occurs within the underlying data models and data manipulation “code” (ETL, SQL scripts, stored procedures, etc.). Under these conditions, it can be difficult for BI/DW practitioners to identify opportunities to adapt agile concepts that continuously deliver business value to BI/DW development. The adoption of agile methods is dependent on the maturity of the BI/DW application in question.
When a brand new BI/DW application is being conceived and agile methods will be used from project inception, there will be some time expenditure associated with requirements gathering, prioritization, and solution design. Iteration planning should be outcome focused rather than deployment focused. During this time, a couple of different types of “spike cards” will be played by the agile team:
- Business Analysts (BAs) will focus on identifying the feature set for the application and will prioritize them according to business value provided by the Product and Business Owners.
- Developers and Solution Architects will then review the feature set and will identify the breadth of data domains involved, the number of data sources to integrate with, data currency requirements, and the of scale of the data sets (# of records). This will allow them to design a solution with the scalability and flexibility required for the known feature set.
- Testers should design a framework for functional, performance, and regression testing. They should also identify patterns for integration testing, design an approach for automated testing, and identify the tool sets to be used throughout.
Once the highest priority features are identified, the BAs can start working on detailed card level data sourcing analysis for those features. During this time, the development team should develop a high level solution architecture, conceptual data models or high level entity relationship diagrams to articulate the vision of the completed system. A benefit of iterative implementation of each data domain with keep the team honest as the design will constantly be challenged. Once the initial set of prioritized cards are written, development can begin and the initial release of the application can be planned.
From this point forward, as seen in more mature BI/DW applications, traditional agile processes can take over and functionality oriented Release and Iteration planning and development cycles can begin. Now the prioritized product backlog, combined with active learning from the user community, will begin to influence and drive the evolution of the application. As increased usage of the system takes place, more is learned about the data. The knowledge earned from this learning may drive re-prioritization of the product backlog. Through the benefit of frequent releases of new capability, re-prioritization can be quickly incorporated into the product. The product owner participates in the team and can justify the re-prioritization and drive the most beneficial make-up of each release.
All of this makes agile just as well suited for BI/DW projects as it is for traditional software development projects. That is not to say that BI/DW projects do not offer unique challenges; they do. However, project teams can adapt to them. In the next post, we’ll talk about lessons learned and best practices from teams that made the successful leap to agile BI/DW.
