SEI Cincinnati’s Lauren McDonald will be speaking at the Women In Analytics conference in July. She introduces the project she will discuss and the lessons she and her team learned along the way.
Bio: Lauren McDonald has been a consultant at SEI Cincinnati for six years. Before joining SEI, she spent nearly a decade working in technology roles at a multinational energy conglomerate. At various stages, she worked as a big data engineer and served as an infrastructure service leader. Lauren is passionate about helping clients solve on-the-ground technology problems that create tangible business impact.
On July 21st, 2021, I will be speaking at the “Women In Analytics” conference in a breakout session for “Platforms & Process: Tools, Data, Infrastructure.” In preparation for this talk, here’s a breakdown of the project I will be discussing — and some of the lessons we learned through our work.
Vision and Strategy for an Event Streaming Platform
When we — myself and another SEI architect — began our engagement with a Cincinnati-headquartered retailer, we quickly realized that their existing methods of sharing data were brittle and hard to scale. They had tightly coupled teams with hard dependencies on the quality and velocity of data. In order to grow, they needed better data solutions and better methods for sharing data.
After interviewing various stakeholders across the company, we were able to understand the challenge from the perspective of their data analysts and scientists, their software engineers, and their business managers. The data analysts and scientists were under pressure to acquire high-quality data, the engineering teams needed to deliver new features at speed in order to compete in eCommerce, and business managers needed up-to-date data to derive insights into performance and to leverage predictive analytics to gain a competitive edge.
With all their needs in mind, we developed a vision and a strategy for an event-streaming platform. Over 3 years later, the platform vision is nearly realized. The platform has been adopted throughout the company’s IT organization with 65+ business event streams, 200+ consumers, and 2000+ users. The platform offers producers of data a way of articulating a client contract with their consumers. It enables businesses to evolve their model without the risk of breaking downstream clients.
Lessons We Learned Along the Way
We learned a lot from building a platform, and my breakout talk will outline some of the lessons and best practices. We started by designing a mission-critical and scalable streaming backbone using Kafka, with full infrastructure automation in a CI/CD pipeline and robust operational monitoring. This enabled us to gain confidence with the engineering community and demonstrate that we could reliably onboard users to the platform. This also freed the platform team to focus on features development instead of manual admin operations or support issues.
Some of the lessons we’ve learned about developing a platform that can be applied to other technologies include:
- The removal of tech debt
- The importance of high devX (developer experience)
- Preparing for constant change
From this engagement, a key takeaway is the importance of cultivating a community of users for the platform. This includes supporting an active group chat channel where users feel comfortable asking for help. If the group is successful, community members will continue to answer questions posed by teams across the enterprise, eventually evolving into an organic hub for sharing similar efforts and ideas. We even encourage an open-commit model of reference patterns for teams to share code and methods or commit features for the platform itself.
Join Us at WIA
On this engagement, I worked closely with the client and another SEI architect. As with all our client work, SEI brings not only strong engineering expertise but also a focus on solving the business problem. We zero in on what best serves the client’s unique needs and we help invigorate the culture of the IT organization to create company-wide impact. I look forward to sharing our experience building a data-streaming platform and getting to know the amazing women in the data & analytics community.
If you’d like to learn more about working with SEI, I encourage you to get in touch. To learn more about the WIA conference, please visit their website.
