In our previous blog on BMC’s data democratization strategy, we described our efforts to make enterprise data and analytics capabilities available to business users throughout the company as part of our journey to become an Autonomous Digital Enterprise. In this installment, we’ll explore how our work thus far has allowed dozens of finance professionals to access new enterprise data and insights and perform self-service analytics. The finance organization is a great example of how data democratization enables business users to apply and advance to high-level enterprise strategy in their day-to-day work.
As part of BMC’s focus on driving innovation to become an Autonomous Digital Enterprise, our finance operation underwent a transformation. The role was becoming more strategic and we were called on to provide information and perspective to guide decisions in all areas of the business. We needed to create new insights, produce more reports, and be more creative in how we analyzed and presented data. Our needs aligned with the company’s mission to open up the enterprise data warehouse (EDW) and make self-service analytics available to business users. Finance became early adopters – and the entire company is benefiting – as we’ll describe in this blog.
Before we get started though, you’ll need some relevant background on our systems and operations. Oracle Financials is the core for our day-to-day finance operations. It is just one of more than 35 source systems (Salesforce and CallidusCloud are among the others) that feed our EDW. The foundation is Netezza, which is now being supplemented and replaced by Snowflake. Similarly, OneStream is supplanting Hyperion as the analysis engine. The EDW includes many departmental data lakes and pools, and employees in the finance organization were among the first to create user-managed databases (UMDBs) in it.
For financial reporting analysis, data flows from Oracle Financials to our corporate performance management toolset, which includes Hyperion and OneStream. Depending on the report or user, Power BI, Tableau, and Alteryx are all used for output.
These aren’t even all the tools and foundational pieces, and you can already get a sense for the integrations and handoffs that need to occur. No single ETL solution is used across the environment (there are six, plus numerous scripts), which adds to the complexity. Scheduling jobs across this environment and making sure they executed in the needed sequence used to be very challenging, especially during the closing cycle when it seemed everyone in the C-suite was watching what we were doing and asking for updates.
Life before Control-M
Here’s what some of our processes were like before we were able to modernize, with the help of Control-M, which brought automation, orchestration and self-service capabilities into the process.
When a customer signs a contract with BMC, our salesperson reports the deal and a service representative books the order in Oracle, where it waits for review by the revenue recognition team. One of those specialists reviews the order to determine the right way to classify it in our accounting system (e.g. service vs. product revenue, immediate vs. deferred, etc.).
Once the order is booked its various details are parsed and used in a variety of reports, which are exported via MicroStrategy, Excel, and other formats. Producing the reports required a lot of time-consuming manual data joins and refreshes to various systems. The join tables were often a mess, and we could spend days going back and forth over whose data was right.
Pre-automation, closing out a quarter was even more time consuming and stressful (and had to be done while we were still booking revenue, which is an ongoing activity). As with revenue booking, quarterly close process requires coordinating outputs from multiple systems in a specific sequence to provide reports. A breakdown in any source data or processing job would hold up the entire process. Quarterly results are eagerly anticipated and the process is closely watched, from the top down. The quarterly close process preparation, execution, and monitoring took hours, during which time the Global Financial Information Services (GFIS) team was nearly constantly responding to update requests.
Our team was monitoring eight or nine systems to make sure it was using the most up-to-date data, and had to manage file transfers and workload executions across all the systems. Scheduling the workloads alone was a significant hassle and the other manual steps were very time consuming. Doing the basic work – and answering emails and phone calls about its status – took up most of the GFIS team’s time. That prevented us from pursuing the type of data analysis and machine learning (ML) development we needed to do to see overall company operations more strategically. Later, our finance professionals were surprised at just how much of the process we could automate.
We’ve been told that BMC tracks more metrics and does more reporting than most software companies. The finance team’s mission to innovate data analysis to produce new insights meant that we’d be doing even more reporting. It became clear that our methods would be unsustainable. Our team had an open mind and some experience with Control-M for managing our workflows, so the finance team explored how it might help us with our booking and closing processes.
Life with Control-M
At this point, finance’s efforts to streamline processes dovetailed with IT’s data democratization initiative. Our IT leadership was creating its strategy and resources to open up the enterprise data warehouse for self service by business users. We became a test case for putting these ideas into practice.
Our approach was to try to automate as many of the data collection and processing tasks as we could, then to orchestrate the reporting processes as much as possible. Eventually, with Control-M, we were able to deploy automation end to end.
Control-M is a workflow automation and orchestration platform. It is application, data, infrastructure, and tool agnostic, which enables it to work across our dozens of systems of record, eliminating the need for multiple proprietary schedulers and ETL tools. Using the GUI in Control-M and the newly available access to the EDW, the finance team was able to build workflows to automatically receive, validate, and process data to complete bookings and closings. Finance professionals also created event-driven update notifications that are sent to business users automatically, which has saved them a lot of time that used to spend responding to requests. The development was self-service, but Control-M has native controls to block unauthorized data access and to prevent user-defined jobs interfering with the execution of others. This ability to orchestrate execution and prevent resource conflicts was immediately valuable, and became more so as reporting, workflows, and complexity all increased.
Going into the project, we thought one of the most time-consuming aspects would be to identify where all the data we wanted to use was located, then map it into workflows. We were pleasantly surprised to learn Control-M could do most of the mapping for us, which saved a lot of time.
To book an order or close a sale today, all the things that needed to happen before we used Control-M still need to happen. The big difference is our staff doesn’t need to do them all. For example, when a needed file or other data input comes in, Control-M automatically receives it and begins processing it. Control-M monitors all the upstream and downstream processes related to a job, and can automatically send status updates and notifications to anyone that wants to be kept informed of the process status. Previously, getting an update often involved sending a request to our systems team in India, and waiting up to 12 hours for a response.
Some of our team took advantage of the self-service capabilities in Control-M and the EDW to create real-time dashboards that show the status of the overall closing process, including key sub-processes and their dependencies. The graphic below depicts the process and includes a screenshot from one of the dashboards.
Now, business users no longer need to wait hours for an update, and our finance team doesn’t have to spend valuable time during the closing process responding to update requests. Our team can do its work, instead of updating colleagues on the status of our work. That is appreciated by staff and executives alike. Control-M has produced similar benefits for booking and other day-to-day processes. Quite simply, it has become the glue that holds our financial reporting together.
The innovation dividend
Since marrying self-service EDW access with automation from Control-M, finance has delivered several valuable innovations thanks to the time and effort we’ve saved. The benefits extend far beyond the finance department and are appreciated by BMC’s executive leadership team.
The time savings have also enabled the finance organization to advance in our efforts to become more strategic. The Global Financial Information Services staff used the time it saved to explore and apply machine learning. Today, an ML solution the team developed is helping solve a longstanding challenge for the company: How to improve forecasting for contract renewals, and improve renewal rates. Doing that required finding answers to difficult questions, such as: When is the best time to contact customers about renewal? Which customers with expiring contracts would benefit most from proactive outreach? Would outreach improve the likelihood the customer would renew? What is the best action to take for each individual customer?
Analytics and ML can help answer those questions. So, GFIS used Control-M to give business users access to the tools they needed to apply analytics and ML. To predict and then proactively mitigate customer churn, we first needed to get a better understanding of customer satisfaction. We already used Salesforce for our CRM, so improving our visibility into customer satisfaction would require us to get more insight into the customer relationship than Salesforce was giving us. With Control-M we could automatically take the data from a lot of sources and push it out to the data sets we need for our ML models.
Taking advantage of our access to the EDW and the self-service analytics enabled by Control-M, we set out to create a dashboard that would reflect the relationship health BMC has with each customer. That visibility would help us throughout the lifetime of our customer engagement by identifying any potential customer dissatisfaction issues long before we talked with the customer about renewing. Control-M manages all the data transfers from disparate sources, and executes all the workloads to produce and distribute the dashboards.
The dashboard helps with day-to-day customer care. For example, if we detect an issue, the system automatically notifies a customer care or client account representative. This new deliverable ensures we proactively give customers the attention and support they need when they need it, and helps BMC deploy customer support and other resources more effectively. Customers that need attention or would benefit from a check in are contacted promptly.
We then built ML models that analyze the dashboard data to predict each customer’s likelihood to renew. Results are output via Tableau, and Control-M manages all the file transfers and job sequencing needed to make it happen. The output has really helped our account teams focus their efforts.
We’ve been able to do more with ML now that much of the necessary data wrangling is automated through Control-M and our BI tools. We have a lot of use cases where we need to predict things, some internally focused and others at a customer level. On the finance side, we use those insights for financial and strategic planning, building business cases, and lots more. Some of the use cases we’ve already developed include:
- A duplicate payment exception report generator that identifies duplicate invoices to avoid multiple payments for the same invoice. In this case, we combined Control-M and SQL for flagging exceptions, reducing processing times, and increasing accuracy.
- An employee shift allowance calculator for calculating payments for employees working in different shifts. By automating this activity with Control-M and SQL, we reduced the time needed to complete this process from two days to just a few minutes. We’ve extended automation to cover additional data loads for weekend and public holiday payouts.
- For cost accounting, we created a new process for consolidating cost adjustment files. The process helps remove manual errors (like data duplication or elimination) and runs in a few minutes; the process took about five hours when it was done manually.
We think there are many more significant opportunities for Control-M to play a lead role in ML use cases moving forward. Before Control-M, we were manually developing Python scripts and batch jobs to pull and process the data we wanted in our ML models. That took a lot of time, which limited the amount of actual analysis and experimentation we could do. Introducing Control-M into ModelOps has already helped and provides a gateway to more innovation. Market research found that many organizations are moving in that direction, as we described in this blog.
Meanwhile, self-service analytics and reporting continues to expand within BMC. The various automated updates Control-M now produces have been extremely popular, and the number of business users that request notifications is growing every quarter. The notifications and visibility into processes have given business users a glimpse into what is possible, and now many are beginning to explore their own use cases for self-service EDW access. They are creating their own dashboards, custom reports, and notifications. The global financial services team alone has eight separate UMDBs in the EDW, and other groups are creating their own.
Overall, these efforts will produce more innovation and efficiency benefits. We’ve already learned a lot about how analytics and ML processes can be automated. The bigger learning is that with the right tools, organizations can put high-level strategy into practice at the business user level. We’ll be sharing more use case examples in the future, so watch this space.
To learn more about Control-M, visit bmc.com/control-m.
