Request a Demo
Close Close

Let's schedule you a demo!

Please tell us how to contact you. We’ll only use your email to schedule a brief demo of Conversionomics at a time that works for you.

Data automation tools are integral to, but not fully integrated into data ops

Posted on November 1, 2019 in Industry Insights by wpengine

We wanted to learn about the benefits and challenges that data automation platforms bring to the analysts that use them. To establish a benchmark, we conducted a survey of more than 300 data analysts, asking them specifically about their data automation practices and needs. Here is what we learned:

Finding #1: Data automation tools are (still) hard to categorize

No one has a definitive classification for the apps and platforms that deal with data, and for a good reason. The data technology landscape is evolving as quickly as the industry. Data is generating faster, we’re storing and processing more of it, and we’re finding new ways to transform and analyze it. So technology vendors today are offering a rich but confusing spectrum of data automation platforms in an attempt to match our needs. To help avoid this confusion, we use our own simple categorization with examples (see Fig 1), but that didn’t stop people from adding tools such as Hubspot or Airtable in the Other category. Every app that collects and displays data in some form is seen as a data automation tool today, blurring the lines even further.

Fig. 1. Data automation tools used.

Data automation tools used by analysts

Finding #2: Spreadsheets dominate, but younger analysts are ready to try out new data tools

It’s obvious from Fig. 1 that using Excel or a similar spreadsheet app is a staple in every analyst’s life. But digging deeper, we discovered that the grip is weakening. Not surprisingly, younger generations are much more willing to adopt newer, more specialized tools. Particularly, the balance is shifting away from Spreadsheets and Data Analytics/ BI tools, and more towards OpenSource tools (e.g. Python, R, Apache Spark) and Visual analytics tools (e.g. Tableau, Domo, Data Studio, Looker). Even after accounting for age differences, however, the use of spreadsheets remains in the 50%+ range for all analysts.

Fig. 2. Data automation tools used by age group.


Finding #3: Few analysts use data automation tools for all common data analysis tasks… for now

Our findings indicate that most analysts do not use their data automation platforms beyond common tasks, such as pulling data from various sources and sharing the results (both at about about 53%). We were surprised to see that only about a third or fewer analysts use their data tools to further transform, pivot, merge or aggregate their data. These tasks are just as “automatable.” The fact that they’re not performed as part of the data preparation/integration process could mean one of several things: 

  1. Data automation platforms don’t have the capability 
  2. Analysts perform the tasks manually in spreadsheets 
  3. They use other tools to transform data
  4. The analysts receive the data already prepped.

Fig. 2. Most common data preparation tasks.

Most common data preparation tasks performed by analysts

The first two scenarios represent missed opportunities to optimize more of the low-value data management tasks that data analysts perform today. Our next survey insight seems to confirm this…

Finding #4: Analysts spend more time and effort on manual data operations than necessary

Fully 67% of the analysts in our survey agreed or strongly agreed with the statement “I manually clean and transform my data using spreadsheets.” Another 61% agreed or strongly agreed with the statement “I spend more time than I’d like on data preparation.” These answers are rather surprising, given the advances that data automation platforms have made in recent years. One possible reason for the discrepancy is that analysts are simply not aware or familiar enough with all features in their data automation platform. That possibility is supported by the fact that only 39% of analysts agreed or strongly agreed with the statement “I use most of the features available in my data preparation tool(s).”

Fig. 3. Time spent and features used in data preparation apps.

Time spend and features used for data prep tasks

Most users do not use most of the features available in an app, that’s a fact. The more complex the app, the more likely that is to be true – and data automation platforms are pretty complex. The burden of proof falls on the platform vendors, then. They need to demonstrate the value of their products in savings and benefits for their clients. Respectively, their clients’ need to invest in properly training their analysts, so they can reap the full benefits of using a powerful data automation platform. For example, we can show how Conversionomics can save 10+ hours per week of repetitive, low-value tasks for an analyst and thousands in data processing and storage costs per month for a company. We also offer customized onboarding and training for clients, so their analysts can be up and running fast.

Finding #5: Analysts feel just “OK” about their data automation tools

About half of analysts agree that they’re well set up for success with their technology. Their confidence comes from feeling technically competent and supported when working with their data tools. But when asked whether they have “the right tool(s) to automate” their data preparation process or “get a lot of value for… the price”, many answered with “neutral”, indicating that enthusiasm is rather muted. With only 46% of analysts thinking that their data tools are easy to learn and use, the hesitation is understandable. Combined with the 38% above who say they don’t use most features, the answers add to the feeling of missing out on the value provided by these tools.

Fig. 4. Level of support and satisfaction with data automation tools.

Levels of support and satisfaction with data automation tools

Again, our follow-up question confirmed the FOMO. When asked “How interested are you in a tool that automates your data preparation process?”, 43% of respondents said that they are “very” or “extremely interested” and 41% said they are “somewhat” interested. This overwhelming interest indicates that analysts see data tools as integral to their data operations. They just haven’t found one that integrates with their process as well as they’d like. So, the search for that “perfect” data automation platform continues, even for those analysts who currently use such tools. 

This article presents findings from our survey of 312 U.S.-based users involved with data analytics and operations in their organization. Online survey conducted June 20-21, 2019.

Copy link
Powered by Social Snap