Setting up an API connection isn’t particularly difficult. There are plenty of open-source solutions available, and even with some customization, an experienced engineer can typically set one up in a few days.
However, that’s just the start. The real challenges (and costs) come once you have that connection in place and it’s time to build your data pipeline. Why? Because, after the initial API connection, you need to address a range of critical questions:
1. Authorization & Access
How will you authorize connections? Most, if not all, data sources require some sort of authorization in order to access the data in the first place. While there are many different systems, about 80% of data sources use token-based authentication.
If you don’t automate authorization, you’ll spend a huge amount of time manually requesting tokens and managing access. Depending on how much data you need to access and how often, having a system to automatically access your data is critical.
At Adverity, we have a whole bunch of features to make this simpler for our users, and you can read all about the benefits of centralized authorization here.
2. API Request Limits
Many APIs impose limits on the number of data requests you can make per day or even per minute. These can get quite complicated depending on what access level or account you hold, and generally, each platform measures limits in different ways. However, what this means is that there is a limit on how much data you can fetch from any given platform, and if you exceed these limits, your data pipeline could fail or require workarounds.
Importantly, these limits also can, and do, change, such as in 2023 when changes to GA4’s API limits caused a lot of problems for data engineers across the industry as they sought to find workarounds (you can read more about this here).
Again, at Adverity, we have a number of different techniques and strategies for solving this challenge, so that our customers never have to think, let alone worry, about rate limits. These include a lot of complicated-sounding terms such as intelligent throttling management and data chunking. If you are that way inclined, you can learn more about them here.
Figure 1 Examples of rate limits and quotas for popular data sources
3. API Rules & Restrictions
Each API comes with its own set of rules and restrictions. Some may limit the type of data you can access or have strict compliance requirements. All these rules and restrictions add additional layers of complexity that need to be tackled to successfully and regularly extract your data.
4. Data Security
What security measures need to be in place? Who will be able to access the data? If you’re handling PII (Personally Identifiable Information), compliance with regulations like GDPR and CCPA is critical, and failure to comply could result in hefty fines. In most cases, simply extracting data to a spreadsheet will not be enough to ensure compliance, so you will have to consider how and where your data is stored and how it is accessed.
5. Data Storage
Where will the data be stored? There are a number of options for storing your data. You could choose data lake solutions such as Amazon S3 or Databricks or database solutions such as Snowflake or Google BigQuery. However, working with these destinations requires additional work on connecting your data source API to the destination API. Whatever solution you choose, you will need to determine how your data interacts with that chosen environment.
6. Data Mapping & Integration
If you’re collecting data from multiple data sources (and let’s face it, you probably are), then it needs to be correctly mapped and integrated before you can use it; otherwise, you’ll be attempting to compare apples to oranges. Now, this process can be done manually, but even with a small amount of data, the process is time-consuming and prone to human error. If dealing with any substantial amount of data, the task becomes almost impossible, making automated mapping and integration an essential component of any modern marketing data pipeline. It’s not just about simply fetching the data - it’s about making it usable.
This is just the initial setup. It doesn’t necessarily include all the things a data pipeline can do, but what, at its most basic, a data pipeline should do. And just for one data pipeline. This process needs to be repeated for every single data source you want to connect to.
Once your system is set up, you’ll need to maintain it. This requires ongoing resources and costs. In fact, this constitutes the biggest long-term financial commitment when building your own solution and the one that is often overlooked.
1. API Updates
APIs are frequently updated, which means that every API you work with will change over time. If you don’t update your data pipeline accordingly, it could, and most likely will, break, leaving you with inaccurate or missing data. Of course, not all APIs are created equal and are updated with varying degrees of regularity and scale. Some APIs are updated very frequently, while others are less than once a year. Some API updates are minor and innocuous, while others tend to be large-scale.
From a cost point of view, this comes back to one of those ‘it depends’ questions regarding which data sources you want to use. Either way, it’s essential to have a team in place monitoring API updates and updating your data pipelines accordingly.
In fact, as a business, API maintenance constitutes quite a large part of what we do, and we spare no resources to ensure that we know when every update will occur, what it consists of, and that all our connectors are updated. Unexciting? Possibly. Essential? Absolutely.
Figure 2 Examples of API update schedules for popular data sources
2. Security & Compliance
Unsurprisingly, security and compliance are not issues that go away after the initial setup; they are factors that your team needs to be constantly vigilant about. Not least because APIs are a common target for cyberattacks. So, who is going to be responsible for securing your API connections, and how will your system handle DNS attacks, data breaches, or other security threats? Again, the time and resources spent on this will multiply with every data source and pipeline you need to maintain.
3. Ongoing data monitoring
Maintaining a data pipeline isn’t a "set it and forget it" job. You’ll need someone (or a team) to manage authorizations, integration issues, and troubleshoot errors if and when they arise. At the same time, data needs to be constantly monitored for accuracy, duplication, and general data quality to ensure your data is fit for purpose.
While bearing in mind all the ‘it depends’ questions, below we’ve provided rough estimates on the average time it takes to set up and maintain 10 data pipelines, based on our own experience. Why 10? Well, according to our research, 99% of marketing teams utilize at least 10 data sources.
Naturally, these estimates will vary from company to company, so we provide them only as a rough guide.
Figure 3 Initial set-up estimated costs for 10 data sources
Figure 4 Estimated annual ongoing maintenance costs for 10 data sources
This brings us to a conservative estimate of roughly $26k for initial set-up and then an ongoing annual maintenance cost of around $90k.
In reality, this cost will be higher or lower depending on your company, your needs, your current setup, and how many and which data sources you want to work with.
To this end, we’ve developed a calculator to help you get a more accurate estimate of what an internal build would cost your business. You can also adjust the expected salaries for each role, as we are aware that in different regions, these costs will vary.
Cost is one thing, but the time to completion is also crucial. Any downtime as you wait for a solution to become operational will impact your business and raise the question of what solution you will use in the interim.
To some extent, this can be mitigated by simply hiring or contracting more resources to expedite the process. However, of course, this needs to be weighed against the additional costs of sourcing and managing those resources.
That said, with a team consisting of a Data Engineer, a Software Developer, and a System Engineer, we estimate an average-sized solution comprised of 10 data pipelines would take a minimum of 8 weeks or about 2 months.
Importantly, if you want to scale by adding new data sources, then costs and resource allocation will need to grow accordingly.
Yes. While we have a vested interest in showing a high cost for an internal build, this is roughly what it costs. And, a significant amount of this cost goes towards the ongoing maintenance of your solution.
This is why we have a whole army of engineers and developers whose sole job is to manage and maintain our APIs. And we need to because we have more than 600 API connectors!
