It has been said that “Data is the New Oil” of the global digital economy. Today’s modern SaaS platforms process enormous amounts of data. Split alone processes over 1 trillion feature flags per month. Like oil, data needs to be extracted, refined, and transported across multiple systems and through different processes. Integration of data between systems when using APIs is one of the most fundamentally important components of any cohesive stack of applications. In this article, we’re going to take a look at some things to keep in mind when setting up an integration.
But First, a Quick History Lesson
If you’ve been in the tech world for a while, you may remember systems that primarily read from and write to simple plain text files—with each row being a new record. In fact, historically the text file was one of the most common ways to exchange data. UNIX tools for text processing would happily read and write to files. They could also pass the text output from one tool to another using the pipe symbol, or ‘|’. In fact, this is where we get the nomenclature still used today, “data pipelines.” Text files require storage, historically on a secure FTP site. A more modern approach to file-based interfaces is an object file store on a public cloud provider, like Amazon S3. Split has an Amazon S3 integration that allows seamless integration of data using this method.
Since the 1970s and the popularization of UNIX and its standard tools, there have been additional ways to automate data exchange between systems. With the explosion of the Internet, there was interest in using the protocols used by the browser to exchange data. In fact, when the REST API model was first documented in the year 2000, it was done by one of the authors of the HTTP specification itself. SaaS software platforms use these REST APIs to communicate with one another without sharing any infrastructure. System A can talk to System B using these APIs, the same as a browser downloading a website or submitting a form.
With that history lesson behind, let’s talk about the common considerations for integrating with web APIs, how Split uses them, and things to keep in mind.
Integration Considerations
Authentication
Consider how your systems need to authenticate with one another. How does System B know what user, account, or organization is represented by System A? SaaS platforms generally operate on top of shared infrastructure in a public cloud provider, using multi-tenancy. Many of Split’s integrations with partners, such as its Segment integration, use secure keys scoped by each Split environment. If you are building a bespoke integration, then Split’s Events API is similarly scoped by Split Environment using an SDK key. Connecting over TLS authentication is also a requirement from most systems when using APIs. This keeps data secure across the wire.
Message Frequency
Consider the frequency of data ingests or exports from source to target. In almost all cases, systems that have API Endpoints for an integration will have Rate Limiting to ensure that systems don’t get overloaded or potentially abused by malicious actors. A system that rate limits should be smart and let the calling system know that it has been rate limited and when to try next. Split’s Admin API Rate Limiting HTTP code 429 will return headers, letting the calling system know when to resume sending messages. As a result, no data is lost.
Individual Message Size
Consider the individual size of each message being sent from the source to the target system when using APIs. HTTP POST requests are not limited in size by the format specification, but most web servers have a limit. Split has taken extreme care to handle large payloads gracefully, but recommends payloads 1MB or less when hitting its Events Endpoint.
Message Body Format
Arguably one of the most important components of any integration is data mapping. Here are the dragons of a complex inner-system integration. Data formats should be inspected to ensure that they match. This includes things such as date. Ask yourself: Is this a timestamp? Is there a number of seconds since the unix epoch? UTC? GMT? Remember that number formatting should take into account quoting and decimal places. Data in an HTTP request is typically sent across in a JSON format. This means that data types aren’t specified in the message itself. These all should be specified and documented to include error scenarios. Split documents all of this information on our Admin API help page. For an example: Click on the HTTP response values in the center of this page to see the data type specified for each field in the JSON.
Step Back & Think About Why You’re Integrating
This article so far has had a heavy technical bent. It’s focused on the how of building an integration. What’s equally important (or even more crucial) is the why. Think about what you are trying to accomplish and why.
Integrations allow data to move from one system to another. But of course, some questions that should always be asked when designing or building an integration are “Will this meet the business needs?” “Will this handle my use case?” “What data do I need in my target system?” Saying one system is integrated with another doesn’t mention the goals of setting up the integration in the first place.
Integrate Smarter
Using Split’s APIs to integrate allows incredible flexibility and speed for creating, updating and deleting feature flags, sending events, gathering impressions, and more! After all, it takes two to tango, and Split’s list of high quality partners makes it easy to have all of your feature flag data where you want it, when you want it.
Get Split Certified
Split Arcade includes product explainer videos, clickable product tutorials, manipulatable code examples, and interactive challenges.
Switch It On With Split
Split gives product development teams the confidence to release features that matter faster. It’s the only feature management and experimentation solution that automatically attributes data-driven insight to every feature that’s released—all while enabling astoundingly easy deployment, profound risk reduction, and better visibility across teams. Split offers more than a platform: It offers partnership. By sticking with customers every step of the way, Split illuminates the path toward continuous improvement and timely innovation. Switch on a trial account, schedule a demo, or contact us for further questions.