Third party integrations: should I use APIs or files?

Integrating with a 3rd party app typically involves two or more apps using APIs (application programming interfaces) in order to automatically interact with one another.

The APIs permit data exchange to occur between the two apps, which ultimately saves time for the user. However, there’s an inevitable toll attached to building such an integration.

It adds a layer of dependency on a 3rd party, whether it’s a vendor, customer, or partner.

At least two parties have to agree on the interfaces, develop the APIs and the client code that uses the APIs, deploy them, then make sure they're secure so that they don’t open any loopholes to the apps.

On top of all of that, they must maintain the integration, which involves ensuring that it works every time and continues to work when there are updates and upgrades made to both apps.

Depending on your role in the relationship between the API provider and API consumer, you may be required to adhere to all sorts of standards that the other party requires—in terms of interfaces, data structures, cyber security, and compliance with various regulatory laws.

In some cases, when working with enterprise corporations who own legacy systems, using secure file shares for data exchange rather than modern REST APIs is required.

In this post, we’ll compare and contrast 3rd party integration via APIs vs. secure file exchanges.

Speed vs. bulk operations

Most APIs work at the single item level.

Sure, you can get a list of objects, but actions such as creating, updating, and deleting items are usually done as record-by-record operations. This usually means that APIs are good for real-time integration at the record level.

However, bulk operations are not commonly supported by APIs and require the client to iterate over the records and perform an API call for each record, which essentially makes them longer and riskier.

That's because APIs hardly ever support the notion of an atomic transactional operation (i.e. all or nothing), so you may end up with only part of the work done.

File exchanges work best when utilizing bulk-level integrations. For example, getting a nightly file of new or updated records to be processed by the other app.

Modern file exchanges, however, allow you to define webhook notifications when files are uploaded, making real-time integration over file exchanges a reality, whether the files contain bulk data or just a single request for changes.

Security

To ensure your APIs security, you’re going to have to:

• Implement authentication mechanisms: make sure that whoever is trying to use them, is who they say they are.
• Implement authorization mechanisms: make sure they have permission to do what they’re trying to do or to access the item they’re trying to access.
• Ensure that data transmitted between the client and the server is encrypted, to avoid man-in-the-middle surprises.

There are a bunch of standards, such as OAuth for authentication, API scopes for authorization, and HTTPS for encryption, but there’s still a learning curve and a technical toll when it comes to implementing them.

Other security measures such as VPNs, a requirement for inbound network rules, or static IPs (used to set outbound network rules), add complexity to the architecture and deployment processes.

Popular file exchange protocols such as SFTP and FTPS have built-in security measures within the protocols:

• Authorization comes in the form of permission for users to access folders and files.
• Public key authentication is considered extremely secure, but you can always use password authentication as well.
• Data encryption on the wire is part of the protocols and there’s a variety of strong ciphers that are used to protect your data.
• Adding VPNs, static IP addresses, or inbound network rules around a file exchange is as easy as it gets.

Moreover, since these protocols have been around forever (over 20 years), it’s safe to say that almost every distinguished CISO and IT person is familiar with them and won’t require any extensive security test to make sure they’re impenetrable (as opposed to your custom APIs).

A survey conducted by Thales Group in 2023 highlighted that 47% of organizations are storing sensitive data in the cloud, with a strategy to encrypt sensitive data, showcasing the security measures in place for cloud storage solutions.

This can be seen as a testament to the security features of cloud-based file exchange solutions. More details can be found on the Thales Group website.

Interoperability

HTTP based APIs allow, by definition, for interoperability to occur. This means that you may access the APIs from a variety of operating systems and programming languages.

However, APIs behave differently when found on standard web pages and are not meant to be consumed by people.

SFTP file exchanges may also be accessed from any operating system and there are programming libraries available for any programming language to access and manage files on remote servers using the SFTP protocol.

In addition, command line interfaces can be used to automate processes without writing a single line of code, so that advanced users can automate the usage of these data exchange points.

Lastly, as opposed to APIs, graphical user interface clients such as FileZilla, WinSCP, Cyberduck and many others allow authorized humans to access the files as well, when necessary, for example, if something were to go wrong or a bug arises.

Scalability

What happens when you need to serve a lot of concurrent API requests? Scaling out API servers may seem fairly easy, as you can just throw more web servers at the job with a network load balancer to distribute the traffic across the servers.

However, this is usually just the first tier that gets hit by concurrent requests. Your APIs will usually query or update a database directly, which may become the next bottleneck.

In addition, managing API rate limits (requests per minute, per hour, per day, etc.) is no easy task, but it is mandatory in order to keep your APIs from being bombarded with requests that can bring them down.

Standard file servers are more difficult to scale out. Usually, you would assign them a different network endpoint, since they don’t naturally work well with network load balancers.

Modern cloud based solutions, such as SFTP To Go, can actually be treated as a serverless solution because they scale up or down depending on the number of concurrent sessions they are handling, and the storage itself is accessible concurrently by all servers.

While not designed to handle millions of concurrent requests, SFTP To Go can handle thousands of concurrent sessions to help you consume and process file-based requests in a timely fashion, together with webhook notifications.

Cost and Maintenance

APIs can be expensive to develop and maintain, as they require specialized programming knowledge as well as regular maintenance to ensure they are functioning properly and securely.

File exchanges don’t solve the problem entirely—they won’t process the data on your behalf, but they do serve as a reliable, asynchronous data exchange point between 3rd parties and your app.

They’re incredibly easy to set up and require hardly any configuration.

As for maintenance, with a managed service, there’s hardly any work to be done on your behalf. If you’d like to set one up on your own, make sure to compare the pros and cons of DIY vs. managed services.

According to a report by Synergy Research Group, companies are able to save up to 20% of their IT budget by utilizing cloud-based solutions, including file sharing services.

This illustrates the cost-efficiency of opting for cloud-based file sharing solutions. You can find more about this on the Synergy Research Group website.

Flexibility

Unlike file exchanges, APIs require implementing a certain interface. That is, defining what arguments an API endpoint takes as input and what it returns so that the interface is rigid.

On the other hand, it’s completely up to the developer to implement and so the functionality they provide can be very flexible. File exchanges help you upload, download, and manage files which are the basic unit exchanged by the parties.

The files reside in folders you may designate to different users or different types of data, but there are no restrictions on what the file actually contains - CSV, JSON, XML, simple text, images, videos or any other type of file.

It is up to the party that downloads and processes the file to verify that it contains data that conforms to the agreed format.

Meet SFTP To Go

SFTP To Go is a secure cloud file storage and sharing service that utilizes advanced cloud infrastructure in order to make sure that your data is secure and durable on the wire and when stored.

You can use SFTP To Go’s web UI and API to manage credentials and their access to your storage, allowing you and other users to access files using secure popular protocols such as SFTP, FTPS, S3, and HTTPS.

With the help of webhook notifications triggered by file system changes, you can create cost-effective file based real-time data integration.

Ready to learn how SFTP To Go can help you integrate data easily? Start your free trial today or reach out to our team to discuss your business needs and objectives.