Connectors have been a traditional enterprise application integration (EAI) tool since the dawn of EAI back in the 1990s. According to Wikipedia, connectors (also called adapters) “are programs that know how to interact with an underlying business application. The adapter performs two-way communication, performing requests from the hub against the application, and notifying the hub when an event of interest occurs in the application (a new record inserted, a transaction completed, etc.).
“Adapters can be specific to an application (e. g., built against the application vendor’s client libraries) or specific to a class of applications (e. g., can interact with any application through a standard communication protocol).”
The problem with traditional connectors are that they are tightly coupled – any change in the data format or interface requirements for either end of any interaction would require an update of the connector, at the risk of a failed interaction. They can also become a single point of failure, a bottleneck that limits scalability and introduces brittleness to the application environment.
The rise of Service-Oriented Architecture (SOA) was in part intended to resolve the limitations of such traditional connectors. By abstracting interfaces to software as Web Services, the theory went, interactions could now be loosely coupled. It was possible to change the underlying software, either on the consumer or provider side, without breaking the interaction, as long as both endpoints conformed to the Web Services interface contract.
Web Services, however, largely fell short in delivering the loose coupling enterprises required to support truly flexible integration. Furthermore, Web Services were really little more than contracted interfaces, and still required an underlying software implementation.
In the 2000s this implementation typically became part of the Enterprise Service Bus (ESB) – a traditional piece of integration middleware exposing connectors much as the earlier generation of EAI middleware did, only now supporting the XML-based Web Services standards.
Then in the late 2000s, cloud computing came on the scene, and enterprises realized that they needed a more lightweight, decentralized approach to integration that didn’t have the limitations of the older ESB technology. These priorities led to the rise of Representational State Transfer (REST) – an architectural style originally intended to build hypermedia applications that generalized the behavior of the World Wide Web, but which soon became a lightweight, HTTP-based approach for building application programming interfaces (APIs).
However, because REST is based upon HTTP, it doesn’t deal well with data formats. After all, you expect standard Internet Media Types when you navigate the Web – HTML, JavaScript, PDF, and the like. To fill this void, the JavaScript Object Notation (JSON) combines the hierarchical structure of XML with the loose typing of JavaScript into a lightweight, web-friendly data format notation that doesn’t require predefined data schemas.
In fact, it’s possible to change a JSON file’s data structure on the fly – an advantage that adds flexibility to be sure, but throws a wrench into connectors or Web Services or any other traditional integration tool that expects data to conform to a particular schema.
Addressing the Challenges of Loose Coupling
SnapLogic is one of the new crop of vendors who are addressing this rather tangled web of strengths and weaknesses of various integration technologies. SnapLogic has implemented a new generation of connector they call Snaps that deal with the challenge of understanding endpoints – in particular, how to deal with changing APIs and applications while avoiding failed interactions.
Snaps are intelligent software components that provide design-time introspection. In other words, schemas and other metadata relevant to the interaction do not need to be known before the integration is set up. Instead, the Snap automatically gathers such metadata, automating the configuration of the integration. In situations where semantic mapping is required, SnapLogic automatically configures the Mapper Snap to facilitate the mapping process by conducting a fuzzy, “best guess” search across input and output fields.
Developers build even complex, multi-step integrations by assembling Snaps into Pipelines simply by snapping them together in a loosely coupled, iterative fashion, as shown in the figure below. Because SnapLogic is REST-based, each Pipeline is abstracted as a RESTful resource, which makes it addressable with a URL just like any other RESTful resource. REST calls can also trigger or schedule Pipelines, as they naturally support asynchronous behavior.
Pipeline illustrating Mapper Snaps and native JSON manipulation (Source: SnapLogic)
Furthermore, SnapLogic is a JSON-centric platform. SnapLogic’s support for JSON documents allows Snap endpoints to directly consume hierarchical data in native format and send them on to downstream Snaps in a Pipeline. As a result, there is no requirement to flatten data into records or to turn a JSON document into a string or BLOB (binary) type.
Traditional connectors often perform data transformations, but they typically require strict, rigid data mapping between endpoints. SnapLogic, in contrast, treats all data as natively hierarchical documents, an architectural decision that provides greater flexibility than traditional connectors.
In particular, Snaps allow developers to reach into these documents at any point. There is also no need to transform the document to flatten out the data. Snaps can also rearrange the data as needed.
This JSON-based document model is a superset of the relational model, so it easily works with relational data as well. However, in the case of relational data, schema changes can wreak havoc for traditional data integration – even changes as simple as adding a column to a table. Such changes, or even more complex examples, are no problem for SnapLogic.
The world of integration connectors will never be the same. True, Snaps are still essentially connectors, but by combining RESTful APIs, native JSON document centricity, and design time metadata introspection, SnapLogic has resolved the knottiest coupling issues inherent in all earlier generation connector technologies. At this point, there’s no reason to integrate any other way.
SnapLogic is an Intellyx client. At the time of writing, no other organizations mentioned in this article are Intellyx clients. Intellyx retains full editorial control over the content of this article.
Comments