Efficient Connector Design — To Reuse or to Overload?

In many iPaaS platforms, you are forced to treat connectors as "black boxes." If a connector doesn't output data in the exact format you need, your only option is to add more transformation nodes (mappers) to your UI. This leads to "topological noise"—crowded schemas that are hard to read and maintain.

In Orchesty, we take a different approach. Our components are classes, not just icons. This allows for a powerful design pattern: Connector Overloading.

The Concept: Base vs. Derived Connectors #

We maintain a collection of Base Connectors. These are reusable, generic components designed to perform a specific API action and return the raw response. They are the "Lego bricks" of our ecosystem.

However, in real-world topologies, you often need to carry a "Golden Thread" of data (the payload) throughout the entire process. Instead of creating unnecessary branches or transformation nodes just to merge data, Orchesty allows you to overload the connector to respect the specific payload of your topology.

The Problem: The "Transformation Fatigue" #

Imagine you need to get a personId from a CRM to create a Lead later. A standard connector would only output the personId. To keep your original order data, you would have to add a Mapper node to merge the two. Do this five times, and your topology is a mess.

Case Study: Data Enrichment (Pipedrive Example) #

Let's look at a fictional payload flowing through our topology:

{
  "orderId": "ORD-123",
  "customerEmail": "developer@orchesty.io",
  "amount": 500
}

1. The Standard Approach (Base Connector) #

A standard GetPersonIdByEmail connector is designed to be universal. It takes an email and returns the ID.

Snippet: Standard Connector Logic

// Standard implementation - returns only the API response
export class GetPersonIdConnector extends AbstractConnector {
    public async handle(userConfig: Config, email: string): Promise<string> {
        const response = await this.client.get(`/persons/search?email=${email}`);
        return response.data.id; 
    }
}

Resulting Payload: "9988". At this point, your original order data is gone from the active stream. To recover it using only generic, "closed" connectors, you would be forced into complex topological branching or cumbersome global state management just to merge the data back together later.

2. The Orchesty Approach (Overloaded Connector) #

Instead of adding a Mapper node in the UI, we simply create a Derived Connector within our Worker. We inherit from the base class but modify how it handles the input and output.

Snippet: Overloaded (Derived) Connector Logic

// Overloaded for a specific topology
// Goal: Enrich the existing payload with the Person ID
export class EnrichPayloadWithPersonId extends GetPersonIdConnector {
    public async handle(input: OrderPayload): Promise<OrderPayload> {
        // We use the logic from the base connector
        const personId = await super.handle(config, input.customerEmail);

        // We return the ENTIRE payload, enriched with the new field
        return {
            ...input,
            personId: personId
        };
    }
}

The Resulting Payload #

The process continues with all the data it needs in a single, clean object:

{
  "orderId": "ORD-123",
  "customerEmail": "developer@orchesty.io",
  "amount": 500,
  "personId": "9988"
}

Why This Matters for Architects #

1. Cleaner Topologies #

By overloading the connector, you eliminate the need for "administrative" nodes in your visual designer. The topology stays focused on the business logic, while the data manipulation happens where it belongs—in the code.

2. Code Sovereignty & Flexibility #

You aren't fighting the platform; you are using its object-oriented nature. If a connector's default behavior doesn't suit your specific use case, you change it in seconds by extending the class.

3. Reduced Complexity #

Less nodes in the UI means fewer points of failure and easier debugging. You can look at a topology and immediately see the high-level data flow, while the "Derived Connector" ensures the payload remains intact and enriched throughout the stream.