Data synchronization for e-shop

The integration between an E-commerce platform and an Enterprise Resource Planning (ERP) system is the backbone of modern retail. However, most platforms struggle when the product catalog reaches 100,000+ items or when flash sales generate thousands of orders per minute.

This case study explores how Orchesty handles these standard yet high-stakes processes using an asynchronous, developer-first approach.

1. The Core Synchronization Workflows #

A typical integration consists of several bidirectional streams. Orchesty treats these as independent topologies, ensuring that a spike in order volume never slows down a price update.

Product Sync (Source of Truth): Whether the ERP or the E-shop holds the "Golden Record," Orchesty synchronizes descriptions, images, and categories.
Dynamic Pricing & Stock: High-frequency updates from the ERP to the E-shop to ensure customers never see an "Out of Stock" error or an outdated price.
Order Fulfillment: Pushing orders from the E-shop to the ERP and returning order statuses (Shipped, Paid, Cancelled) back to the customer.

E-shop integration processes covering product sync, stock updates, pricing, and order fulfillment — Orchesty handles product synchronization, stock and pricing updates, and order fulfillment as separate but coordinated process streams.

2. The "Delta Sync" Magic: One Topology to Rule Them All #

In traditional systems, you often need two separate processes: one for the "Initial Bulk Import" and another for "Incremental Updates."

In Orchesty, they are the same thing.

Using the Delta Pattern combined with a Cron Event:

The First Run: When you enable the topology, Orchesty detects no previous state and automatically performs a full synchronization of the entire catalog.
Subsequent Runs: Orchesty tracks the "Last Sync" timestamp. The next time the Cron fires, the process only fetches items that have changed since the last run.

The Benefit: You build, test, and maintain only one topology. The platform handles the intelligence of deciding whether to fetch one item or one million.

Delta synchronization pattern showing first full sync followed by incremental updates only — The same Orchesty topology handles both the initial full import and all subsequent delta synchronizations.

3. Performance: Protecting the PHP E-shop #

Most modern e-shops (like WooCommerce, Prestashop, or Magento) are built on PHP. While powerful, they can easily be overwhelmed by too many simultaneous API calls, leading to a "508 Resource Limit Reached" error.

Orchesty acts as a Protective Buffer:

Rate Limiting: You define exactly how many requests per second the e-shop can handle within the Application settings.
Throttling: If the ERP sends a massive burst of price updates, Orchesty’s asynchronous queues hold the messages and "drip-feed" them to the e-shop at a speed it can safely process.
Zero Impact on UX: Because the integration is asynchronous and decoupled, the speed of the sync never affects the loading speed of the storefront for the actual customers.

Comparator Pattern for Product Sync

When the source system cannot filter records by lastUpdated, Orchesty uses a Comparator to make the synchronization efficient. The source may still send a broad dataset, but only the items that actually changed are allowed to continue through the topology.

The Comparator is intentionally placed after the Mapper. This means the decision is made only after the payload is transformed into the structure that truly matters for the target system. If the source changes fields that are irrelevant for the final output, the Comparator filters them out and prevents unnecessary downstream updates.

Product synchronization flow with Mapper and Comparator filtering unchanged product records before downstream processing

4. Reliability: The "Zero Data Loss" Policy #

Order transfer is a business-critical process. If an ERP is down for maintenance, you cannot afford to lose a single order.

Persistent Queues: Orders from the e-shop are safely stored in Orchesty. If the ERP doesn't respond, the message stays in the queue.
The Trash System: If a message fails repeatedly (e.g., due to a data validation error in the ERP), it moves to the Trash.
Manual Recovery: An administrator can see the failed order and fix the issue.

Error Analysis & Source Correction

In a professional integration, the Trash system serves primarily as a diagnostic and analytical hub. If a process fails - for example, due to a typo in a customer's address - Orchesty captures the error and the exact state of the data.

Instead of patching the data mid-flight within the integration layer, the administrator uses Orchesty to identify the root cause. The correction is then made directly in the E-shop (the source of truth). This fix triggers a new webhook, initiating a fresh process with correct data. This workflow ensures total data integrity: the information is accurate at the source, in the ERP, and in the fulfillment logs, preventing any future discrepancies across the ecosystem.

5. Architectural Cleanliness #

By using Derived Connectors, developers keep the visual topology clean. Instead of a messy web of mappers to handle complex ERP data structures, the logic is hidden inside a clean, reusable component.

This allows the Business Architect to see a clear flow: E-shop Order → Tax Calculation Node → ERP Order Creation.

Summary of Results #

Scalability: The same topology handles 500 or 500,000 products without code changes.
Efficiency: Delta synchronization reduces API traffic by up to 95% after the initial run.
Safety: Built-in rate limiting prevents the e-shop from crashing under high synchronization loads.
Resilience: The Trash system ensures 100% data integrity, even during server outages.

Data comparator guide — the Comparator pattern in depth, including the ready-made orchesty/comparator-worker for high-volume catalogs.
ID mapping guide — keeps stable references between e-shop and ERP without creating duplicates on the round trip.
Self-correcting integrations guide — how a topology repairs missing prerequisites itself instead of relying on cron ordering (e.g. categories before products).
Operational visibility — Trash workflow, Limiter view, and the dashboards behind the "100% data integrity" promise.