Chapter 3: Page through a catalog with a batch

In Chapter 2 a connector fetched one product. Real syncs deal with many, so now you'll import a topology built around a batch node.

By the end you'll be able to run a batch that pages through a whole catalog and see how a mapper reshapes each record.

Prerequisites #

Nothing new to set up. Reuse the pieces from Chapter 2: the mock server is running and the Products Test (Simulation) application is installed, authorized, and active.

What's new in this chapter #

Batch. Like a connector, but the platform calls it repeatedly (one page at a time) and it emits each record as its own message (a fan-out), so the rest of the topology processes records one by one. A batch can also emit its records in different modes (one per record, in chunks, per page, or all at once); those modes are described in Concepts: Batch.

Mapper. A plain custom node whose only job is to reshape each record from the source format into a simpler target shape, the everyday pattern for adapting one system's data to another's.

Cron trigger. This topology is meant to run on a schedule, but you can trigger it manually from the UI to test it.

Import and run #

Import demo-products-test-bulk-sync.tplg.json from the demo worker's src/topologies/ folder, then Publish and Enable it.

Trigger it manually from the Cron node. Send nothing to process the full default catalog (100 products), or pass a small cap for a quick first run:

{ "maxProducts": 20 }

The batch pages through the catalog and the mapper runs once per product.

Where next #

That single run just fanned out up to 100 downstream messages at once. In Chapter 4 you'll import a topology that adds a rate limiter to this exact flow, so the platform throttles the downstream calls to a safe pace.