30% Operational Efficiency Gain for an Automotive eCommerce Web Data Pipeline

The client is a large-scale eCommerce operation focused exclusively on the automotive aftermarket — parts, accessories, and consumables sold directly to consumers and trade buyers across North America. With around 300 employees and a catalogue running into several million SKUs, the business sits in a segment where data quality and freshness are not operational niceties — they are the product.

Pricing accuracy, stock availability, competitor positioning, and vehicle fitment data all need to reflect what is actually happening across dozens of supplier and competitor sites at any given moment. The internal engineering team understood this well. What they needed was a data layer that could deliver on that requirement reliably, at scale, and without consuming the majority of their capacity to maintain it.

Before approaching us, the team had been running a web data pipeline built incrementally in-house over several years. It had carried the business through a period of real growth and was, in functional terms, working. The issue was the cost of keeping it working.

Automotive eCommerce places specific demands on web data collection. Supplier catalogues change frequently. Vehicle fitment records — specifying which part fits which make, model, and year combination — arrive structured differently across sources and need to be normalised before they are usable. A customer ordering a part that does not fit their vehicle generates a return, a refund, a complaint, and a lost relationship. Getting compatibility data right is not optional.

The pipeline had four identifiable weak points:

• Collection failures — A significant share of scheduled scraping jobs were failing on the first attempt. Automated retries absorbed some of the impact, but failures that required manual investigation were pulling engineers away from development work on a near-daily basis.
• Schema fragility — When supplier sites updated their layouts — which happened regularly and without notice — extraction logic broke. The time between a source changing and the fix going live averaged several days, during which affected data went stale.
• Fitment data normalisation — Compatibility records from different sources arrived in incompatible formats. The post-processing step that normalised them into the internal schema had grown organically and had become one of the more fragile parts of the pipeline.
• Infrastructure maintenance — The team was operating and monitoring collection servers whose sole purpose was to run scraping jobs. That responsibility sat on the engineering team's plate alongside everything else.

The combined effect was that a meaningful share of skilled engineering time was going into maintaining existing infrastructure rather than building anything new. The team had raised this internally. What they needed was a realistic alternative — not a managed service that handed control to a third party, but a properly engineered data API they could own, call on demand, and integrate directly into their existing systems.

The engagement started with a technical scoping exercise covering the client's catalogue structure, their existing pipeline architecture, and the specific data types they needed: real-time pricing, stock levels, structured product specifications, and vehicle fitment records across their supplier and competitor set.

We designed and delivered a custom web scraping API built specifically around their data requirements. The API handles all collection logic — requesting pages, parsing structured and semi-structured content, managing rate limits and request scheduling, detecting and adapting to layout changes on source sites — and exposes clean, consistently structured endpoints that the client's engineering team can call directly from their own systems.

The architecture gave the client full control over when and how data is requested, without requiring them to manage anything below the API layer. Their engineers work with predictable, well-documented endpoints. The complexity of keeping extraction logic current with source-side changes sits with us, not with them.

The rollout was phased. High-churn data — pricing and stock levels — went first, where the freshness gains were most immediately valuable to the commercial team. Structured product data and fitment records followed once the client's team had validated output quality and integrated the endpoints into their catalogue management workflows.

Vehicle compatibility data received particular attention during design. Rather than delivering raw fitment records in source format and leaving normalisation to the client, the API returns compatibility data in a single consistent schema regardless of how it arrived at the source. The client's post-processing step, which had been a persistent maintenance burden, was effectively replaced by the API's output contract.

Client name and identifying details have been anonymised at the client's request.