Accurate and efficient product variant matching using the transformer architecture

Abstract

In the extremely competitive market of e-commerce, it is crucial that customers can discover products effortlessly. On many websites, potential customers are required to turn to the search engine when looking for alternative colors or sizes, as the current product page does not link to any of the product variants. Adding these links can increase sales by 5% or more, but unfortunately, a retailer does not know which variants to list because every variant is seen as an entirely different product in their database. Hence, this thesis focuses on techniques for identifying product variants and clustering algorithms to solve this problem. Product identification is investigated using TF-IDF, bi-encoders (BEs), Cross Encoders (CEs), Vision Transformers (ViTs) and the generative Large Language Models (LLMs), which are trained to measure similarity between products. Product clustering is investigated using the DBSCAN algorithm. This thesis proposes three algorithms which modify the standard algorithm to produce computationally efficient algorithms, such as refined clustering, Triangle-Inequality DBSCAN (TI-DBSCAN) and Greedy DBSCAN. In this work, CEs have achieved the highest clustering quality of near 89% Adjusted Rand Index (ARI), while TI-DBSCAN provided the best quality optimization algorithm with a mean pairwise computation reduction of ∼69%. The research leverages Squadra Machine Learning’s PowerRelate dataset, which focuses on technical subjects such as hardware stores and electrical components. The findings demonstrate significant practical implications for e-commerce platforms, showcasing that modern transformer-based artificial intelligence models are effective in accurately identifying and clustering product variants