- Article
22 Pages
E-commerce search engines often fall short in delivering accurate and fair results due to their reliance on keyword matching and bias towards larger companies. This paper introduces a novel search engine architecture that integrates advanced AI techniques to address these limitations. Our approach combines Large Language Models (LLMs) for semantic query understanding and Multimodal LLMs (specifically LLaVA) for extracting detailed information from product images to enrich product descriptions. More specifically, we employ a fine-tuned version of the CLIP model to create a unified embedding of product images and descriptions, storing them in a vector database to enable efficient semantic search. Moreover, we fine-tuned the LLaVA model on e-commerce-specific data, significantly improving its performance in generating accurate product descriptions, which were also added to the vector database. The integration of computer vision (CV) techniques allows for a better interpretation of visual product attributes, while the use of vector databases facilitates rapid and relevant search results. Our system also incorporates a recommendation mechanism that leverages these advanced models to provide wider product recommendations. Our experiments demonstrate substantial improvements: the fine-tuned LLaVA model achieved an 80.57% human evaluation rating compared to 62.89% for the original model. Moreover, the overall search accuracy increased after fine-tuning the system by 25 percent points, from 70% to 95%. Our work promotes a more equitable platform for e-commerce search engines while significantly improving user satisfaction.
AI
21 October 2026









