Price Scraping Robot
The problem
Job pricing and quoting can be a manual and time-consuming process, especially if you require costings from suppliers. A typical process would involve getting these costings and storing them in your data store whether a database or excel file. You would then add your business logic and manually prepare a quotation.
This repetitive process is time consuming and susceptible to human error. Furthermore, and especially in the current economic climate with volatile costs and inflation, your supplier costings need to be continuously checked for changes and updates. If the supplier has thousands of product costing configurations this may not be feasible at all, and you would need to rely on quoting every job manually.
The solution: A price scraping robot
A scraping robot can take the time and human error out of this quotation process. Website or document scraping can be used for a range of applications including testing, automation and data extraction. Pricetech offers data scraping and can scrape large amounts of pricing data directly from supplier websites or documentation for quotations, business automation, competitive analysis or general information.
Benefits:
- Automation is time saving
- Human error eliminated
- Better user experience
- Feasible to scrape large datasets
The scraping robots we develop use a C# Selenium SDK through an Azure function app. The scraper retrieves huge datasets across supplier websites for various product configurations and exports the data to an Excel sheet, data store or does some other business logic with the retrieved data such as prepare a quotation based on user inputs that are generated directly from your supplier website in realtime. You can then add your markups and service fees in code, completely automating your pricing and quotation processes.
Let's take a look at how we set up a price scraping robot for one of our recent customers. In this example we are scraping a website to automate a customers quotation process, data is drawn directly from their suppliers website for costings. Their supplier has over 4 trillion available options so the quotation process used to be manually run by sales staff who had to visit the supplier website to get the costings for each quotation. We were able to automate this process using data scraping, once the data is collected and the customers markups and service fees are added, a quotation is automatically generated and sent to the customers lead.
Tools we use:
- Visual Studio 2022 Community Edition
- Microsoft Azure Function App
- .NET Core Razor Pages Web App
- SQL Database
- Selenium SDK
- SendGrid
- IronPDF
- Microsoft Excel
Step 1:
A job starts with a lead, and in this example the customer gets leads through their website from their prospective clients who are looking for bespoke packaging boxes. We used .NET Core Razor pages for this website. Their client is looking for printed retail boxes and completes the form on the website, where we capture all the required fields that are needed for a quotation and store the details in a SQL database. After we have done this, we trigger an Azure Function app to run in the background.
Step 2:
The function app securely reads from the SQL database. The app then launches the Selenium webdriver to scrape the data from the supplier website, using the form fields we captured in Step 1 and inputs these into the 3rd party supplier website. Once we have the required costings, we apply the business logic in the function app and update the SQL database.
Step 3:
We generate a PDF quotation using the IronPDF SDK and send the quotation to the clients email address using the Sendgrid api. This all happens automatically and is estimated to save over 20 hours a week for the customer who used to have to manually quote each enquiry. Furthermore, there is no human error in the quotation process and a better user experience for the client who receives an instant quote as oopposed to having to wait for a sales representative to get back to them.