De-duplication via Batch File Delivery


While we've long had the ability to allow lead generators to deduplicate leads prior to submission using our pre-ping functionality in SuppressionList, this functionality requires real-time querying prior to real-time submission. We've found that sometimes lead generators have unique business processes which require identifying known duplicates prior to implementing these processes, ensuring they deliver the highest quality lead data.

To accommodate these unique workflows, you can build a flow allowing users to POST or upload a CSV to LeadConduit, check the client's SuppressionList for known duplicates, and return a CSV with the results for each line item using Batch File Delivery. Using this basic flow, duplicates can be checked quickly and easily without implementing a real-time query from a form.

This article assumes:

  • You already have a SuppressionList for real-time pre-ping of duplicates. If not, you'll want to review the SuppressionList documentation for creating and populating a list.


1) Create a separate flow for your Batch De-duplication

  • Add the Sources you want to offer Batch Deduplication
  • Update the Fields section, removing all fields except for the field(s) you are deduplicating against (typically email or phone number)

2) Add the SuppressionList Marketplace Integration to your Flow

  • You'll need the List Name from your SuppressionList where your duplicate list is maintained.sl_dedupe.png
  • Select the SuppressionList Query Item option from the Integration list
  • Add the List Name and Field you are De-duplicating against to the appropriate sectionsquery_item.png

3) Add and Configure the Batch File Delivery Marketplace Integration to your Flow

  • Select Batch File Delivery from the Add Delivery options
  • Use the following recommended settings for your first Batch File Delivery


NOTE You will have to create a new batch file delivery configuration for each Vendor, but you will only need one step in your flow.


File Settings - Include Hour and Minute in the file name



Column Settings - Use the Select fields from Flow option and then pare it down to "email" or "phone_1" and "suppressionlist.query_id.found"



Delivery Schedule - Set to Every 15 minutes



Delivery Method - Select Email and add the email address(es) of the Vendor



4) Edit the Batch File Delivery Step to set Source Conditions

You need to add a conditional statement on the Batch File Configuration mapping so only leads from a specific Vendor are sent to the Batch File Delivery created for that Vendor.



Save your Flow!


5) Create more Batch File Configurations

  1. Switch to the Batches view under Libraries
  2. Find the Batch File Configuration you just created and Make a Copycopy_batch.png
  3. Name the New Batch appropriately and step through the configuration. The only other setting you will need to change is the To field to the appropriate vendor email address(es).update_email.png
  4. Save the new Batch File Configuration

6) Map the additional Batch File Configurations to the Flow

  1. Open and Edit your Batch De-Duplication flow, expand the Batch File Delivery Step, and click Edit Field Mappings
  2. Add a new field mapping line (or copy the existing Batch File Configuration line) and add the appropriate settings, so each Vendor/Source is delivering to the appropriate Batch File Configurationcomplex_mapping.png


Save the Flow!


7) Send the Submission Docs to Vendors

Now you can provide each vendor with Submission Docs wherein they can either upload a CSV or post to the LeadConduit endpoint. The Vendor will receive an email within 15 minutes with a CSV attached.


8) What Vendors Need to Know

The file Vendors receive will contain two columns:

  • The primary key provided (Email or Phone Number)
  • The SuppressionList Item Found result (true or false)




email suppressionlist.query_item.found Duplicate?  false NO true YES



Was this article helpful?
0 out of 0 found this helpful
Have more questions? Submit a request


You must be logged in to comment.