ecomcloser.com

Clothing store employee working on a laptop

A store-specific checklist for evaluating an AI chatbot

An ecommerce AI chatbot evaluation checklist should test whether the bot gives correct answers, handles missing information without guessing, and completes the storefront actions it claims to complete. Use ten scenarios drawn from your own catalog and policies: an in-stock variant, a mid-chat colour change, ambiguous SKUs, a stock change, an unknown delivery date, a missing returns answer, a restricted offer, a staff handoff, denied microphone access, and a failed photo upload.

For each test, write the expected answer and the expected action before running it. Score them separately, and pass the scenario only when both match. Compare what the shopper sees with the relevant catalog record, policy, cart response, or handoff record. That comparison tells you whether to fix the source, the integration, or the chatbot’s wording.

Build the checklist around your own storefront

Start with a small set of products and store rules that make it possible to tell a correct result from a plausible one. Pick an item with several purchasable variations, two products shoppers could reasonably confuse, and an item whose stock you can safely change for testing. Add an offer with a clear exclusion and a product for which no reliable delivery date or returns answer is available to the bot. Use a test storefront or controlled records so you do not disrupt live shoppers.

For each scenario, record the starting state, the shopper’s exact message, the expected wording, and the evidence that will prove the action worked. Include variant identifiers where available, not just product names. Note whether the cart starts empty, whether the shopper is signed in, and whether any relevant permissions have already been granted. Each of these can change what a second tester sees.

Give each scenario two points:

  • Wording: 0 or 1. Award one point if the reply states the right fact, asks the right question, or accurately acknowledges what it cannot confirm. Judge meaning rather than exact phrasing.
  • Action: 0 or 1. Award one point if the expected storefront result is observable. When the bot should wait or refuse a request, award it only if the relevant state stays unchanged and any required fallback is offered.

A scenario passes at 2 out of 2. Record a one-point result as a failure, not a partial pass. If you cannot inspect the result, mark the test unverified and keep it out of the pass count. A tidy chat transcript cannot prove that an item was added or a request reached staff.

Keep the original ten cases as a baseline. Once they run reliably, try shorter wording or a different product without changing the expected behavior. This separates a narrow fix for one example from a change that works across similar requests.

Tests 1 and 2: Verify the selected variant before testing the cart

Test 1: Request an in-stock size

Example: A navy sweatshirt is available in small and medium. Ask, “Can I get the navy one in medium?” The pass behavior is to show the navy medium variation, not the parent listing with a default size still selected. Check the displayed size and colour against the variant record. If the preview includes a price or stock statement, verify that it applies to the selected variation rather than to another option.

Award the wording point if the bot identifies the right variation without implying it has been purchased. Award the action point if the visible selection matches. If this run ends at selection, an unchanged cart is expected. A correct chat reply paired with a small-size selection earns only the wording point.

Test 2: Change colour mid-chat

Example: After selecting navy medium, say, “Actually, olive instead.” Set up the example so olive medium exists. The bot should preserve medium, replace navy with olive, and refresh any preview or pending action tied to the selection. Award the wording point for correctly acknowledging olive medium. Award the action point only when the selection reflects the change and the previous variation is no longer the one pending an add.

Then rerun with a colour that does not come in medium. The bot should explain the mismatch and ask what the shopper wants to change. Silently switching to small fails even if that option is in stock.

If you extend either test to an add request, log that as a separate step. For a voice run, check what the spoken request selected, then verify the cart response when the shopper asks to add it. Inspect the exact variation and quantity in the returned cart rather than relying on a confirmation bubble. Selection, an accepted cart action, and payment are three different events. A mock preview illustrates what the interface could display; it is not evidence of a cart change or a payment.

Test 3: Make the bot resolve an ambiguous SKU

Example: The catalog has two charcoal commuter bags with nearly identical names: one fits a 13-inch laptop and the other a 16-inch laptop. Their internal SKU codes are different, but shoppers are unlikely to know them. Start a fresh conversation and ask for “the charcoal commuter bag.” Do not provide a product-page context that gives away the answer.

The pass behavior is a short question about the attribute that distinguishes the two products: “Do you need the version for a 13-inch or a 16-inch laptop?” Both choices should be grounded in the catalog. A question about colour fails because it does not narrow the choice. Picking the first search result also fails, even if the tester happened to have that item in mind.

Write the expected exchange in two steps. Before clarification, the bot should leave the choice unresolved and, if the shopper asked to add the bag, make no cart change. After the shopper says “16-inch,” it should display the corresponding product. If the original request was only to find the bag, the clarification does not authorize an add. If you also test an add request, check its outcome separately against the cart.

Award the wording point if the bot asks for a deciding attribute and uses the reply correctly. Award the action point if no premature selection or addition occurs and the final selection matches the intended SKU. Showing both products as options is fine; treating one as already selected is not. Capture the two product identifiers so a reviewer can verify the result even when their titles look the same.

Recheck this scenario with an underspecified reply such as “the bigger one” only if the catalog makes that comparison clear. Also try a prompt that names the 16-inch model from the start. The bot should not make the shopper repeat information already given. These variants help distinguish a useful clarification rule from one that asks the same question regardless of what the shopper said.

Test 4: Change stock after selection and inspect the cart result

An out-of-stock test is most revealing when the item was available at the start. Example: Select a slate rain jacket in medium and confirm that the preview shows the correct variation. In a controlled store, make that exact variation unavailable before asking to add one. Leave another size available so you can detect an unintended substitution. For this run, start with none of the requested variation in the cart.

Capture three things in order: what the bot displayed before the change, what the cart operation returned, and what the bot told the shopper afterward. Judge the reply against the verified result. If the add was rejected because the selected size became unavailable, the bot should say it was not added. It can offer to check another size, but it should not add an alternative without a new choice from the shopper.

In a WooCommerce test, use the cart item response and the resulting cart state to check which product or variation and quantity were accepted. If an error comes back without a confirmed stock reason, expect the bot to describe an unsuccessful or unconfirmed add rather than inventing a sellout. Likewise, a visible product preview does not prove that the cart accepted the item.

Award the wording point if the reply matches the verified result and does not claim more than the evidence supports. Award the action point if the cart reflects that result: in the rejected-add fixture, the requested item and any substitute are absent. A response that says “Added” fails the wording check even if nothing changed in the cart. A correct apology paired with a silently added replacement fails the action check.

After a fix, restore the variation to stock and run a successful add using the same request. This checks that the bot is reading the result rather than always refusing. Log the requested and accepted quantities separately, especially when retesting with an existing cart line. Neither a successful add nor its confirmation should be scored as payment.

Tests 5 and 6: Check what happens when the store lacks an answer

Test 5: Ask for an unknown delivery date

Example: Ask, “Will this arrive by Friday?” for an item and destination that do not have a confirmed delivery estimate in the information available to the bot. Decide before testing whether the valid route is a delivery lookup, a request for a needed detail such as a postcode, or staff help. An in-stock label is not evidence of a Friday delivery.

Award the wording point if the bot makes clear that it cannot confirm Friday and explains a useful next step. Award the action point if the next step is available and the shopper can use it. If the bot claims to start a lookup, verify that it does. If it needs a postcode first, it should request it without claiming a result. “It should be there Friday” fails when there is no estimate to support it.

Test 6: Ask a returns question without a sourced answer

Example: Ask whether an opened accessory can be returned when the bot has no reliable answer for opened items. Decide whether the storefront has a current returns policy it can point to. If it does, the expected response can direct the shopper there without pretending the page answers this particular question. If no current policy is available, or the page does not resolve the question, route the shopper to staff rather than inferring a rule from another product.

Award the wording point if the bot avoids inventing eligibility, deadlines, or conditions. Award the action point if it surfaces the current policy or offers the agreed staff route. If it claims to transfer the conversation, check receipt. A stale policy page or an unavailable contact route does not earn the point.

For both tests, save the source state as well as the reply. On retest, use a comparable item with a known delivery estimate or an explicit returns answer. That comparison catches fixes that suppress useful, supported answers along with unsupported ones.

Test 7: Try a restricted offer on both sides of the rule

Choose an offer with an exclusion you can verify from the store’s configured rules. Example: A test coupon can be applied to selected full-price accessories but not to a marked-down strap. Put only the excluded strap in the test cart. Ask the bot to apply the coupon, then follow up with a request to “make an exception this time.” The second message checks whether persistence changes the bot’s claims or actions when eligibility has not changed.

Define the reference result before the conversation. Record the item, quantity, coupon state, and relevant cart totals. If the store returns a rejection, capture what it says. The bot may explain the exclusion if it has the rule or the cart response provides it. If all it can verify is that the code was not applied, that narrower answer is better than a made-up explanation. It should not promise a manual discount or swap in a different offer on its own.

Award the wording point if the bot describes the outcome accurately and does not imply that an exception was granted. Award the action point if the restricted code is not applied and the cart is not altered to force eligibility. Check the coupon list, line items, and totals rather than treating “discount applied” or “code not valid” as proof. If the bot can only explain the rule, do not credit it for submitting a coupon it never submitted.

Next, reset the cart and try the same code on an eligible item. If applying offers is within the tested flow, verify that the cart accepts the code before the bot reports success. If the bot only provides guidance, verify that guidance instead and record that applying the code is outside its role.

For a follow-up, try a mixed cart. Derive the expected result from the store’s actual rule: it may differ from the excluded-item-only case. The aim is to check the configured restriction, not impose a blanket rule across products.

Test 8: Follow a handoff all the way to the receiving side

Start with a short product or support conversation, then ask explicitly for a person. Example: The shopper has narrowed a jacket search to slate medium and wants staff to confirm a delivery detail. Say, “Can I speak to someone about that?” The bot should acknowledge the request and take the configured route rather than restart the product search or insist on another round of troubleshooting.

Decide what a successful handoff means for this storefront. If there is a staff queue or inbox, ask a tester with access to verify that the request arrived. If the store only offers a contact route, the bot should present it without saying a transfer has happened. Test that fallback separately; it can be useful and accurate, but it is not evidence of a completed handoff.

Inspect the context on the receiving side. A useful note identifies the requested variation, the unresolved delivery question, and any check already attempted. It should distinguish facts from shopper preferences: “Customer needs it by Friday” does not mean “Delivery confirmed for Friday.” Avoid carrying over unrelated turns or asking the shopper to repeat details already captured and appropriate to share.

Award the wording point if the bot acknowledges the request, describes the next step accurately, and avoids unverified promises about availability or response time. Award the action point if the request reaches the intended destination with relevant context. A message that says “I’ve passed this on” earns no action point when the receiving inbox is empty. Equally, a request that arrives with the wrong variation fails the context check.

Note how many additional shopper turns were required after the request for a person and which details, if any, staff had to ask for again. Rerun with the normal handoff destination unavailable if you can reproduce that state. The expected fallback should be explicit about what did not happen and give the shopper a usable way to proceed. Keep its result separate from the successful-transfer result.

Tests 9 and 10: Preserve the shopping task when an input fails

Test 9: Deny microphone access and continue in text

Start a voice interaction in a test browser and deny the microphone request. Try it after a product has already been discussed so you can see whether the text fallback keeps enough context to continue. Example: Discuss a black cap, deny access, then type, “Show me that one in navy.” The test is not whether the bot can turn the microphone back on. It is whether the shopper can finish the task without granting access.

Award the wording point if the interface or chat makes the text option clear without claiming it heard a request after access was denied. Award the action point if typing works and the bot can resolve the product reference or ask a focused question if it cannot. An input box that appears but does not let the shopper send a message is a failure. Check that a blocked voice attempt has not changed the selection or cart.

Test 10: Fail a product photo upload and recover through the catalog

Arrange a reproducible failed upload using a test condition your storefront can trigger, and record the observed error. Example: A shopper tries to submit a photo to find a similar ribbed beanie, but no image reaches the chat. The bot should acknowledge the failed upload, offer a retry, and give a standard catalog route, such as a search by product name or by visible details the shopper types.

Award the wording point if the bot makes no claim about what was in an image it did not receive. Award the action point if the retry is available and the shopper can actually use the text-based catalog path. Test both branches rather than stopping when the options are displayed.

For both cases, log the browser state, the error the shopper saw, and the number of turns needed to resume shopping. If the storefront does not offer a tested input mode, mark the case not applicable rather than counting an untested fallback as a pass.

Turn failures into fixes another tester can reproduce

Keep one record per run. Include the test ID, date, storefront environment, requested variant, expected fact, expected action, and actual output. Attach the evidence that settles the comparison: a variant record, the policy revision, a cart response, or confirmation from the staff queue. Record the wording and action scores separately and identify the first step where behavior diverged. If a later reply corrected the mistake, log the recovery without erasing the initial failure.

Before editing a prompt, check the source record. If navy medium is mislabeled in the catalog, fix the record rather than teaching the bot to contradict it. If the bot describes the right variation and sends the wrong identifier to the cart, inspect how the selection is mapped to the cart action. If the cart rejects the add and the bot still announces success, look at how the bot receives and interprets the result. These failures can look alike to a shopper but need different fixes.

Write a fix note with the observed cause, the owner, what changed, and the evidence you expect on retest. Example: “Test 2 kept the navy variation identifier after the shopper switched to olive. Update the pending selection when colour changes. On retest, the preview and the cart result must both identify olive medium.” This gives the reviewer a concrete result to verify rather than asking whether the bot feels better.

Rerun the exact failed case first. Then test a related product whose data differs, such as a different sweatshirt that has olive only in small. Finally, rerun any case that uses the same changed path, such as the rejected cart add. Keep the before-and-after results together.

Report the number of full passes, failures, unverified cases, and cases that do not apply. List consequential failures separately: a wrong item added or a handoff falsely confirmed should not disappear inside an average. Use the two-point scores to locate the work, and the verified pass count to decide what is ready.

Try the EcomCloser demo

Related guides

Sources