The project focuses on overcoming language barriers and unfamiliarity with local products. The solution proposed was a digital device attached to shopping carts, featuring AI image recognition for product identification and an easier checkout process.
Team
Tools
Figma
Photoshop
Miro
Microsoft Word
Timeline
3 Weeks
Our goal is to understand the needs of the contemporary grocery shopper, channeling this knowledge to inform the design of a refined experience for grocery store patrons that maximizes positive and empowering embodied patterns of interaction (Toombs, A., Kaziunas, E., 2023)
We started by studying and researching our problem closely, then made a prototype of our product. After testing and improving the prototype based on feedback and previous findings, we created our final product.
After having a clear idea of the pain points and goal we are trying to solve, we all decided to start brainstorming ideas and how the shopper’s flow should look.
Let’s put the barcode
The main challenge was simplifying the task of remembering and identifying produce names for shoppers. Observations showed that while people recognize produce visually and know its name in their native language, they don't necessarily recall the local U.S. names. This led to reframing the problem to focus on seamless addition of produce to the payment cart. Initially, we considered barcodes next to produce names for easy scanning. However, this approach faced the challenge of weighing produce sold in boxes and priced per pound.
Overcoming the challenge of measuring the weight of produce
Weighing machine at the end of each aisle
Shoppers have the option to weigh their produce at the end of each aisle, generating a receipt with a barcode that embeds the weight of the produce. Additionally, a digital barcode solution is considered to minimize paper waste.
Putting the weighing machine in the cart itself
We considered transforming the bottom of the small basket attached to the cart into a weighing machine, enabling users to see the weigh and feed it into the cart after scanning the item.
After testing in our studio, we found that Idea 1, involving double barcode scanning and taking produce to the aisle's end, was effort-intensive and inconvenient. Consequently, we discarded it in favor of Idea 2. This approach, integrating a weighing machine within the cart, seemed more practical and user-friendly, significantly reducing the need for shoppers to move around unnecessarily.
Wait! How will a user scan the barcode?
While initially assuming shoppers would use their phones for scanning, we observed that frequent phone use for other tasks made repeatedly opening an app like Kroger cumbersome. This led to the decision to not use mobile phones as the primary interactive device.
We briefly considered smart watches but dismissed this idea to ensure inclusivity, recognizing that not all shoppers wear or prefer watches.
The alternative proposed was to attach a digital screen to the shopping cart. The challenge was facilitating scanning through this screen, leading us to explore various ideas to address this.
Detachable digital screen
Shoppers can detach the digital screen from the cart and scan the barcode to identify the item.
Tethered and detachable barcode
We can tether the barcode with the aisle so that user can pull it and scan the barcode by putting it on top of the digital screen. We tethered it so that people don’t steal it or put it in the wrong place.
Scanner attached to barcode
We also thought of having the detachable scanner attached to a digital screen through which the shopper can scan the barcode easily.
We decided to go with the Idea-2 because of the following reasons:
Detachable digital screens can be messy at times as it might require the user to detach and attach the screen again and again. Also, shoppers need to attach and detach effectively which has its own learning curve and can be frustrating at times.
We discarded the idea-3 because we thought this would lead to addition of lot of items to the cart and this will also come with an additional capital cost.
Idea-2 seemed the most reasonable as it can be easy for a shopper to pull the barcode and scan it by putting it on the digital screen.
Jotting down our final flow
After brainstorming the ideas of all the components of our flow and performing those experience we came up with the following flow:
So, there would be a digital interactive device (similar iPad) attached to the shopping cart. The bottom of the basket inside the cart will be used as the weighing machine.
If a shopper wants to add a produce into the cart, the user will pull the barcode and put it on top of the digital screen to scan and recognize the produce. Then the user will weigh the produce by putting it into the basket, which will weigh the weight of the produce and will be added to digital interactive device attached to the cart automatically. User can then click on ‘Add to cart’ to add the item in the shopping cart. This way the user will keep on adding the items similarly and in the end can simply checkout in the shopping cart itself.
The flows that we created while jotting our solutions
Designing our digital interactive screen
To design the digital interactive screen, we decided to sketch the wireframes and flow on whiteboards. Then we started discussing each other's sketch and started brainstorming by scraping and taking some ideas of it. Taking Fig. 4 as reference we decided to design interactive screens.
Showcasing the initial wireframes
Final Wireframe
Improving on the initial wireframes illustrated in Fig 5, we realized that there was an absence of feedback when a produce was added to the cart. To bridge this gap, we introduced a buffer, signaling shoppers that the item is currently being weighed. This visual cue enhances transparency and sets clear expectations, reducing potential confusion about the process. Building on this, we further refined the checkout process: the 'Add to Cart' button now activates only once the item's weight is confirmed. This ensures that shoppers have a clear sequence of steps to follow. By calculating the price per lb based on the actual weight of the produce added to the cart, shoppers are provided with a clear and transparent breakdown of the total cost. However, to emphasize purchase completion, the discard option was designed as a secondary action. This design subtly guides shoppers towards finalizing their purchases, emphasizing transaction completion, while retaining the flexibility for them to change their mind.
Showcasing the final wireframes
After having our sketch ready, we decided to conduct usability testing to test our solution. We decided to perform a paper prototyping by attaching the sketches to the cart and changing the pages depending on his/her/them interactions with it.
Pre – Usability Testing
Prior to usability testing, we focused on determining our testing objectives. Specifically, we aimed to evaluate if our solution seamlessly integrates into the user experience and assists in recalling product names. For this, we asked shoppers to use our cart with the attached digital device to purchase two of their preferred produce items.
During the usability testing
We secured consent from each participant and briefed them on the procedure. They were informed that a small basket, positioned near the handle of the cart, would simulate a weighing machine. Participants were encouraged to verbalize their thoughts during the task. The task involved selecting two produce items for purchase.
Seema performing our task. The notebook was acting as a digital interactive device
Location: Kroger, 528 S College Avenue, Bloomington, Indiana
Post-usability testing
At the end, we thanked our participant for taking out time and helping us with the usability testing. We also observed some things during our usability testing, and this was the right time to talk about it and understand what they were thinking while performing it.
Analysis and insights
Initial Confusion in Task Execution
Seema initially struggled to choose between scanning or weighing items, feeling that scanning QR codes disrupted her usual shopping routine. She expressed that while placing items in the cart or basket felt natural, scanning barcodes seemed intrusive. This highlights the need for new technologies to blend smoothly into established consumer practices.
Perceived Unnatural Action
Mahesh found the scanning process "unnatural" and was uncertain about remembering to scan barcodes in the aisle, indicating a disconnect between this new process and regular shopping habits.
Security Concerns
Mohini raised concerns about the potential for theft with the portable checkout system, indicating the need for robust security measures to ensure the system's integrity and user trust.
General Appreciation with Reservations
Ivan valued the system's convenience and its potential to reduce reliance on employee assistance during checkout, while Seema appreciated the streamlined shopping experience, especially as it could prevent the need to revisit aisles for information.
So, after usability testing it was clear that scanning the barcode was a bit “unnatural” (Mahesh) and breaking the habitual flow of shoppers of adding items directly into their cart. To overcome this, we again brainstormed and came up with
Putting barcode on each produce
Implementing barcode stickers on produce allows for automatic scanning and weight recognition as items are added to the cart, streamlining the shopping process.
AI Image recognition
We proposed integrating a scanner on our digital device to instantly weigh and display items added to the cart. Utilizing the LLaVA model, known for superior object detection accuracy, it can distinguish similar products and identify items in complex, cluttered arrangements.
We liked Idea 2 (AI Image recognition) as compared to Idea 1 (Putting a barcode on each produce) as putting up barcode on each item can be tedious and can create problems for shop owners.
Showcasing the final wireframes
Showcasing the initial wireframes. Sketch by Pei Sun.
In our final solution, we decided to go with the AI image processing camera which will be attached to the back of the cart. The blue part will act as a weighing machine.
When a shopper adds any produce in the cart, the digital interactive device/tablet will scan the produce and will recognize, which will show up on digital interactive device. Then it will automatically calculate the weight which is shown as buffer thing in the Fig 8 right top image. So, it will finally show the weight and price. Then the shopper can click on ‘Add to cart’ then it will show up in payment checking counter. Then the device will show ‘Item added successfully.’
Impact
It was important to understand how our solution with improvements can impact the lives of our shoppers
Sense of Assurance
Shoppers on seeing the name (shoppers can choose whether they want to see the name in their native language in the digital interactive device) in their native language gives them a feeling of assurance that they have bought the same produce that they were looking for.
No More Remembering US Local Names
This was our primary pain point and now our shoppers don’t need to worry about remembering the US names. Now they don’t have to go back again to Aisle to see the name of the produce they purchased.
No More Moving Here and There
Now shoppers can directly check out. Now they don’t need to find and talk to staff members. Also, they don’t need to take out the items again during the checkout to scan and package them.
Limitations
Some constraints that we noticed with our design, and attempted resolving, are
Anti-theft system
We could conceptualize a system where the digital display’s camera actively monitors shopping activities to prevent theft. This system prompts a warning when an unscanned item is added to a cart and alerts store personnel if the warning is unheeded. While promising, we acknowledge the inherent privacy challenges. We're exploring ways to maintain a balance between the anti-theft approach and safeguarding user privacy.
We could focus on stationing a security guard at the exit doors and have the shopping cart stand nearby. This would ensure that all shoppers return their shopping carts to their rightful location and allow the guard to check on any suspicious activity.
Future work
In the limited time frame of the project, we concentrated on addressing a key issue using straightforward features. Additionally, having identified other pain points during our observation, we aim to outline potential strategies for enhancing the shopping experience
Personalized Interface
Once the user logs in, the app will apply the language set in their Kroger account, ensuring everything they see is easy to understand in advance.
Easy Checkout
Once the user logs in, if they have linked their credit card to the Kroger app, they can seamlessly complete the shopping using only the in-store monitor/app.
Smart Shopping List
If the user puts their shopping list on their phone with a defined format, they can use the in-store app to scan it to show on the monitor. It provides a side-by shopping list, which helps shoppers to aim for the items they need accurately and efficiently.
Floor Information
If we merge detailed store maps into the app, shoppers can find items with clear instructions, removing the redundant searching time and creating a better experience.
If these solutions are implemented, the in-store shopping experience will be much improved to provide shoppers with a smoothen embodied shopping experience
Conducted observations and contextual inquiries at Kroger, documenting public behaviors and interactions.
Conducted in-depth secondary research to gather valuable insights.
Collaborated with the team to develop an affinity diagram, to refine the problem space.
Designed initial wireframe and iteratively enhanced this to produce the final prototype.
Conducted usability testing with an international student to assess the prototype and gather feedback to inform further refinements and improvements.
Co-authored the digital documentation.