Embodied Shopping

MS HCI/d | I541 Project

2023

The project focuses on overcoming language barriers and unfamiliarity with local products. The solution proposed was a digital device attached to shopping carts, featuring AI image recognition for product identification and an easier checkout process.

Team

Sriram Hemanth Kumar

Pei Sun

Monisha Vimal Manshukani

Ravi Jangir

Duration

3 Weeks

Problem

Our goal is to understand the needs of the contemporary grocery shopper, channeling this knowledge to inform the design of a refined experience for grocery store patrons that maximizes positive and empowering embodied patterns of interaction (Toombs, A., Kaziunas, E., 2023)

Design Journey

We started by studying and researching our problem closely, then made a prototype of our product. After testing and improving the prototype based on feedback and previous findings, we created our final product.

Week 1

Identify Project Goal

Narrowing down target user base and finding the potential shoppers for doing user interviews.

Narrowing down target user base and finding the potential shoppers for doing user interviews.

Week 1 - 2

Observation

1 on 1 user interviews to understand user’s mindset of using the wearable devices and finding their initial ideas about faking data.

1 on 1 user interviews to understand user’s mindset of using the wearable devices and finding their initial ideas about faking data.

Week 2

Secondary research

Affinity Diagramming

Affinity Diagramming

Week 2

Define Problem Statement

Based on the primary research and secondary research to define the problem statements.

Based on the primary research and secondary research to define the problem statements.

Week 2 - 3

Sketch for Solutions

Quickly sketching our ideas based on the problem statements.

Quickly sketching our ideas based on the problem statements.

Week 3

Prototype

Based on usability testing to update the low-fidelity wireframe.

Based on usability testing to update the low-fidelity wireframe.

Ideation

After having a clear idea of the pain points and goal we are trying to solve, we all decided to start brainstorming ideas and how the shopper’s flow should look.

Let’s put the barcode

The main challenge was simplifying the task of remembering and identifying produce names for shoppers. Observations showed that while people recognize produce visually and know its name in their native language, they don't necessarily recall the local U.S. names. This led to reframing the problem to focus on seamless addition of produce to the payment cart. Initially, we considered barcodes next to produce names for easy scanning. However, this approach faced the challenge of weighing produce sold in boxes and priced per pound.

Overcoming the challenge of measuring the weight of produce

Weighing machine at the end of each aisle

Shoppers have the option to weigh their produce at the end of each aisle, generating a receipt with a barcode that embeds the weight of the produce. Additionally, a digital barcode solution is considered to minimize paper waste.

Putting the weighing machine in the cart itself

We considered transforming the bottom of the small basket attached to the cart into a weighing machine, enabling users to see the weigh and feed it into the cart after scanning the item.

After testing in our studio, we found that Idea 1, involving double barcode scanning and taking produce to the aisle's end, was effort-intensive and inconvenient. Consequently, we discarded it in favor of Idea 2. This approach, integrating a weighing machine within the cart, seemed more practical and user-friendly, significantly reducing the need for shoppers to move around unnecessarily.

Wait! How will a user scan the barcode? 

While initially assuming shoppers would use their phones for scanning, we observed that frequent phone use for other tasks made repeatedly opening an app like Kroger cumbersome. This led to the decision to not use mobile phones as the primary interactive device.


We briefly considered smart watches but dismissed this idea to ensure inclusivity, recognizing that not all shoppers wear or prefer watches.


The alternative proposed was to attach a digital screen to the shopping cart. The challenge was facilitating scanning through this screen, leading us to explore various ideas to address this.

Detachable digital screen

Shoppers can detach the digital screen from the cart and scan the barcode to identify the item. 

Tethered and detachable barcode

We can tether the barcode with the aisle so that user can pull it and scan the barcode by putting it on top of the digital screen. We tethered it so that people don’t steal it or put it in the wrong place. 

Scanner attached to barcode

We also thought of having the detachable scanner attached to a digital screen through which the shopper can scan the barcode easily. 

We decided to go with the Idea-2 because of the following reasons: 


  • Detachable digital screens can be messy at times as it might require the user to detach and attach the screen again and again. Also, shoppers need to attach and detach effectively which has its own learning curve and can be frustrating at times. 

  • We discarded the idea-3 because we thought this would lead to addition of lot of items to the cart and this will also come with an additional capital cost. 

  • Idea-2 seemed the most reasonable as it can be easy for a shopper to pull the barcode and scan it by putting it on the digital screen. 

Jotting down our final flow 

After brainstorming the ideas of all the components of our flow and performing those experience we came up with the following flow: 


So, there would be a digital interactive device (similar iPad) attached to the shopping cart. The bottom of the basket inside the cart will be used as the weighing machine.  


If a shopper wants to add a produce into the cart, the user will pull the barcode and put it on top of the digital screen to scan and recognize the produce. Then the user will weigh the produce by putting it into the basket, which will weigh the weight of the produce and will be added to digital interactive device attached to the cart automatically. User can then click on ‘Add to cart’ to add the item in the shopping cart. This way the user will keep on adding the items similarly and in the end can simply checkout in the shopping cart itself. 

The flows that we created while jotting our solutions  

Designing our digital interactive screen 

To design the digital interactive screen, we decided to sketch the wireframes and flow on whiteboards. Then we started discussing each other's sketch and started brainstorming by scraping and taking some ideas of it. Taking Fig. 4 as reference we decided to design interactive screens. 

Showcasing the initial wireframes 

Final Wireframe 

Improving on the initial wireframes illustrated in Fig 5, we realized that there was an absence of feedback when a produce was added to the cart. To bridge this gap, we introduced a buffer, signaling shoppers that the item is currently being weighed. This visual cue enhances transparency and sets clear expectations, reducing potential confusion about the process. Building on this, we further refined the checkout process: the 'Add to Cart' button now activates only once the item's weight is confirmed. This ensures that shoppers have a clear sequence of steps to follow. By calculating the price per lb based on the actual weight of the produce added to the cart, shoppers are provided with a clear and transparent breakdown of the total cost. However, to emphasize purchase completion, the discard option was designed as a secondary action. This design subtly guides shoppers towards finalizing their purchases, emphasizing transaction completion, while retaining the flexibility for them to change their mind. 

Showcasing the final wireframes 

Usability Testing 

After having our sketch ready, we decided to conduct usability testing to test our solution. We decided to perform a paper prototyping by attaching the sketches to the cart and changing the pages depending on his/her/them interactions with it. 

Pre – Usability Testing 

Prior to usability testing, we focused on determining our testing objectives. Specifically, we aimed to evaluate if our solution seamlessly integrates into the user experience and assists in recalling product names. For this, we asked shoppers to use our cart with the attached digital device to purchase two of their preferred produce items.

During the usability testing 

We secured consent from each participant and briefed them on the procedure. They were informed that a small basket, positioned near the handle of the cart, would simulate a weighing machine. Participants were encouraged to verbalize their thoughts during the task. The task involved selecting two produce items for purchase.

Seema performing our task. The notebook was acting as a digital interactive device 
Location: Kroger, 528 S College Avenue, Bloomington, Indiana 

Post-usability testing 

At the end, we thanked our participant for taking out time and helping us with the usability testing. We also observed some things during our usability testing, and this was the right time to talk about it and understand what they were thinking while performing it. 

Analysis and insights 

Initial Confusion in Task Execution

Seema initially struggled to choose between scanning or weighing items, feeling that scanning QR codes disrupted her usual shopping routine. She expressed that while placing items in the cart or basket felt natural, scanning barcodes seemed intrusive. This highlights the need for new technologies to blend smoothly into established consumer practices.

Perceived Unnatural Action

Mahesh found the scanning process "unnatural" and was uncertain about remembering to scan barcodes in the aisle, indicating a disconnect between this new process and regular shopping habits.

Security Concerns

Mohini raised concerns about the potential for theft with the portable checkout system, indicating the need for robust security measures to ensure the system's integrity and user trust.

General Appreciation with Reservations

Ivan valued the system's convenience and its potential to reduce reliance on employee assistance during checkout, while Seema appreciated the streamlined shopping experience, especially as it could prevent the need to revisit aisles for information.

Improvements

So, after usability testing it was clear that scanning the barcode was a bit “unnatural” (Mahesh) and breaking the habitual flow of shoppers of adding items directly into their cart. To overcome this, we again brainstormed and came up with

Putting barcode on each produce

Implementing barcode stickers on produce allows for automatic scanning and weight recognition as items are added to the cart, streamlining the shopping process.

AI Image recognition

We proposed integrating a scanner on our digital device to instantly weigh and display items added to the cart. Utilizing the LLaVA model, known for superior object detection accuracy, it can distinguish similar products and identify items in complex, cluttered arrangements.

We liked Idea 2 (AI Image recognition) as compared to Idea 1 (Putting a barcode on each produce) as putting up barcode on each item can be tedious and can create problems for shop owners.  

Final Solution

Showcasing the final wireframes

Showcasing the initial wireframes. Sketch by Pei Sun.

In our final solution, we decided to go with the AI image processing camera which will be attached to the back of the cart. The blue part will act as a weighing machine.


When a shopper adds any produce in the cart, the digital interactive device/tablet will scan the produce and will recognize, which will show up on digital interactive device. Then it will automatically calculate the weight which is shown as buffer thing in the Fig 8 right top image. So, it will finally show the weight and price. Then the shopper can click on ‘Add to cart’ then it will show up in payment checking counter. Then the device will show ‘Item added successfully.’  

Conclusion

Impact 

It was important to understand how our solution with improvements can impact the lives of our shoppers

Sense of Assurance

Shoppers on seeing the name (shoppers can choose whether they want to see the name in their native language in the digital interactive device) in their native language gives them a feeling of assurance that they have bought the same produce that they were looking for. 

No More Remembering US Local Names

This was our primary pain point and now our shoppers don’t need to worry about remembering the US names. Now they don’t have to go back again to Aisle to see the name of the produce they purchased. 

No More Moving Here and There

Now shoppers can directly check out. Now they don’t need to find and talk to staff members. Also, they don’t need to take out the items again during the checkout to scan and package them.

Limitations

Some constraints that we noticed with our design, and attempted resolving, are

Anti-theft system

  • We could conceptualize a system where the digital display’s camera actively monitors shopping activities to prevent theft. This system prompts a warning when an unscanned item is added to a cart and alerts store personnel if the warning is unheeded. While promising, we acknowledge the inherent privacy challenges. We're exploring ways to maintain a balance between the anti-theft approach and safeguarding user privacy.

  • We could focus on stationing a security guard at the exit doors and have the shopping cart stand nearby. This would ensure that all shoppers return their shopping carts to their rightful location and allow the guard to check on any suspicious activity.  

Future work 

In the limited time frame of the project, we concentrated on addressing a key issue using straightforward features. Additionally, having identified other pain points during our observation, we aim to outline potential strategies for enhancing the shopping experience

Personalized Interface

Once the user logs in, the app will apply the language set in their Kroger account, ensuring everything they see is easy to understand in advance.

Easy Checkout

Once the user logs in, if they have linked their credit card to the Kroger app, they can seamlessly complete the shopping using only the in-store monitor/app. 

Smart Shopping List

If the user puts their shopping list on their phone with a defined format, they can use the in-store app to scan it to show on the monitor. It provides a side-by shopping list, which helps shoppers to aim for the items they need accurately and efficiently. 

Floor Information

If we merge detailed store maps into the app, shoppers can find items with clear instructions, removing the redundant searching time and creating a better experience.

If these solutions are implemented, the in-store shopping experience will be much improved to provide shoppers with a smoothen embodied shopping experience 

Contribution

  • Conducted observations and contextual inquiries at Kroger, documenting public behaviors and interactions.

  • Conducted in-depth secondary research to gather valuable insights. 

  • Collaborated with the team to develop an affinity diagram, to refine the problem space.

  • Designed initial wireframe and iteratively enhanced this to produce the final prototype.

  • Conducted usability testing with an international student to assess the prototype and gather feedback to inform further refinements and improvements.

  • Co-authored the digital documentation. 

References

  1. Andrade, M. S. (2006). International students in English-speaking universities. Journal of Research in International Education, 5(2), 131-154. 

  2. Brown, L., Edwards, J., & Hartwell, H. (2010). A taste of the unfamiliar. Appetite, 54(1), 202-207. 

  3. Klepek, M., & Bauero, R. (2020). In-store vs. online shopping. Journal of Consumer Behavior, 10(4), 234-245. 

  4. Klepek, M., & Bauero, R. (2020). Why do retail customers hesitate for shopping grocery online? Technological and Economic Development of Economy, 26(6), 1444-1462. 

  5. Krishna, A. (2012). An integrative review of sensory marketing: Engaging the senses to affect perception, judgment and behavior. Journal of Consumer Psychology, 22(3), 332-351. 

  6. Krishna, A. (2012). An integrative review of sensory marketing. Journal of Consumer Psychology, 22(3), 333-352. 

  7. Krishna, A., & Schwarz, N. (2014). Sensory marketing, embodiment, and grounded cognition: A review and introduction. Journal of Consumer Psychology, 24(2), 159-168. 

  8. LLaVA: Large Language and Vision Assistant. https://llava-vl.github.io/ 

  9. Merleau-Ponty, M. (1962). Phenomenology of Perception. Routledge & Kegan Paul. 

  10. Midjourney 2023-10-09 02:58:05 - apple, line art, white background [Midjourney AI Generated Image]. 

  11. Peck, J., & Childers, T. L. (2003). Individual differences in haptic information processing: The “need for touch” scale. Journal of Consumer Research, 30(3), 430-442. 

  12. Poyrazli, S., & Lopez, M. D. (2007). International students and discrimination. The Journal of Psychology, 141(3), 263-280. 

  13. Swagler, M. A., & Ellis, M. V. (2003). Taiwanese student adjustment. Journal of Counseling Psychology, 50(4), 420-437. 

  14. Toombs, A., Kaziunas, E. (2023). Project 2: Embodied shopping. Human-Computer Interaction Program, Indiana University Bloomington.

  15. Turner, Y., & Thompson, E. (2014). International student migrations. In Y. Turner (Ed.), Internationalizing the University (pp. 167-188).

  16. Palgrave Macmillan UK. Yakhlef, A. (2015). The corporeality of consumption: embodied cognizance and sensation. Marketing Theory, 15(2), 135. https://journals.sagepub.com/doi/10.1177/1470593115569016 

Embodied Shopping

MS HCI/d | I541 Project

2023

The project focuses on overcoming language barriers and unfamiliarity with local products. The solution proposed was a digital device attached to shopping carts, featuring AI image recognition for product identification and an easier checkout process.

Team

Sriram Hemanth Kumar

Pei Sun

Monisha Vimal Manshukani

Ravi Jangir

Duration

3 Weeks

Problem

Our goal is to understand the needs of the contemporary grocery shopper, channeling this knowledge to inform the design of a refined experience for grocery store patrons that maximizes positive and empowering embodied patterns of interaction (Toombs, A., Kaziunas, E., 2023)

Design Journey

We started by studying and researching our problem closely, then made a prototype of our product. After testing and improving the prototype based on feedback and previous findings, we created our final product.

Week 1

Identify Project Goal

Narrowing down target user base and finding the potential shoppers for doing user interviews.

Narrowing down target user base and finding the potential shoppers for doing user interviews.

Week 1 - 2

Observation

1 on 1 user interviews to understand user’s mindset of using the wearable devices and finding their initial ideas about faking data.

1 on 1 user interviews to understand user’s mindset of using the wearable devices and finding their initial ideas about faking data.

Week 2

Secondary research

Affinity Diagramming

Affinity Diagramming

Week 2

Define Problem Statement

Based on the primary research and secondary research to define the problem statements.

Based on the primary research and secondary research to define the problem statements.

Week 2 - 3

Sketch for Solutions

Quickly sketching our ideas based on the problem statements.

Quickly sketching our ideas based on the problem statements.

Week 3

Prototype

Based on usability testing to update the low-fidelity wireframe.

Based on usability testing to update the low-fidelity wireframe.

Ideation

After having a clear idea of the pain points and goal we are trying to solve, we all decided to start brainstorming ideas and how the shopper’s flow should look.

Let’s put the barcode

The main challenge was simplifying the task of remembering and identifying produce names for shoppers. Observations showed that while people recognize produce visually and know its name in their native language, they don't necessarily recall the local U.S. names. This led to reframing the problem to focus on seamless addition of produce to the payment cart. Initially, we considered barcodes next to produce names for easy scanning. However, this approach faced the challenge of weighing produce sold in boxes and priced per pound.

Overcoming the challenge of measuring the weight of produce

Weighing machine at the end of each aisle

Shoppers have the option to weigh their produce at the end of each aisle, generating a receipt with a barcode that embeds the weight of the produce. Additionally, a digital barcode solution is considered to minimize paper waste.

Putting the weighing machine in the cart itself

We considered transforming the bottom of the small basket attached to the cart into a weighing machine, enabling users to see the weigh and feed it into the cart after scanning the item.

After testing in our studio, we found that Idea 1, involving double barcode scanning and taking produce to the aisle's end, was effort-intensive and inconvenient. Consequently, we discarded it in favor of Idea 2. This approach, integrating a weighing machine within the cart, seemed more practical and user-friendly, significantly reducing the need for shoppers to move around unnecessarily.

Wait! How will a user scan the barcode? 

While initially assuming shoppers would use their phones for scanning, we observed that frequent phone use for other tasks made repeatedly opening an app like Kroger cumbersome. This led to the decision to not use mobile phones as the primary interactive device.


We briefly considered smart watches but dismissed this idea to ensure inclusivity, recognizing that not all shoppers wear or prefer watches.


The alternative proposed was to attach a digital screen to the shopping cart. The challenge was facilitating scanning through this screen, leading us to explore various ideas to address this.

Detachable digital screen

Shoppers can detach the digital screen from the cart and scan the barcode to identify the item. 

Tethered and detachable barcode

We can tether the barcode with the aisle so that user can pull it and scan the barcode by putting it on top of the digital screen. We tethered it so that people don’t steal it or put it in the wrong place. 

Scanner attached to barcode

We also thought of having the detachable scanner attached to a digital screen through which the shopper can scan the barcode easily. 

We decided to go with the Idea-2 because of the following reasons: 


  • Detachable digital screens can be messy at times as it might require the user to detach and attach the screen again and again. Also, shoppers need to attach and detach effectively which has its own learning curve and can be frustrating at times. 

  • We discarded the idea-3 because we thought this would lead to addition of lot of items to the cart and this will also come with an additional capital cost. 

  • Idea-2 seemed the most reasonable as it can be easy for a shopper to pull the barcode and scan it by putting it on the digital screen. 

Jotting down our final flow 

After brainstorming the ideas of all the components of our flow and performing those experience we came up with the following flow: 


So, there would be a digital interactive device (similar iPad) attached to the shopping cart. The bottom of the basket inside the cart will be used as the weighing machine.  


If a shopper wants to add a produce into the cart, the user will pull the barcode and put it on top of the digital screen to scan and recognize the produce. Then the user will weigh the produce by putting it into the basket, which will weigh the weight of the produce and will be added to digital interactive device attached to the cart automatically. User can then click on ‘Add to cart’ to add the item in the shopping cart. This way the user will keep on adding the items similarly and in the end can simply checkout in the shopping cart itself. 

The flows that we created while jotting our solutions  

Designing our digital interactive screen 

To design the digital interactive screen, we decided to sketch the wireframes and flow on whiteboards. Then we started discussing each other's sketch and started brainstorming by scraping and taking some ideas of it. Taking Fig. 4 as reference we decided to design interactive screens. 

Showcasing the initial wireframes 

Final Wireframe 

Improving on the initial wireframes illustrated in Fig 5, we realized that there was an absence of feedback when a produce was added to the cart. To bridge this gap, we introduced a buffer, signaling shoppers that the item is currently being weighed. This visual cue enhances transparency and sets clear expectations, reducing potential confusion about the process. Building on this, we further refined the checkout process: the 'Add to Cart' button now activates only once the item's weight is confirmed. This ensures that shoppers have a clear sequence of steps to follow. By calculating the price per lb based on the actual weight of the produce added to the cart, shoppers are provided with a clear and transparent breakdown of the total cost. However, to emphasize purchase completion, the discard option was designed as a secondary action. This design subtly guides shoppers towards finalizing their purchases, emphasizing transaction completion, while retaining the flexibility for them to change their mind. 

Showcasing the final wireframes 

Usability Testing 

After having our sketch ready, we decided to conduct usability testing to test our solution. We decided to perform a paper prototyping by attaching the sketches to the cart and changing the pages depending on his/her/them interactions with it. 

Pre – Usability Testing 

Prior to usability testing, we focused on determining our testing objectives. Specifically, we aimed to evaluate if our solution seamlessly integrates into the user experience and assists in recalling product names. For this, we asked shoppers to use our cart with the attached digital device to purchase two of their preferred produce items.

During the usability testing 

We secured consent from each participant and briefed them on the procedure. They were informed that a small basket, positioned near the handle of the cart, would simulate a weighing machine. Participants were encouraged to verbalize their thoughts during the task. The task involved selecting two produce items for purchase.

Seema performing our task. The notebook was acting as a digital interactive device 
Location: Kroger, 528 S College Avenue, Bloomington, Indiana 

Post-usability testing 

At the end, we thanked our participant for taking out time and helping us with the usability testing. We also observed some things during our usability testing, and this was the right time to talk about it and understand what they were thinking while performing it. 

Analysis and insights 

Initial Confusion in Task Execution

Seema initially struggled to choose between scanning or weighing items, feeling that scanning QR codes disrupted her usual shopping routine. She expressed that while placing items in the cart or basket felt natural, scanning barcodes seemed intrusive. This highlights the need for new technologies to blend smoothly into established consumer practices.

Perceived Unnatural Action

Mahesh found the scanning process "unnatural" and was uncertain about remembering to scan barcodes in the aisle, indicating a disconnect between this new process and regular shopping habits.

Security Concerns

Mohini raised concerns about the potential for theft with the portable checkout system, indicating the need for robust security measures to ensure the system's integrity and user trust.

General Appreciation with Reservations

Ivan valued the system's convenience and its potential to reduce reliance on employee assistance during checkout, while Seema appreciated the streamlined shopping experience, especially as it could prevent the need to revisit aisles for information.

Improvements

So, after usability testing it was clear that scanning the barcode was a bit “unnatural” (Mahesh) and breaking the habitual flow of shoppers of adding items directly into their cart. To overcome this, we again brainstormed and came up with

Putting barcode on each produce

Implementing barcode stickers on produce allows for automatic scanning and weight recognition as items are added to the cart, streamlining the shopping process.

AI Image recognition

We proposed integrating a scanner on our digital device to instantly weigh and display items added to the cart. Utilizing the LLaVA model, known for superior object detection accuracy, it can distinguish similar products and identify items in complex, cluttered arrangements.

We liked Idea 2 (AI Image recognition) as compared to Idea 1 (Putting a barcode on each produce) as putting up barcode on each item can be tedious and can create problems for shop owners.  

Final Solution

Showcasing the final wireframes

Showcasing the initial wireframes. Sketch by Pei Sun.

In our final solution, we decided to go with the AI image processing camera which will be attached to the back of the cart. The blue part will act as a weighing machine.


When a shopper adds any produce in the cart, the digital interactive device/tablet will scan the produce and will recognize, which will show up on digital interactive device. Then it will automatically calculate the weight which is shown as buffer thing in the Fig 8 right top image. So, it will finally show the weight and price. Then the shopper can click on ‘Add to cart’ then it will show up in payment checking counter. Then the device will show ‘Item added successfully.’  

Conclusion

Impact 

It was important to understand how our solution with improvements can impact the lives of our shoppers

Sense of Assurance

Shoppers on seeing the name (shoppers can choose whether they want to see the name in their native language in the digital interactive device) in their native language gives them a feeling of assurance that they have bought the same produce that they were looking for. 

No More Remembering US Local Names

This was our primary pain point and now our shoppers don’t need to worry about remembering the US names. Now they don’t have to go back again to Aisle to see the name of the produce they purchased. 

No More Moving Here and There

Now shoppers can directly check out. Now they don’t need to find and talk to staff members. Also, they don’t need to take out the items again during the checkout to scan and package them.

Limitations

Some constraints that we noticed with our design, and attempted resolving, are

Anti-theft system

  • We could conceptualize a system where the digital display’s camera actively monitors shopping activities to prevent theft. This system prompts a warning when an unscanned item is added to a cart and alerts store personnel if the warning is unheeded. While promising, we acknowledge the inherent privacy challenges. We're exploring ways to maintain a balance between the anti-theft approach and safeguarding user privacy.

  • We could focus on stationing a security guard at the exit doors and have the shopping cart stand nearby. This would ensure that all shoppers return their shopping carts to their rightful location and allow the guard to check on any suspicious activity.  

Future work 

In the limited time frame of the project, we concentrated on addressing a key issue using straightforward features. Additionally, having identified other pain points during our observation, we aim to outline potential strategies for enhancing the shopping experience

Personalized Interface

Once the user logs in, the app will apply the language set in their Kroger account, ensuring everything they see is easy to understand in advance.

Easy Checkout

Once the user logs in, if they have linked their credit card to the Kroger app, they can seamlessly complete the shopping using only the in-store monitor/app. 

Smart Shopping List

If the user puts their shopping list on their phone with a defined format, they can use the in-store app to scan it to show on the monitor. It provides a side-by shopping list, which helps shoppers to aim for the items they need accurately and efficiently. 

Floor Information

If we merge detailed store maps into the app, shoppers can find items with clear instructions, removing the redundant searching time and creating a better experience.

If these solutions are implemented, the in-store shopping experience will be much improved to provide shoppers with a smoothen embodied shopping experience 

Contribution

  • Conducted observations and contextual inquiries at Kroger, documenting public behaviors and interactions.

  • Conducted in-depth secondary research to gather valuable insights. 

  • Collaborated with the team to develop an affinity diagram, to refine the problem space.

  • Designed initial wireframe and iteratively enhanced this to produce the final prototype.

  • Conducted usability testing with an international student to assess the prototype and gather feedback to inform further refinements and improvements.

  • Co-authored the digital documentation. 

References

  1. Andrade, M. S. (2006). International students in English-speaking universities. Journal of Research in International Education, 5(2), 131-154. 

  2. Brown, L., Edwards, J., & Hartwell, H. (2010). A taste of the unfamiliar. Appetite, 54(1), 202-207. 

  3. Klepek, M., & Bauero, R. (2020). In-store vs. online shopping. Journal of Consumer Behavior, 10(4), 234-245. 

  4. Klepek, M., & Bauero, R. (2020). Why do retail customers hesitate for shopping grocery online? Technological and Economic Development of Economy, 26(6), 1444-1462. 

  5. Krishna, A. (2012). An integrative review of sensory marketing: Engaging the senses to affect perception, judgment and behavior. Journal of Consumer Psychology, 22(3), 332-351. 

  6. Krishna, A. (2012). An integrative review of sensory marketing. Journal of Consumer Psychology, 22(3), 333-352. 

  7. Krishna, A., & Schwarz, N. (2014). Sensory marketing, embodiment, and grounded cognition: A review and introduction. Journal of Consumer Psychology, 24(2), 159-168. 

  8. LLaVA: Large Language and Vision Assistant. https://llava-vl.github.io/ 

  9. Merleau-Ponty, M. (1962). Phenomenology of Perception. Routledge & Kegan Paul. 

  10. Midjourney 2023-10-09 02:58:05 - apple, line art, white background [Midjourney AI Generated Image]. 

  11. Peck, J., & Childers, T. L. (2003). Individual differences in haptic information processing: The “need for touch” scale. Journal of Consumer Research, 30(3), 430-442. 

  12. Poyrazli, S., & Lopez, M. D. (2007). International students and discrimination. The Journal of Psychology, 141(3), 263-280. 

  13. Swagler, M. A., & Ellis, M. V. (2003). Taiwanese student adjustment. Journal of Counseling Psychology, 50(4), 420-437. 

  14. Toombs, A., Kaziunas, E. (2023). Project 2: Embodied shopping. Human-Computer Interaction Program, Indiana University Bloomington.

  15. Turner, Y., & Thompson, E. (2014). International student migrations. In Y. Turner (Ed.), Internationalizing the University (pp. 167-188).

  16. Palgrave Macmillan UK. Yakhlef, A. (2015). The corporeality of consumption: embodied cognizance and sensation. Marketing Theory, 15(2), 135. https://journals.sagepub.com/doi/10.1177/1470593115569016 

Embodied Shopping

MS HCI/d | I541 Project

2023

The project focuses on overcoming language barriers and unfamiliarity with local products. The solution proposed was a digital device attached to shopping carts, featuring AI image recognition for product identification and an easier checkout process.

Team

Sriram Hemanth Kumar

Pei Sun

Monisha Vimal Manshukani

Ravi Jangir

Duration

3 Weeks

Problem

Our goal is to understand the needs of the contemporary grocery shopper, channeling this knowledge to inform the design of a refined experience for grocery store patrons that maximizes positive and empowering embodied patterns of interaction (Toombs, A., Kaziunas, E., 2023)

Design Journey

We started by studying and researching our problem closely, then made a prototype of our product. After testing and improving the prototype based on feedback and previous findings, we created our final product.

Week 1

Identify Project Goal

Narrowing down target user base and finding the potential shoppers for doing user interviews.

Narrowing down target user base and finding the potential shoppers for doing user interviews.

Week 1 - 2

Observation

1 on 1 user interviews to understand user’s mindset of using the wearable devices and finding their initial ideas about faking data.

1 on 1 user interviews to understand user’s mindset of using the wearable devices and finding their initial ideas about faking data.

Week 2

Secondary research

Affinity Diagramming

Affinity Diagramming

Week 2

Define Problem Statement

Based on the primary research and secondary research to define the problem statements.

Based on the primary research and secondary research to define the problem statements.

Week 2 - 3

Sketch for Solutions

Quickly sketching our ideas based on the problem statements.

Quickly sketching our ideas based on the problem statements.

Week 3

Prototype

Based on usability testing to update the low-fidelity wireframe.

Based on usability testing to update the low-fidelity wireframe.

Ideation

After having a clear idea of the pain points and goal we are trying to solve, we all decided to start brainstorming ideas and how the shopper’s flow should look.

Let’s put the barcode

The main challenge was simplifying the task of remembering and identifying produce names for shoppers. Observations showed that while people recognize produce visually and know its name in their native language, they don't necessarily recall the local U.S. names. This led to reframing the problem to focus on seamless addition of produce to the payment cart. Initially, we considered barcodes next to produce names for easy scanning. However, this approach faced the challenge of weighing produce sold in boxes and priced per pound.

Overcoming the challenge of measuring the weight of produce

Weighing machine at the end of each aisle

Shoppers have the option to weigh their produce at the end of each aisle, generating a receipt with a barcode that embeds the weight of the produce. Additionally, a digital barcode solution is considered to minimize paper waste.

Putting the weighing machine in the cart itself

We considered transforming the bottom of the small basket attached to the cart into a weighing machine, enabling users to see the weigh and feed it into the cart after scanning the item.

After testing in our studio, we found that Idea 1, involving double barcode scanning and taking produce to the aisle's end, was effort-intensive and inconvenient. Consequently, we discarded it in favor of Idea 2. This approach, integrating a weighing machine within the cart, seemed more practical and user-friendly, significantly reducing the need for shoppers to move around unnecessarily.

Wait! How will a user scan the barcode? 

While initially assuming shoppers would use their phones for scanning, we observed that frequent phone use for other tasks made repeatedly opening an app like Kroger cumbersome. This led to the decision to not use mobile phones as the primary interactive device.


We briefly considered smart watches but dismissed this idea to ensure inclusivity, recognizing that not all shoppers wear or prefer watches.


The alternative proposed was to attach a digital screen to the shopping cart. The challenge was facilitating scanning through this screen, leading us to explore various ideas to address this.

Detachable digital screen

Shoppers can detach the digital screen from the cart and scan the barcode to identify the item. 

Tethered and detachable barcode

We can tether the barcode with the aisle so that user can pull it and scan the barcode by putting it on top of the digital screen. We tethered it so that people don’t steal it or put it in the wrong place. 

Scanner attached to barcode

We also thought of having the detachable scanner attached to a digital screen through which the shopper can scan the barcode easily. 

We decided to go with the Idea-2 because of the following reasons: 


  • Detachable digital screens can be messy at times as it might require the user to detach and attach the screen again and again. Also, shoppers need to attach and detach effectively which has its own learning curve and can be frustrating at times. 

  • We discarded the idea-3 because we thought this would lead to addition of lot of items to the cart and this will also come with an additional capital cost. 

  • Idea-2 seemed the most reasonable as it can be easy for a shopper to pull the barcode and scan it by putting it on the digital screen. 

Jotting down our final flow 

After brainstorming the ideas of all the components of our flow and performing those experience we came up with the following flow: 


So, there would be a digital interactive device (similar iPad) attached to the shopping cart. The bottom of the basket inside the cart will be used as the weighing machine.  


If a shopper wants to add a produce into the cart, the user will pull the barcode and put it on top of the digital screen to scan and recognize the produce. Then the user will weigh the produce by putting it into the basket, which will weigh the weight of the produce and will be added to digital interactive device attached to the cart automatically. User can then click on ‘Add to cart’ to add the item in the shopping cart. This way the user will keep on adding the items similarly and in the end can simply checkout in the shopping cart itself. 

The flows that we created while jotting our solutions  

Designing our digital interactive screen 

To design the digital interactive screen, we decided to sketch the wireframes and flow on whiteboards. Then we started discussing each other's sketch and started brainstorming by scraping and taking some ideas of it. Taking Fig. 4 as reference we decided to design interactive screens. 

Showcasing the initial wireframes 

Final Wireframe 

Improving on the initial wireframes illustrated in Fig 5, we realized that there was an absence of feedback when a produce was added to the cart. To bridge this gap, we introduced a buffer, signaling shoppers that the item is currently being weighed. This visual cue enhances transparency and sets clear expectations, reducing potential confusion about the process. Building on this, we further refined the checkout process: the 'Add to Cart' button now activates only once the item's weight is confirmed. This ensures that shoppers have a clear sequence of steps to follow. By calculating the price per lb based on the actual weight of the produce added to the cart, shoppers are provided with a clear and transparent breakdown of the total cost. However, to emphasize purchase completion, the discard option was designed as a secondary action. This design subtly guides shoppers towards finalizing their purchases, emphasizing transaction completion, while retaining the flexibility for them to change their mind. 

Showcasing the final wireframes 

Usability Testing 

After having our sketch ready, we decided to conduct usability testing to test our solution. We decided to perform a paper prototyping by attaching the sketches to the cart and changing the pages depending on his/her/them interactions with it. 

Pre – Usability Testing 

Prior to usability testing, we focused on determining our testing objectives. Specifically, we aimed to evaluate if our solution seamlessly integrates into the user experience and assists in recalling product names. For this, we asked shoppers to use our cart with the attached digital device to purchase two of their preferred produce items.

During the usability testing 

We secured consent from each participant and briefed them on the procedure. They were informed that a small basket, positioned near the handle of the cart, would simulate a weighing machine. Participants were encouraged to verbalize their thoughts during the task. The task involved selecting two produce items for purchase.

Seema performing our task. The notebook was acting as a digital interactive device 
Location: Kroger, 528 S College Avenue, Bloomington, Indiana 

Post-usability testing 

At the end, we thanked our participant for taking out time and helping us with the usability testing. We also observed some things during our usability testing, and this was the right time to talk about it and understand what they were thinking while performing it. 

Analysis and insights 

Initial Confusion in Task Execution

Seema initially struggled to choose between scanning or weighing items, feeling that scanning QR codes disrupted her usual shopping routine. She expressed that while placing items in the cart or basket felt natural, scanning barcodes seemed intrusive. This highlights the need for new technologies to blend smoothly into established consumer practices.

Perceived Unnatural Action

Mahesh found the scanning process "unnatural" and was uncertain about remembering to scan barcodes in the aisle, indicating a disconnect between this new process and regular shopping habits.

Security Concerns

Mohini raised concerns about the potential for theft with the portable checkout system, indicating the need for robust security measures to ensure the system's integrity and user trust.

General Appreciation with Reservations

Ivan valued the system's convenience and its potential to reduce reliance on employee assistance during checkout, while Seema appreciated the streamlined shopping experience, especially as it could prevent the need to revisit aisles for information.

Improvements

So, after usability testing it was clear that scanning the barcode was a bit “unnatural” (Mahesh) and breaking the habitual flow of shoppers of adding items directly into their cart. To overcome this, we again brainstormed and came up with

Putting barcode on each produce

Implementing barcode stickers on produce allows for automatic scanning and weight recognition as items are added to the cart, streamlining the shopping process.

AI Image recognition

We proposed integrating a scanner on our digital device to instantly weigh and display items added to the cart. Utilizing the LLaVA model, known for superior object detection accuracy, it can distinguish similar products and identify items in complex, cluttered arrangements.

We liked Idea 2 (AI Image recognition) as compared to Idea 1 (Putting a barcode on each produce) as putting up barcode on each item can be tedious and can create problems for shop owners.  

Final Solution

Showcasing the final wireframes

Showcasing the initial wireframes. Sketch by Pei Sun.

In our final solution, we decided to go with the AI image processing camera which will be attached to the back of the cart. The blue part will act as a weighing machine.


When a shopper adds any produce in the cart, the digital interactive device/tablet will scan the produce and will recognize, which will show up on digital interactive device. Then it will automatically calculate the weight which is shown as buffer thing in the Fig 8 right top image. So, it will finally show the weight and price. Then the shopper can click on ‘Add to cart’ then it will show up in payment checking counter. Then the device will show ‘Item added successfully.’  

Conclusion

Impact 

It was important to understand how our solution with improvements can impact the lives of our shoppers

Sense of Assurance

Shoppers on seeing the name (shoppers can choose whether they want to see the name in their native language in the digital interactive device) in their native language gives them a feeling of assurance that they have bought the same produce that they were looking for. 

No More Remembering US Local Names

This was our primary pain point and now our shoppers don’t need to worry about remembering the US names. Now they don’t have to go back again to Aisle to see the name of the produce they purchased. 

No More Moving Here and There

Now shoppers can directly check out. Now they don’t need to find and talk to staff members. Also, they don’t need to take out the items again during the checkout to scan and package them.

Limitations

Some constraints that we noticed with our design, and attempted resolving, are

Anti-theft system

  • We could conceptualize a system where the digital display’s camera actively monitors shopping activities to prevent theft. This system prompts a warning when an unscanned item is added to a cart and alerts store personnel if the warning is unheeded. While promising, we acknowledge the inherent privacy challenges. We're exploring ways to maintain a balance between the anti-theft approach and safeguarding user privacy.

  • We could focus on stationing a security guard at the exit doors and have the shopping cart stand nearby. This would ensure that all shoppers return their shopping carts to their rightful location and allow the guard to check on any suspicious activity.  

Future work 

In the limited time frame of the project, we concentrated on addressing a key issue using straightforward features. Additionally, having identified other pain points during our observation, we aim to outline potential strategies for enhancing the shopping experience

Personalized Interface

Once the user logs in, the app will apply the language set in their Kroger account, ensuring everything they see is easy to understand in advance.

Easy Checkout

Once the user logs in, if they have linked their credit card to the Kroger app, they can seamlessly complete the shopping using only the in-store monitor/app. 

Smart Shopping List

If the user puts their shopping list on their phone with a defined format, they can use the in-store app to scan it to show on the monitor. It provides a side-by shopping list, which helps shoppers to aim for the items they need accurately and efficiently. 

Floor Information

If we merge detailed store maps into the app, shoppers can find items with clear instructions, removing the redundant searching time and creating a better experience.

If these solutions are implemented, the in-store shopping experience will be much improved to provide shoppers with a smoothen embodied shopping experience 

Contribution

  • Conducted observations and contextual inquiries at Kroger, documenting public behaviors and interactions.

  • Conducted in-depth secondary research to gather valuable insights. 

  • Collaborated with the team to develop an affinity diagram, to refine the problem space.

  • Designed initial wireframe and iteratively enhanced this to produce the final prototype.

  • Conducted usability testing with an international student to assess the prototype and gather feedback to inform further refinements and improvements.

  • Co-authored the digital documentation. 

References

  1. Andrade, M. S. (2006). International students in English-speaking universities. Journal of Research in International Education, 5(2), 131-154. 

  2. Brown, L., Edwards, J., & Hartwell, H. (2010). A taste of the unfamiliar. Appetite, 54(1), 202-207. 

  3. Klepek, M., & Bauero, R. (2020). In-store vs. online shopping. Journal of Consumer Behavior, 10(4), 234-245. 

  4. Klepek, M., & Bauero, R. (2020). Why do retail customers hesitate for shopping grocery online? Technological and Economic Development of Economy, 26(6), 1444-1462. 

  5. Krishna, A. (2012). An integrative review of sensory marketing: Engaging the senses to affect perception, judgment and behavior. Journal of Consumer Psychology, 22(3), 332-351. 

  6. Krishna, A. (2012). An integrative review of sensory marketing. Journal of Consumer Psychology, 22(3), 333-352. 

  7. Krishna, A., & Schwarz, N. (2014). Sensory marketing, embodiment, and grounded cognition: A review and introduction. Journal of Consumer Psychology, 24(2), 159-168. 

  8. LLaVA: Large Language and Vision Assistant. https://llava-vl.github.io/ 

  9. Merleau-Ponty, M. (1962). Phenomenology of Perception. Routledge & Kegan Paul. 

  10. Midjourney 2023-10-09 02:58:05 - apple, line art, white background [Midjourney AI Generated Image]. 

  11. Peck, J., & Childers, T. L. (2003). Individual differences in haptic information processing: The “need for touch” scale. Journal of Consumer Research, 30(3), 430-442. 

  12. Poyrazli, S., & Lopez, M. D. (2007). International students and discrimination. The Journal of Psychology, 141(3), 263-280. 

  13. Swagler, M. A., & Ellis, M. V. (2003). Taiwanese student adjustment. Journal of Counseling Psychology, 50(4), 420-437. 

  14. Toombs, A., Kaziunas, E. (2023). Project 2: Embodied shopping. Human-Computer Interaction Program, Indiana University Bloomington.

  15. Turner, Y., & Thompson, E. (2014). International student migrations. In Y. Turner (Ed.), Internationalizing the University (pp. 167-188).

  16. Palgrave Macmillan UK. Yakhlef, A. (2015). The corporeality of consumption: embodied cognizance and sensation. Marketing Theory, 15(2), 135. https://journals.sagepub.com/doi/10.1177/1470593115569016 

sriramph