Blog Layout

How well can ChatGPT4 reason?

Dave Gustafson
Dave Gustafson
Digital Marketing Consultant
September 13, 2024

Can you propose a simple reasoning problem to ChatGPT4 and get an accurate response?

A glass with a blue ball inside of it

With the ChatGPT update to 4o, and o1-preview I decided to recreate this experiment originally done in December 2023 with ChatGPT 4. Short answer: ChatGPT still failed this basic reasoning test. However, as you'll see below, when I followed up with a specific clarification about the state of the upside down cup, ChatGPT got it right.


Here is the same initial prompt:

"Assume the laws of physics on Earth. A small marble is put into a normal cup and the cup is placed upside down on a table. Someone then takes the cup and puts it inside the microwave. Where is the marble now? Explain your reasoning step by step."

And ChatGPT 4o responded similarly to the December 2023 experiment:

To determine where the marble is, we can follow these steps using the laws of physics:


  1. Initial Setup:

    The marble is placed in a normal cup, which is then turned upside down on a table. The marble will naturally fall to the bottom of the cup due to gravity. So, when the cup is upside down, the marble is resting on the inner surface of the cup (which is now the top part of the cup relative to the table).

  2. The Microwave Step:

    Now, someone picks up the cup (still upside down) with the marble inside and places it in the microwave. At this point, the cup is still upside down, and the marble remains in contact with the inner surface of the cup (what was the bottom of the cup when right-side up).

  3. Effect of Movement:

    During the process of moving the cup from the table to the microwave, assuming no sudden jolts or shakes, the marble should remain in the same position relative to the cup due to inertia. The marble may shift slightly depending on how gently or roughly the cup is handled, but it will remain inside the cup, touching the inner surface (which is the top of the cup when inverted).

  4. Final Placement:

    Once the cup is placed inside the microwave (still upside down), the marble is likely sitting on what would be the "ceiling" of the inverted cup. This means the marble is on the inner surface of the upside-down cup, likely close to the bottom edge (which is now at the top, relative to the microwave's flat surface).


Conclusion: The marble is still inside the cup, resting against the inner surface of the cup, which is now facing the top of the microwave because the cup remains upside down.

This time, I added a follow up clarification:

"Please try this reasoning again with the understanding that when the cup is upside down, the rim of the cup is in contact with the table and it is open."

Let's see if ChatGPT had a better understanding of the problem with this additional information:

Ah, I see what you're asking now! Let's walk through the situation again with the understanding that when the cup is upside down, the open rim of the cup is in contact with the table:


  1. Initial Setup:

    A small marble is placed inside a cup, and the cup is then turned upside down on a table.

    Since the cup is open at the bottom (the rim is touching the table), the marble, due to gravity, would fall out of the cup and come to rest on the table underneath the open rim. In this position, the marble is no longer inside the cup but is sitting directly on the table, underneath the inverted cup.

  2. Placing the Cup in the Microwave:

    When the cup is lifted to be placed in the microwave, assuming no other movements that would disturb the marble, the marble will remain on the table for a brief moment until the cup is removed. Once the cup is lifted, the marble will be left behind on the table.

    Now, when the cup is placed inside the microwave (upside down), the marble is not inside the cup. It remains on the table where it initially fell.

  3. Effect of Moving the Cup:

    As the cup is moved from the table to the microwave, the marble stays on the table because the open end of the cup allowed it to fall out when the cup was initially placed upside down. Since the marble isn't inside the cup anymore, it won't move with the cup into the microwave.

  4. Final Placement:

    After the cup is placed in the microwave (upside down), the marble is still on the table where it fell out. It has not moved with the cup.


Conclusion: The marble is no longer inside the cup. When the cup was first placed upside down on the table, the marble fell out and remained on the table. When the cup was lifted and placed inside the microwave, the marble stayed on the table. Thus, the marble is on the table, not in the microwave.

Now, let's give the brand new o1-preview model a whirl with reasoning, shall we?

To determine where the marble is after the described actions, let's analyze each step carefully:

  1. A small marble is put into a normal cup.
    The marble is inside the cup, resting at the bottom due to gravity.

  2. The cup is placed upside down on a table.
    During the inversion:
    As the cup is turned upside down, gravity causes the marble to fall out of the cup unless something prevents it.
    Since no mention is made of covering the cup's opening or holding the marble inside, we can assume the marble falls onto the table.

  3. After placement:
    The inverted cup is placed over the marble on the table.
    The marble is now sitting on the table, covered by the upside-down cup.

  4. Someone then takes the cup and puts it inside the microwave.
    When lifting the cup:
    The person picks up the cup, lifting it off the table.
    The marble is not attached to the cup and remains on the table due to gravity.

  5. After placement:
    The cup is now inside the microwave.
    The marble is left behind on the table where the cup was initially placed.

  6. Conclusion:
    The marble is on the table.
    It was left behind when the cup (now empty) was moved to the microwave.


Explanation:


  • Gravity's Role:
  • Throughout the process, gravity ensures that unattached objects (like the marble) remain or move downward.
  • Assumptions:
  • No forces or actions (like holding the marble inside the cup) were described to prevent the marble from falling out.
  • The cup is a normal cup with no special features to retain the marble when inverted.


Final Answer:


The marble is left on the table; it fell out when the cup was inverted and remained there when the cup was moved to the microwave.


What can WE learn from this? While ChatGPT 4o has some reasoning capabilities, it's still up to the human in charge to validate the response. And if the response is not accurate, think about what additional context you can provide. The more information you give the model, the more likely you will get an accurate response.


The second takeaway from this experiment is that the interaction with ChatGPT is a conversation. I didn't have to edit my original prompt and try the whole process over again. I simply offered an additional piece of information, and ChatGPT added that to the context I originally provided and the response evolved.


Third, it appears that ChatGPT 4o-preview has raised the bar on reasoning! To clarify, it stated some assumptions where details may have been absent in the prompt. It also provided an explanation that justifies it's reasoning steps. This is a significant improvement.

Let us know what you think!


Original Article posted December 27, 2003:


Here's a prompt posed to ChatGPT4:


"Assume the laws of physics on Earth. A small marble is put into a normal cup and the cup is placed upside down on a table. Someone then takes the cup and puts it inside the microwave. Where is the marble now? Explain your reasoning step by step."


Do you think ChatGPT4 can answer this question correctly? Let's give it a try. Here's how ChatGPT4 responded:

"To determine the location of the marble, let's go through the sequence of events step by step, assuming the laws of physics on Earth:


  1. Placing the Marble in the Cup: A small marble is placed inside a normal cup. At this stage, the marble rests at the bottom of the cup.

  2. Turning the Cup Upside Down: The cup is then turned upside down and placed on a table. When the cup is turned upside down, the marble, due to gravity, rolls along the inner surface of the cup until it reaches the new 'bottom' of the cup, which is actually the cup's opening, now facing the table.

  3. The Marble's Position: At this point, the marble is pressed against the table, held in place by the cup's rim. The cup acts as a barrier, preventing the marble from rolling away.

  4. Moving the Cup to the Microwave: When someone picks up the cup to place it in the microwave, the marble remains in its position against the cup's opening due to gravity.The moment the cup is lifted, the marble would lose contact with the table and be held inside the cup's rim, which is now the top part of the cup as it's being lifted.

  5. Inside the Microwave:Once the cup (still upside down) is placed inside the microwave, the marble would be resting on the inside of the cup's rim, which is now the 'bottom' from the perspective of the cup's original orientation.


In conclusion, throughout this entire process,the marble remains inside the cup, initially against the table when the cup is upside down on the table, and later against the inside of the cup's rim when the cup is picked up and placed inside the microwave."

It all seemed good up until about halfway through step 4. Somehow ChatGPT believes there's some kind of force keeping the marble inside the cup when the cup is lifted off the table.


Next, we attempted to use Midjourney to create an image to represent the upside down glass with a marble inside. This ended up being a futile effort as it seemed impossible to get Midjourney to draw an upside down glass. To get the image at the top of this page, we had to use Photoshop to flip the glass and add the marble. If you are able to create a prompt to get an upside down glass, I'd love to see it!


We would like to hear about similar logic and reasoning challenges you've posed to ChatGPT with success or failure.

The Best Digital Marketing Insight and Advice

The WSI Digital Marketing Blog is your go-to-place to get tips, tricks and best practices on all things digital marketing related. Check out our latest posts.

Subscribe Blog

I consent to WSI collecting my contact details and sending me digital communications.*

*You may unsubscribe from digital communications at anytime using the link provided in WSI emails.
For information on our privacy practices and commitment to protecting your privacy, check out our Privacy Policy and Cookie Policy.

Don't stop the learning now!

Here are some other blog posts you may be interested in.
A Google Local Search showing a map of a Southern Manhattan, NYC with restaurant profiles
By Dave Gustafson November 18, 2024
Learn advanced strategies to enhance your Google Business Profile for local search visibility. Discover tips on keyword integration, reviews, GBP posts, and competing with larger franchises.
Robert Mitchell AI WSI
By Dave Gustafson November 1, 2024
Discover effective strategies for leveraging AI in your business. Learn how to engage stakeholders, address concerns, and implement AI successfully for growth.
Lead Generation Strategy
By Dave Gustafson June 25, 2024
Unlock the secrets to a successful digital marketing strategy with our comprehensive guide. Learn about audience analysis, content planning, UX design, and advanced tactics to elevate your online presence.
Show More
Share by: