8
$\begingroup$

We have had posts previously discussing how ChatGPT might affect answers on Chemistry SE. Here, I am opening a discussion about how it might affect traffic and types of questions posed on Q&A sites.

It seems ChatGPT is perfect for homework-type questions where students hope for a quick answer to a standard problem. You don't have to wait, you don't leave a track (if you are cheating), it doesn't cost anything, you don't have to admit you are asking a "stupid" question. So I wonder whether the number of blatant homework questions has decreased in the past months, as students get better at using ChatGPT.

I am also wondering whether the business model of paid services is getting undermined.

The type of problem for which ChatGPT gives the least reliable answers, in my experience, is quantitative problems. For example, it claimed that $\log 0.01 = -1$. So maybe students would (or should?) still use Q & A sites to get answers to quantitative questions.

Are there hard data about this, or perhaps already some research literature? Does anyone have anecdotal evidence in changes how people ask homework questions?

Update: here is some data about stackoverflow visitor stats: https://www.similarweb.com/blog/insights/ai-news/stack-overflow-chatgpt/

Update 2: Here is some data about the chemistry site (ChatGPT was launched on November 30, 2022, and pandemic lock-downs in the US started in April 2020). The SEDE query is adapted from one for StackOverflow, which had a strong decrease in questions after ChatGPT appeared: enter image description here

$\endgroup$
12
  • 2
    $\begingroup$ I haven't noticed a decrease in participation (on the contrary, there seems to have been a surge of late, but that might be self-deception). I don't think chatgpt is quite ripe enough to compete with expert advice. It's good for churning out correlated words but not necessarily coherent and accurate answers. $\endgroup$
    – Buck Thorn Mod
    Apr 17, 2023 at 19:50
  • $\begingroup$ But there is blood in the water, and we are feeding the sharks with our contributions. Quite the catch-22. $\endgroup$
    – Buck Thorn Mod
    Apr 17, 2023 at 19:51
  • $\begingroup$ Have you run data.stackexchange.com/chemistry/query/649165/… recently? Some of the changes are seasonal, though, and might not reflect migration from or to other resources. $\endgroup$
    – Buck Thorn Mod
    Apr 17, 2023 at 19:53
  • $\begingroup$ Unrelated but on the topic of changes in participation. Is there a consensus explanation for the dropoff in questions during 2021? $\endgroup$
    – Buck Thorn Mod
    Apr 17, 2023 at 19:56
  • 3
    $\begingroup$ Well, there was quite a bit of questions from people disillusioned with this novelty. I don't think it's really good for any kind of questions. $\endgroup$
    – Mithoron
    Apr 18, 2023 at 16:41
  • 1
    $\begingroup$ @BuckThorn Explanation for the dropoff in questions? Afaict, in general, that's how internet site dynamics works. $\endgroup$
    – Mithoron
    Apr 18, 2023 at 17:53
  • 2
    $\begingroup$ @Mithoron That's not a particularly informative comment, akin to "it is what it is". $\endgroup$
    – Buck Thorn Mod
    Apr 19, 2023 at 6:59
  • 1
    $\begingroup$ One can only hope that those seeking answers to homework questions go elsewhere… $\endgroup$
    – Jon Custer
    Apr 21, 2023 at 1:47
  • 1
    $\begingroup$ @JonCuster Yes, like office hours, tutoring, study group - any scenario where you can talk to someone in person to help you learn. It's fine to use SE Chemistry if you have questions about subtleties, but to learn the basics, it is not ideal. $\endgroup$
    – Karsten Mod
    Apr 21, 2023 at 10:56
  • $\begingroup$ From the link you posted: "ChatGPT users miss out on the debate and just get an answer, which can seem quicker and more efficient. " But that is contrary to SO philosophy, which is typically light on debate ("this is not a forum....!"). $\endgroup$
    – Buck Thorn Mod
    Jun 6, 2023 at 11:10
  • $\begingroup$ This is also the kicker: "In addition, it’s likely that some of OpenAI’s accumulated wisdom on coding techniques comes from digesting content on Stack Overflow and similar sites." All along we've been feeding the AI engine. The emergence of something like ChatGPT or CoPilot was predictable but it being here now has taken me at least by surprise. $\endgroup$
    – Buck Thorn Mod
    Jun 6, 2023 at 11:13
  • $\begingroup$ It's nice to see that ChatGPT et al have yet to put a serious dent on participation here. $\endgroup$
    – Buck Thorn Mod
    Jun 20, 2023 at 5:18

1 Answer 1

4
$\begingroup$

TL,DR: The answers of the automates still require a critical mind.

"where students hope for a quick answer to a standard problem." I assume the AI algorithms are aware of the database wikipedia. Though it isn't a substitute to primary literature, there still are examples where I observe large discrepancies with the anticipated answer. In the example of the screen photo, for example, hallucination of https://deepai.org/chat about nationality by place of birth, and highly successful long life in an other country

enter image description here

(screen photo, just asked now, April 19th, 2023).

I perceive "chemist and crystallographer" a bit shallow in the description; maybe chemical crystallography still is not known well enough. However, no word, just utter silence about the joint work by Bürgi, Dunitz, Lehn, and Wipf about the trajectory angle one equally encounters early in organic chemistry?

$\endgroup$
2
  • $\begingroup$ By way of comparison: i.stack.imgur.com/fQN0B.png $\endgroup$
    – Loong
    Apr 19, 2023 at 19:51
  • $\begingroup$ @Loong Good point. New (non-chemistry) research topic: comparing the quality of AI robots' answers. With enough text generated by each of them when addressing the same set of questions, perhaps one can discern the answers with fragments as "typical ChatGPT dialect" or "characteristic for deep.ai dialect" similar as e.g. English by different speakers differs in vocabulary and pronunciation. (And Jack Dunitz was from Glasgow ... ref). $\endgroup$
    – Buttonwood
    Apr 20, 2023 at 20:13

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .