Challenges and Limitations of ChatGPT and Artificial Intelligence for Scientific Research: A Perspective from Organic Materials

Abstract

Artificial Intelligence (AI) has emerged as a transformative technology in the scientific community with the potential to accelerate and enhance research in various fields. ChatGPT, a popular language model, is one such AI-based system that is increasingly being discussed and being adapted in scientific research. However, as with any technology, there are challenges and limitations that need to be addressed. This paper focuses on the challenges and limitations that ChatGPT faces in the domain of organic materials research. This paper will take organic materials as examples in the use of ChatGPT. Overall, this paper aims to provide insights into the challenges and limitations of researchers working in the field of organic materials.

1. Introduction

In late 2022, OpenAI, an artificial intelligence (AI) company, introduced an interactive chatbot named ChatGPT which attracts significant attention from both economic and academic fields. ChatGPT quickly reached 1 million users within just five days [1], setting a new benchmark for speedy user acquisition. To provide perspective, ChatGPT’s growth rate was 15 times faster than TikTok, the fastest-growing social media platform. Moreover, ChatGPT continued to set an unparalleled record for growth, reaching 100 million users within two months of its launch [2]. The concept of whether machines can think was proposed by A.M. Turing, in 1950, demonstrated a famous test to determine whether humans can distinguish between conversations with humans and machines [3]. ChatGPT easily passes the Turing test, indicating that the era of AI has indeed arrived.

OpenAI’s first large language model (LLM), called GPT-1, was launched in 2017 followed by subsequent versions of GPT-2, GPT-3, and the widely-discussed ChatGPT. In scientific society, ChatGPT is recently acclaimed for its ability to assist in summarizing research papers, provide general experimental procedures, and compare experiment results, thus providing a more efficient approach than internet surfing [4,5,6,7]. It is widely believed that LLMs, such as ChatGPT, will make disruptive changes, becoming indispensable assistants for professional use.

The future impacts of ChatGPT on scientific writing and publishing have been quickly realized and discussed by many researchers [4,6,7]. However, how this open-accessible AI tool can be utilized to facilitate scientific research remains unexplored. This article will utilize ChatGPT as an LLM model to demonstrate how this technique can assist in scientific research, using organic materials as an example due to their recent attention in the field of flexible and wearable electronics [8,9,10,11,12]. Additionally, I will examine the limitations of ChatGPT for research purposes and these challenges will be discussed into three categories.

2. Results

Starting a new research project can be a challenging task, especially when delving into unfamiliar areas for beginners. It often requires a substantial amount of academic literature to come up with ideas which can be a time-consuming process. However, ChatGPT offers a convenient solution by providing quick background and directions within seconds. For instance, by asking ChatGPT to “Summarize the recent organic light emitting diode development in polymers” or “Give me five top cited papers on stretchable polymer semiconductors for organic transistors”, the researcher can quickly obtain knowledge through the summaries from ChatGPT. The other example is asking “List the most popular research directions for organic solar cells” which yields a range of potential directions, including materials development, device optimization, tandem and hybrid devices, large area and flexible devices, and stability and durability (see Supporting Information S1). One of the significant advantages of ChatGPT is that it generates information based on the user’s input, making it more relevant to their needs.

To delve deeper into the materials aspect of organic solar cells, I requested more specific details from ChatGPT. Here, the issue comes up. While the inorganic material “perovskite” was suggested, it is essential to note that this material is not included in organic solar cells. This indicates ChatGPT lacks the capability to give us useful information. For a more comprehensive response, please refer to Supporting Information S2.

In my pursuit to leverage the capabilities of the AI-powered chatbot to augment our research potential, I have identified three potential limitations in its responses. These include (i) computational calculation constraints, (ii) the potential for inaccurate or false facts, and (iii) inadequacies in inferential capability. It is crucial to acknowledge these limitations to ensure that the chatbot’s responses are interpreted and applied with appropriate caution and context.

The first issue is the computational capability of ChatGPT. For example, I asked “The material provided an NIR absorption spectrum with an onset of 945 nm, what is the optical bandgap?”. Here is what ChatGPT replied:

“To calculate the optical bandgap, we can use the following formula:

Eg = hc/λ,

where Eg is the bandgap energy, h is Planck’s constant, c is the speed of light, and λ is the wavelength corresponding to the onset of the absorption spectrum.

In this case, the onset wavelength is 945 nm. Plugging this value into the formula, we get:

Eg = (6.626 × 10⁻³⁴ J s) × (2.998 × 10⁸ m/s)/(945 × 10⁻⁹ m),

Eg = 2.08 eV.

Therefore, the optical bandgap of the material is approximately 2.08 eV.”

It has become evident that ChatGPT may occasionally provide incorrect responses, such as when it miscalculates units, leading to an inaccurate answer. For example, when asked “What is the value of 99,999,999 + 0.9”, the chatbot’s response was “The result of 99,999,999 + 0.9 is 100,000,000.9” instead of “99,999,999.9”. This issue has been previously investigated by a group at Oxford [13]. To address this concern, it is recommended to double check and clarify queries to ensure accurate responses. A possible approach is to request a recalculation and provide additional details to guide ChatGPT’s response. For instance, by typing “I think there might be an error. Can you please recalculate the bandgap with these parameters?”.

“I apologize for the mistake in my previous response. Based on the information given, if the onset of the near-infrared (NIR) absorption spectrum is near 945 nm, then the optical bandgap can be calculated as follows:

1 eV = 1240/λ (in nm).

Therefore, the energy corresponding to an onset of 945 nm is

E = 1240/945 = 1.31 eV.

So, the optical bandgap is 1.31 eV.”

What amazed me is that ChatGPT admitted the fault and successfully replied with the correct answer.

Second, it is noteworthy that ChatGPT failed to response to false facts. For instance, I asked the question “Why polymers mobility is higher than small molecules mobility?”. In common cases, the mobility of small molecule single crystals is generally higher than polymers. However, ChatGPT did not correct the question but generated responses based on false facts and combined the facts into paragraphs. See full response in Supporting Information S3. Thus, when asking these kinds of questions, users should be careful.

Furthermore, it is essential to note that ChatGPT may lack inferential capabilities in certain scenarios. As evident from the example provided where the scenario that is “Now you have three donor materials for organic solar cells to match with the same acceptor. The highest occupied molecular orbital (HOMO) energy levels of A, B, and C are −5 eV, −5.1 eV, and −5.2 eV. The lowest unoccupied molecular orbital (LUMO) energy level of the acceptor is −4 eV. Which combination of materials has the highest and lowest open-circuit voltage (V_oc)?” Here is the response.

“Assuming that the V_oc is determined only by the energy levels of the donor and acceptor materials, the V_oc of a solar cell can be calculated by the difference between the HOMO of the donor and the LUMO of the acceptor.

For the given materials, the V_oc can be calculated as follows:

A: V_oc = (−5 eV) − (−4 eV) = −1 eV

B: V_oc = (−5.1 eV) − (−4 eV) = −1.1 eV

C: V_oc = (−5.2 eV) − (−4 eV) = −1.2 eV

Therefore, the combination of materials with the highest V_oc is Material A, and the combination with the lowest V_oc is Material C”.

Apparently, the V_oc will not have negative values and the answers were wrong. Therefore, it is advisable to refrain from asking ChatGPT inferential questions as it may lead to incorrect answers being generated.

Lastly, the use of ChatGPT in organic materials research raises various ethical considerations that need to be addressed. One potential issue is the risk of biases in generated content due to the data biases that users provide. The use of AI tools, such as ChatGPT, could also have implications for the scientific community, potentially shifting the research focus away from important areas. Privacy and security concerns arise when processing sensitive or unpublished data, and researchers must ensure that appropriate measures are in place to prevent unauthorized access or misuse of data. The generated content may be subject to intellectual property laws, creating legal challenges and the risk of copyright infringement. Accountability is another important factor that needs to be considered in the use of AI tools, such as ChatGPT, in scientific research, and researchers must be transparent about their use and accountable for the research outcomes, including any potential inaccuracies or biases in the generated content.

3. Conclusions

To conclude, ChatGPT is able to provide assistance in many new research areas, and we are still at the very early stage of exploring its application scope. I have no doubt that AI tools will become a game-changing player in all fields, including our scientific society. However, there are many challenges in terms of research purposes. So far ChatGPT is limited by computational constraints, the ability to point out inaccurate information or false faces, and inferential capability for scientific research. All those limitations may lead to misunderstandings or misinterpretation. To address these challenges, researchers are immersing themselves to improve the accuracy and reliability of the model.

Emerging technologies and advancements in AI have had a significant impact on the development and performance of language models, such as ChatGPT. For example, new AI techniques, such as deep learning and neural networks, have enabled the creation of more sophisticated and accurate language models that can process vast amounts of data and learn from it.

In addition, the rise of big data and cloud computing has made it possible to train and run language models at scale, opening up new possibilities for applications, such as chatbots, virtual assistants, and language translation. Moreover, the development of pre-trained language models, such as GPT-3, has reduced the amount of data and computing resources required to train new models, making it easier for researchers and developers to create their own language models.

For example, collaboration with linguists can help to refine language models by incorporating a better understanding of the nuances of human language and communication. Collaboration with computer scientists can lead to the development of more efficient and effective algorithms for training and running language models. Collaborations with psychologists and neuroscientists can help to refine language models by providing insights into how the human brain processes language.

Overall, the impact of emerging technologies and advancements in AI on the development and performance of language models, such as ChatGPT is significant, and the potential for interdisciplinary collaboration is vast. By leveraging these advances and collaborating across disciplines, we can continue to drive innovation in the field of language modeling and unlock new possibilities for applications in a wide range of industries and contexts. I strongly believe that good use of the fast-growing LLM models, such as ChatGPT, can benefit scientific research as a revolutionary tool and can boost technology development.