Construction and Evaluation of QOL Specialized Dictionary SqolDic Using Vocabulary Meaning and QOL Scale

Round 1

Reviewer 1 Report

Overview

The authors propose an automated system to estimate QOL from users’ conversations. This system is intended to alleviate the burden that conventional in-person QA questionnaire methods impose on the responder.

Strengths

1. The paper proposes an impactful, because useful, practical solution to the problem of estimating the QOL of people.
2. Leveraging a text corpus comprising human conversations as training data to address the QOL estimation problem appears a resourceful, and potentially impactful, approach since people often discuss various aspects of their lives in their conversations. The solution then would alleviate the need for them to discuss additionally their life conditions in a questionnaire which imposes additional time investments from users to rethink various aspects of their life in a seemingly artificial setting of an interview.
3. The automated QOL questionnaire answering system itself seems to have a very innovative workflow (ref. Figure 4).

Weakness

It seems, however, that obtaining a corpus of human conversations may not be so easy owing to ethical restrictions regarding privacy protection. In which case the solution would not work?

Comments

1. Figure 1 is not clear in reference to the text in the Introduction. What is the connection between the first and the second box? Further in the second box, what does PN stand for in “PN dictionary”? This “PN” is indeed explained later in the Methods discussion. I assume it is positive-negative. So perhaps this Figure is better suited to be placed after the discussion of the Methods. A figure in the Introduction is a good idea; however, it would have to be a less detailed Figure (a better abstraction of the approach) in the Introduction that does not reveal specifics without explaining or offering an idea of what it might be.
2. It is only in line 140 that it becomes apparent the paper is proposing a solution for Japanese text. It would be good if this were mentioned already somewhere in the Abstract and Introduction. In the line of text-based systems, the language considered in the study is usually an important and distinguishing characteristic of the work.
3. Line 145: what are considered to be “basic words and phrases”? Are they the words “contained in the SF-36v2 QOL questionnaire and the words contained in the description section regarding SF-36v2 on the SF-36v2 seller’s website?” If so, then maybe a different terminology such as QOL-focused terms could be used at point 2. Additionally, it could be rereferenced in the text in lines 151 to 152, so the reader can draw the connection.
4. In equation 1, how was the weight itself calculated?
5. Section 3.3 – Discussion of the system that automatically answers the QOL questionnaire. Can the system select which user utterances are actually relevant to the QOL questionnaire themes? It is unclear how the input is selected. I do not mean to imply that if the system has no such mechanism that it takes away anything from the contribution of the work – it still seems strong. However, it needs to be clarified that the system relies on preselected user input known to inform the QOL estimation task because, in general, conversations can be unrelated too.

Questions for the authors

How many instances in SqolDic?

Table 1. Could the table be modified to show at least one or two representative examples from all eight QOL scales? It seems to show only one, i.e. “physical functioning.” Was there a reason for presenting just one scale?

I think since the SqolDic is a main resource contribution of this work, it should be better clarified in terms of how representative each QOL type is in terms of instances count, etc. So if Table 1 could be modified as a Table showing summary statistics that would solve this issue. Also what was the expertise/background of the annotator who created the dataset? Was it created by multiple people? Such a discussion is necessary for the research community to discover how reliable and meaningful a resource is.

Can the SqolDic dictionary not be publicly released?

Typos, Language

Figure 2: “Automatical” à “Automatic”

Line 195: “described” is redundantly repeated.

Author Response

We are grateful to you for your comments and suggestions to improve the paper. Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 2 Report

The authors present a refinement of methods for analyzing the state of human users from textual information generated as they interact with artificial agents. The utility of this work has broad potential application as we collectively engage more with these types of agents. Further, the method described and analyses presented indicate substantive incremental progress toward the goal of AI that is more responsive to changes in human behavior. I note below some suggested changes that may improve the final product.

The language use could use a polish, as some disfluencies tend to distract from the substance. Please note especially a strong tendency toward passive voice and occasional disagreement between subject and object (the latter issue I only noticed in the introduction).

The introduction includes points which may require some further explanation. Notably:

• Establishing that QOL is an appropriate, established metric for evaluating and adjusting to proximal changes in interaction behavior. The multi-faceted metric is conventionally (I believe) used for a holistic understanding of an individual's well-being, rather than for tracking changes of state, as indicated here. It may well be that a kind of baseline QOL metric informs or moderates changes that would arise from previously used adjustment strategies that are overly proximal, but the authors do not seem to make this case (at least not until line 173, and then with only technical justification). 
• Line 108-109 makes a conclusion that does not logically follow from the two previous citations. Findings indicating that people disclose more negative information to agents could easily be counteracted by a decrease in total information. Further, neither of the references indicate that the information gathered is real-time. 

Other notes:

• Figure 1 may appear too early. I found myself unable to follow the steps presented because many of the components had only been briefly introduced. Later, I came back to it and found it useful, but by then I was scolling over a few pages. 
• Figures 5 & 6 are virtually indecipherable. The colors are too similar, and their labels are difficult to maintain in working memory while also tracking the unexplained abbreviations for factors in QOL that haven't appeared in full since six pages ago. 
• In both the introduction and conclusion, the authors make the case for improving long-term relationship building between humans and AI. This feels like a leap from the advancements described. I agree in the general merit of this goal, but the method described extends analysis only three conversational turns backward. This does not qualify as a relationship developed over time, in my opinion. 

Author Response

We are grateful to you for your comments and suggestions to improve the paper. Please see the attachment.

Author Response File: Author Response.pdf