Bridging the Gap in the Adoption of Trustworthy AI in Indian Healthcare: Challenges and Opportunities

Abstract

The healthcare sector in India has experienced significant transformations owing to the advancement in technology and infrastructure. Despite these transformations, there are major challenges to address critical issues like insufficient healthcare infrastructure for the country’s huge population, limited accessibility, shortage of skilled professionals, and high-quality care. Artificial intelligence (AI)-driven solutions have the potential to lessen the stress on India’s healthcare system; however, integrating trustworthy AI in the sector remains challenging due to ethical and regulatory constraints. This study aims to critically review the current status of the development of AI systems in Indian healthcare and how well it satisfies the ethical and legal aspects of AI, as well as to identify the challenges and opportunities in adoption of trustworthy AI in the Indian healthcare sector. This study reviewed 15 articles selected from a total of 1136 articles gathered from two electronic databases, PubMed and Google Scholar, as well as project websites. This study makes use of the Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews (PRISMA-ScR). It finds that the existing studies mostly used conventional machine learning (ML) algorithms and artificial neural networks (ANNs) for a variety of tasks, such as drug discovery, disease surveillance systems, early disease detection and diagnostic accuracy, and management of healthcare resources in India. This study identifies a gap in the adoption of trustworthy AI in Indian healthcare and various challenges associated with it. It explores opportunities for developing trustworthy AI in Indian healthcare settings, prioritizing patient safety, data privacy, and compliance with ethical and legal standards.

1. Introduction

Artificial intelligence systems refer to computational models that use machine learning techniques to operate autonomously or with human assistance. These systems take data as input, interpret human-defined goals, and then use statistical and mathematical techniques to make decisions, recommendations, or predictions that influence digital or real-world circumstances [1]. Trustworthy artificial intelligence (AI) [2] refers to AI systems that are built and run with reliability, ethics, and human values in mind. According to the Independent High-Level Expert Group on Artificial Intelligence study [3,4] from the European Union, this means ethics, legality, and robustness that respect human autonomy, rights, and dignity without compromising or unfairly affecting people’s privacy [5].

Trustworthy AI is a framework of principles and practices that aims to ensure the safety, fairness, transparency, and social usefulness of AI technologies as shown in Figure 1. There are numerous applications for AI in healthcare; however, if AI tools and technologies are adopted in this field without taking into account issues such as explainability, transparency, data privacy, and equity or treating all users equally, there may be significant consequences.

1.1. Background

The adoption and implementation of AI technologies has been expanding the significance of healthcare systems for a few years now as they improve different aspects of diagnosis, clinical trials, and patient care, including healthcare robotics, thus improving the whole process involved in healthcare for doctors, healthcare workers, and patients. One such sphere is medical imaging, where AI algorithms support specialists in trying to identify cancers early [6,7], which is often not achievable through early diagnostics alone. In addition, AI-based models are developed to predict disease outbreaks [8,9] and develop risk stratification models for patients.

In general, there are adoptions of AI technology in improving treatment planning and decision making in congenital heart disease [10], telemedicine [11] for remote consultations, health monitoring [12], and management of chronic diseases such as diabetes [13] or hypertension in real time [14]. This is especially useful in rural areas and in developing nations with a shortage of healthcare professionals. In addition to patient care, the use of AI is a game changer in the field of drug discovery [15], reducing the time to provide advanced therapeutics to patients by predicting the outcome of new medications and their interactions in a cost-effective manner. In addition, it improves the management of hospital resources by improving administrative procedures, staff and bed allocation, and supply chain management [16].

Just as many other developing nations, India is committed to the innovation and adoption of artificial intelligence in healthcare. As illustrated in Figure 2, there is an increasing trend in the applications of AI in healthcare [17]. The artificial intelligence (AI) market in India was estimated to be worth USD 374.7 million in 2023. The value is predicted to increase significantly to approximately USD 6.9 billion in 2032. This trend indicates the growing interest in research and innovations in AI-based applications in the Indian healthcare sector.

India has an estimated population of 1.428 billion people and is the sixth largest country in the world with an area of 3,287,263 square kilometers. The Indian government does not own the entire healthcare system. In reality, there are private organizations working in partnership with the public sector to meet the challenging demands of a diverse population for high-quality healthcare facilities. Some of the key challenges faced by the Indian healthcare sector, highlighted by the WHO, include inadequacy of qualified healthcare professionals and infrastructure, non-uniform accessibility to healthcare services throughout the country, the inaffordability of healthcare facilities primarily for people living in rural areas, and the lack of awareness among the masses about essential healthcare services, preventive care, and treatment options. The application of artificial intelligence (AI), along with machine learning (ML), robotics, sensor technologies, and the Internet of Medical Things (IoMT), has a great potential to make a greater impact in the Indian healthcare sector, and there is much evidence to prove it. AI-enabled technological solutions have led to better-informed decision-making and personalized diagnosis [18] and treatment taking into account genetic, environmental, and lifestyle factors. In particular, there has been growing evidence of the benefits of technological innovation, and particularly artificial intelligence, within healthcare systems in developed and low- and middle-income countries [19].

During the COVID-19 pandemic [20], the Indian government used artificial intelligence technologies to contain the spread of the disease and the distribution of vaccines. NITI Aayog, India’s policy think tank, had taken an initiative to detect tuberculosis by analyzing X-rays using AI technology, especially in those parts of the country where there is limited access to TB diagnostic tools. Wipro GE healthcare [21] had launched an AI-enabled cath lab in India to improve the affordability and quality of cardiac care for patients with cardiovascular disease. Similarly, Apollo Hospitals, in collaboration with Microsoft, developed AI tools for cardiovascular disease management. Healthcube, an Indian healthcare technology company, has developed and deployed portable diagnostic tool that allow real-time health monitoring such as glucose levels, blood pressure, and so on. The eSanjeevani platform, an initiative of the India’s Ministry of Health and Family Welfare, applies AI and IoT-based technologies for remote consultations and telemedicine services. The platform, with the support of the Common Service Centres, was instrumental during the COVID-19 pandemic and was widely used throughout India to bridge the healthcare gaps between the rural and urban population.

Artificial intelligence has the potential to make a positive impact in the Indian healthcare sector; however, there is the need for trustworthy AI that complies with ethical and legal considerations ensuring accuracy, reliability, transparency, accountability, and fairness to build trust and acceptance among stakeholders of the healthcare sector. Transparency in healthcare helps clarify the way decisions are made. Ethical AI [22] ensures impartial treatment that is easily accessible to everyone regardless of their diverse demographic background, respects patient autonomy by ensuring that patients have the right to make their own informed decisions about their own healthcare, and prevents misuse of sensitive data by AI systems, mitigating the risk of data breaches, especially in a diverse country such as India. Without these principles, AI could exacerbate the issue of protecting patient privacy and data security, limit accountability of the treatment process, and compromise patient safety. Baldassarre et al. [23] emphasize the importance of key legal and ethical considerations, underscoring the need for AI systems to be explicable, equitable, and accountable. Despite the growing discourse on trustworthy AI globally, the practical application in the Indian healthcare context remains underexplored. Challenges such as limited access or availability of unbiased datasets that can be used to train and validate AI-based models, lack of robust regulatory frameworks, and insufficient awareness in the general public persist. These issues underscore the need to understand the current state of AI development in the Indian healthcare sector, with a focus on how these AI systems meet ethical and legal standards. Moreover, identifying the challenges and opportunities for adopting trustworthy AI in the healthcare sector can provide actionable insights for stakeholders, including policymakers, healthcare professionals, and technologists, to create a legal and ethical AI ecosystem in the Indian healthcare sector.

1.2. Ethical and Legal Considerations in AI for Healthcare

The key ethical considerations [22] about AI applications in healthcare are as follows:

(a)

Patient Privacy and Data Security: Vast amounts of sensitive data which may be personal are used in AI systems; therefore, it is crucial to ensure patient privacy. Furthermore, the possibility of a data breach cannot be denied, making it essential to have a proper mechanism for secure data storage and data transfer, preventing unauthorized data access.

(b)

Informed Consent and Transparency: Patients should be informed and consent must be obtained before using their data for any diagnosis or treatment by AI systems. At the same time, AI systems should not be viewed as “black boxes”, which specifies that there should be transparency in the operation mechanism of AI systems and the risks involved in them.

(c)

Unbiased and Fair: AI models should be trained without bias in training data, and impartial treatment should be easily accessible to everyone regardless of their diverse demographic background.

(d)

Liability and Accountability: In the treatment process with AI systems, accountability is required to determine who is responsible for any incorrect treatments or recommendations: developer, healthcare provider, or organization. Adequate legal and regulatory frameworks must be in place to establish liability and proper compensation to affected patients for those decisions.

(e)

Explainability and Interpretability: Ensuring explainability and transparency in AI systems builds trust and improves cooperation among stakeholders in the healthcare system. Transparent decision-making processes that are interpretable for both patients and medical professionals make it easy to understand the requirement for a certain diagnosis or a recommended treatment plan, particularly in critical medical conditions.

The key legal considerations [23] about AI applications in healthcare can be stated as:

(a)

Data Privacy and Protection Laws: Healthcare data, such as patient medical records, treatment histories, and other information, may contain personal information and are generally sensitive in nature. These data are likely to be exploited by AI systems, making them susceptible to misuse and data breaches. To alleviate any exploitation, such a system is to be governed by data protection laws such as the General Data Protection Regulation (GDPR) of Europe [24], the Health Insurance Portability and Accountability Act (HIPAA) of the United States [25], and the Digital Personal Data Protection Act (DPDP) [26] of 2023 of India.

(b)

Regulatory Approval and Compliance: Before deploying an AI healthcare system in clinical settings, it must be approved by regulatory bodies. Even a small modification in the algorithms of AI systems may require approval from regulatory bodies. For example, the European Medical Device Regulation (MDR) requires that medical devices, including AI-powered systems, have CE markings. Likewise, in the United States, the Food and Drug Administration (FDA) is responsible for regulating AI/ML-based products [27].

(c)

Intellectual Property Rights: Developers can seek legal protection, copyrights, and licensing for their original work or innovations made in the AI healthcare system.

(d)

Cross-Border Regulations: AI healthcare systems which are deployed for any purpose in different nations must adhere to the laws and regulations prevalent in the multiple jurisdictions of different nations [28]. For example, an organization deploying an AI healthcare system in the US and Europe must adhere to both the HIPAA and GDPR regulations present in the US and Europe, respectively.

(e)

Patient Consent: Patients are entitled to see their medical records, ask for updates, and decide with whom their information is shared [29]. Before using patient data for an AI system to diagnose or treat them, they should be informed and their consent must be obtained.

1.3. Research Objectives

The advancement of AI-enabled technological solutions has led to a paradigm shift in the healthcare sector of India. The country is faced with numerous issues and challenges in an attempt to provide quality healthcare facilities to its diverse population, which the country can overcome using AI-driven solutions. However, not much is known about the advances made in the application of trustworthy AI in the healthcare sector of India. This scoping review intends to investigate the current state of trustworthy AI application, the challenges to be addressed, and future opportunities in the Indian healthcare sector.

1.4. Contributions

In this study, the following contributions are made to the discourse on legal and ethical aspects of AI in the context of Indian healthcare:

(a)

Scoping Review of Trustworthy AI in Indian Healthcare Sector: This study conducts a critical evaluation of the current state of the development of the AI system in Indian healthcare, assessing its adherence to ethical and legal standards.

(b)

Identification of Key Challenges and Opportunities: This study attempts to identify the challenges and opportunities associated with the adoption of trustworthy AI in the healthcare sector of India and offers future prospects and actionable recommendations.

(c)

Emphasis on Policy Frameworks: The study explores the potential of AI in healthcare sectors and emphasizes the need for robust legal and regulatory frameworks designed for India’s healthcare ecosystem in accordance with international standards on AI ethics and governance.

2. Methodology

2.1. Study Design and Setting

The developments and applications of trustworthy AI in Indian healthcare are the primary focus of this scoping review article. The Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews (PRISMA-ScR) [30,31] are used throughout this scoping review process. PRISMA-ScR is a framework used in the review process for a systematic and structured approach to present the findings of research studies. The study protocol was registered in the Open Science Framework [32].

2.2. Search Criteria and Selection Method

The search aimed to look for the current state of the development of AI systems in the Indian healthcare sector and to see how well they satisfy key ethical and legal considerations of AI, as well as to identify the challenges and opportunities in the adoption of trustworthy AI in the Indian healthcare sector. Keywords such as healthcare sector, trustworthy artificial intelligence, privacy, and India were used to conduct the search in the electronic databases and project websites in India. There were no restrictions on the type of publications published from 2015 to March 2024, although only articles written in English were considered for review. The scoping review study was initiated by searching two electronic databases and project websites. The initial search was performed manually on 16 May 2023, utilizing two databases, PubMed/Medline and Google Scholar. Subsequently, on 29 March 2024, another manual search was performed in these databases to identify any additional literature published since the initial search. We investigated databases and websites (see Appendix A,

Table A1. The databases and websites pertaining to trustworthy AI in healthcare in India.

Database/Source	Search Query	No. of Articles Returned
PubMed	((“india”[MeSH Terms] OR “india”[All Fields] OR “india s”[All Fields] OR “indias”[All Fields]) AND ((“health”[MeSH Terms] OR “health”[All Fields] OR “health s”[All Fields] OR “healthful”[All Fields] OR “healthfulness”[All Fields] OR “healths”[All Fields]) AND ((“privacies”[All Fields] OR “privacy”[MeSH Terms] OR “privacy”[All Fields]) AND ((“artificial intelligence”[MeSH Terms] OR (“artificial”[All Fields] AND “intelligence”[All Fields]) OR “artificial intelligence”[All Fields] OR (“antagonists and inhibitors”[MeSH Subheading] OR (“antagonists”[All Fields] AND “inhibitors”[All Fields]) OR “antagonists and inhibitors”[All Fields] OR “ai”[All Fields])) AND (“2013/05/12 00:00”:“3000/01/01 05:00”[Date—Publication] AND “review”[Publication Type] AND “loattrfull text”[Filter])) AND (“2013/05/12 00:00”:“3000/01/01 05:00”[Date—Publication] AND “review”[Publication Type] AND “loattrfull text”[Filter])) AND (“2013/05/12 00:00”:“3000/01/01 05:00”[Date—Publication] AND “review”[Publication Type] AND “loattrfull text”[Filter]))) AND ((y_10[Filter]) AND (review[Filter]) AND (fft[Filter]))	21
Google Scholar	’health sector’ OR ’healthcare sector’ AND ’trustworthy’ AND ’artificial intelligence’ AND ’privacy’ AND ’INDIA’	1110
Institution/Project Websites (accessed on 21 September 2024)	https://www.prnewswire.com/, https://www.ciengine.com/, https://cxotoday.com/, www.wadhwaniai.org/, https://techstory.in/	5

) related to trustworthy AI initiatives in the healthcare sector of India.

Articles that were found using our search criteria were initially imported into the Sysrev Web application [33]. The software is publicly available and is used mainly for systematic review and evidence-based research (SER), involving humans and machines collaborating to evaluate digital documents. In general, there are four phases involved in Sysrev, namely, creating data sources, defining labels, user review, and data export. Using the tool, after importing the searched articles, duplicates were first eliminated and then the articles were screened. Two reviewers had independently reviewed the title and abstract of the articles, and each reviewer categorized the article as ‘Yes’ if the article was to be included or ‘No’ if it was to be rejected. However, if there was a tie, then it was reviewed by the third reviewer independently to resolve the uncertainty. The review results were then exported to an Excel file for data charting and further analysis. Secondly, both the reviewers independently reviewed the full text of the articles to make final decisions about inclusion. This time, the third reviewer was not involved in reaching the consensus.

3. Results

3.1. Characteristics of the Studies Included in the Review

The PRISMA-Scoping Review diagram in Figure 3 illustrates that of the 1136 articles that were retrieved from the two electronic databases and the project websites, 15 articles satisfied the inclusion requirements for the scoping review.

This study explores various applications of AI in the Indian healthcare sector. Ten articles are peer-reviewed, as shown in

Table 1. Characteristics of publications on AI-driven healthcare innovations in India.

Study	Aim (Purpose)	Algorithm(s) Used
S Kaur et al., 2022 [34]	To develop a tool for data collection and analysis for prediction of dengue infection and risk estimation.	RF
S Shaik et al., 2023 [35]	To predict heart illness tolerance using machine learning algorithms. This concept enables the mobile app to identify the heart problem based on the patient’s heartbeat.	NB, DT, RF, SRP
S Kadian et al., 2017 [36]	To use two technological tools, telehealth and artificial doctor. Telehealth served as a communication medium between rural people and hospitals. Artificial doctor provides alternative treatment for unknown diseases.	ANN
KP Kalita et al., 2023 [37]	To combine the concept of blockchain and ensemble machine learning algorithms for data collection and storage, analyzing maternal data securely and reliably. Also, to predict risk levels of pregnancy complications.	RF
RK Deka et al., 2023 [38]	To develop a method to transform numerical data points into images for classification while keeping medical data private.	CNN
Geetha et al., 2022 [39]	To develop a technique for secure transmission of sensitive medical images in an open medium, preserving health data privacy.	PIOE-SMIM
Kumar, Anil et al., 2019 [40]	To develop a model to anonymize electronic health records (EHRs) with minimal loss of information and increased medical data privacy.	SVM, k-NN
Shanmuga Sundari et al., 2023 [41]	To use machine learning methods to forecast neuro-clinical data. The developed models predict the psychiatric disorders known as impulse control disorders (ICDs) in patients.	AB, SVM, NB, LR
Alnavar, Kruthika et al., 2021 [42]	To develop blockchain-based machine learning models to manage medical records and mitigate fraudulent health insurance claims.	RF, SVM
Malik, V et al., 2021 [43]	To develop models to predict the chance of survival of patients and response to medication for breast cancer.	ANN, KM

Abbreviations: RF = Random Forest, NB = Naive Bayes, DT = Decision Tree, SRP = Sparse Random Projection, ANN = Artificial Neural Network, CNN = Convolutional Neural Network, PIOE-SMIM = Pigeon-Inspired Optimization with Encryption-Based Secure Medical Image Management, SVM = Support Vector Machine, k-NN = k-Nearest Neighbors, AB = AdaBoost, LR = Logistic Regression, KM = K-Means Clustering.

, while five articles are additional articles (informal literature) discussed in Section 3.2 identified from other sources such as project reports and institutional websites. The selected articles are evaluated against key ethical and legal considerations in AI healthcare applications, using a defined scale of “Full”, “Partial”, and “Not Found” (see

Table 2. Evaluation of the selected articles against key ethical considerations for trustworthy AI in the healthcare sector.

Study	Patient Privacy and Data Security	Informed Consent and Transparency	Unbiasness and Fairness	Liability and Accountability	Explainability and Interpretability
S Kaur et al., 2022 [34]	Partial	Not Found	Partial	Not Found	Partial
S Shaik et al., 2023 [35]	Partial	Not Found	Partial	Not Found	Partial
S Kadian et al., 2017 [36]	Partial	Not Found	Partial	Not Found	Not Found
KP Kalita et al., 2023 [37]	Full	Partial	Not Found	Partial	Partial
RK Deka et al., 2023 [38]	Full	Partial	Partial	Not Found	Partial
Geetha et al., 2022 [39]	Full	Not Found	Partial	Not Found	Not Found
Kumar, Anil et al., 2019 [40]	Full	Not Found	Not Found	Not Found	Not Found
Shanmuga Sundari et al., 2023 [41]	Not Found	Partial	Not Found	Not Found	Partial
Alnavar, Kruthika et al., 2021 [42]	Partial	Partial	Not Found	Not Found	Not Found
Malik, V et al., 2021 [43]	Partial	Not Found	Not Found	Not Found	Not Found

and

Table 3. Evaluation of the selected articles against key legal considerations for trustworthy AI in the healthcare sector.

Study	Data Privacy and Protection Laws	Regulatory Approval and Compliance	Intellectual Property Rights	Cross-Border Regulations	Patient Consent
S Kaur et al., 2022 [34]	Partial	Not Found	Not Found	Not Found	Not Found
S Shaik et al., 2023 [35]	Not Found	Not Found	Not Found	Not Found	Not Found
S Kadian et al., 2017 [36]	Partial	Not Found	Not Found	Not Found	Not Found
KP Kalita et al., 2023 [37]	Full	Partial	Not Found	Not Found	Partial
RK Deka et al., 2023 [38]	Partial	Not Found	Not Found	Not Found	Not Found
Geetha et al., 2022 [39]	Partial	Not Found	Not Found	Not Found	Not Found
Kumar, Anil et al., 2019 [40]	Partial	Not Found	Not Found	Not Found	Not Found
Shanmuga Sundari et al., 2023 [41]	Not Found	Not Found	Not Found	Not Found	Not Found
Alnavar, Kruthika et al., 2021 [42]	Partial	Not Found	Not Found	Not Found	Not Found
Malik, V et al., 2021 [43]	Not Found	Not Found	Not Found	Not Found	Not Found

). The scale “Full” indicates complete alignment of the system with the relevant key considerations for trustworthy AI. “Partial” signifies the system addresses some of the key considerations relevant to trustworthy AI, and “Not Found” means there is no mention of the relevant key considerations for trustworthy AI.

A machine-learning-based tool is developed by Kaur et al. [34] to predict the presence of dengue infection and risk levels, based on symptomatic and clinical investigations. The system performs a real-time diagnosis and alerts patients to possible internal hemorrhage. The model has a high accuracy of more than 90% and high sensitivity and specificity values, with experimental evaluation indicating its performance efficiency and utilization. In this work, some aspects of trustworthy AI can be indirectly inferred. For example, the high accuracy of the model implies constant performance, which is a crucial component of trustworthy AI, and the real-time diagnosis represents equitable healthcare delivery. The study uses patient data but does not explicitly mention patient privacy, and data security. Shaik and Kakulapati [35] developed software using artificial intelligence algorithms to predict tolerance to heart disease. The mobile app uses the heartbeat to identify heart problems, enabling doctors to make an accurate diagnosis of heart conditions. The AI algorithms perform comparative learning from end-to-end graphical demonstrations. The key aspects of trustworthy AI in healthcare satisfied by the proposed work include heart disease prediction accuracy and the end-to-end graphical demonstrations of outcomes, indicating transparency. The proposed system may be helpful for doctors to visualize the results and understand the logic behind the predictions as a step toward explainability and interpretability and the social usefulness aspects of trustworthy AI.

Healthcare in India faces challenges in rural areas due to lack of expert advice and timely treatment. Computer science and telecommunication can help address this issue by developing telehealth and artificial doctors, which are addressed by Kadian and Chaturvedi [36]. Telehealth uses telecommunication resources to connect remote villages to city hospitals, while artificial doctors use artificial neural networks to provide the best treatment options for unknown diseases. These technologies harness human intelligence and improve healthcare in rural areas, inherently promoting accessibility, one of the key ethical considerations for the applications of AI in healthcare. The concept of central data storage partially addresses patient data privacy but does not discuss the compliance to specific data privacy laws.

Kalita et al. [37] proposed an approach to improve maternal healthcare by combining blockchain and predictive data analytics using an ensemble machine learning algorithm. The blockchain-based pregnancy complications prediction system uses a publicly accessible dataset to securely store maternal health data and predict the risk levels of pregnancy complications. The effectiveness of the system has been evaluated, addressing issues such as data security, reliability, transparency, informed decisions, and accountability in healthcare services. The use of blockchain technology ensures secure and reliable maternal data collection with informed consent; however, in the work, explicit mechanisms of patient consent and interpretability of the system are not discussed.

Deka et al. [38] presented a method to represent individual samples in an image, regardless of dimension, using a convolutional neural network (CNN) for classification. To preserve data privacy, images are randomly rotated for classification. Comparison of these approaches with related techniques shows promising results. However, the author does not discuss other key aspects of trustworthy AI, such as informed consent and transparency, unbiasness and fairness, data privacy and protection laws, and patient consent.

Geetha et al. proposed a technique [39] to solve the issue of security and privacy for health-related applications, where sensitive medical image data must be transmitted in an open medium without compromising the individual’s privacy. An encryption technique was proposed which can be applied to the medical domain with an optimal secret key generation process. In their work, a new method named Pigeon-Inspired Optimization with Encryption-Based Secure Medical Image Management (PIOE-SMIM) has been proposed achieving a promising result of MSE, RMSE, and Peak Signal-to-Noise Ratio (PSNR). Again, the proposed techniques do not address most of the key legal and ethical considerations for trustworthy AI.

Kumar et al. proposed a hybrid technique [40] to preserve the privacy of the Electronic Health Record (EHR). The author has demonstrated the utility of data by applying ML techniques, namely the Support Vector Machine (SVM) and k Nearest Neighbors (k-NN) classification algorithms. The proposed privacy models achieve protection against attribute disclosure, thus managing to protect privacy of individuals fully satisfying patient privacy and data security consideration for trustworthy AI; however, other key considerations are not found in the work.

Shanmuga Sundari et al. [41] used conventional machine learning algorithms and the AdaBoost ensemble technique to develop a model to predict Cerebellar Ataxia (CA) disease with high accuracy. The model helps to detect neurological disorders early through symptoms by analyzing gait (AoG) to automatically classify abnormalities in individuals according to their walking patterns. The authors do not explicitly mention any key considerations for trustworthy AI; however, the focus on comparative analysis of several ML models indicates a transparent evaluation process and the selection of the most reliable model for early detection of neurological disorders. The use of AdaBoost indicates to some extent the interpretability aspect of AI.

Kruthika et al. [42] proposed a machine learning model integrated with blockchain technology to manage patient data and categorize whether a newly admitted patient is covered by critical illness insurance. The blockchain-based smart contract with machine learning application provides a viable alternative to minimize fraudulent claims made to insurance companies by insurers. The system manages to build trust and transparency through the use of smart contracts among stakeholders. However, informed consent is not mentioned when using patient or claimant data. The use of blockchain technology also ensures data security and unauthorized access but does not explicitly address patient privacy. Other key considerations are not found in the article.

The alarming rise in breast cancer cases in India—roughly 180,000 instances were recorded in 2020; was the subject of Malik et al.’s study [43]. The authors improved on two aspects of their proposal: drug responsiveness during breast cancer treatment and prediction of patient survival. With an accuracy of 94%, they presented a framework that examines diverse omics data, including proteomics and genomes, to accurately predict the progression of breast cancer. The proposed neural network-based regression model is highly beneficial for precision oncology. The high accuracy of the model implies constant performance, which is a crucial component of trustworthy AI. The integration of multi-omics data involves the use of sensitive patient data; however, most of the key legal and ethical considerations for trustworthy AI are not found in the article.

3.2. Gray Literature (Table 4)

Apollo Hospitals has developed a Clinical Intelligence Engine (CIE) [44,45], which is essentially a decision support tool developed by leveraging an advanced AI-powered chatbot. The clinical decision support that can be used in the Apollo24|7 platform tool assists in primary care of patients and helps analyze patient health data and provide personalized, actionable insights to both patients and doctors. In another work, HealthPlix Technologies designed a system to streamline the process of treatment of patients, particularly for those with chronic diseases. Their AI-powered electronic medical records (EMRs) [46] which can be used both online and offline, help healthcare professionals manage patient health records with ease of access and provide personalized care. The Wadhwani AI [47], a nonprofit organization-led TRACE-TB project, funded by the United States Agency for International Development (USAID), aims to develop and implement cutting-edge artificial intelligence (AI) solutions in order to tackle COVID-19, eliminate the spread of tuberculosis by 2025 in India, and other infectious diseases in India. In the same line, Wadhwani AI [48], partnering with the National Centre for Disease Control (NCDC) under the Union Health Ministry, Government of India, developed tools to automate the existing process of the disease surveillance system in India using AI. It primarily helps to collect, collate, compile, analyze, and distribute real-time health data.

An Indian startup, JiviAI [49], has claimed to develop a large language model, Jivi Medx, that can be applied to the healthcare industry to improve the precision and effectiveness of communication between patients and healthcare professionals. The model facilitates an exhaustive examination of patient health records, enabling an early and accurate diagnosis of the disease based on symptoms. The Jivi MedX system can also be used to support medical research and examination while streamlining administrative processes.

Table 4. Recent developments in AI-powered healthcare innovations in India.

Study	Purpose
PR Newswire, 2023 [44,45]	Developing an AI-powered chatbot by Apollo Hospitals that provides personalized, real-time health insights to improve diagnosis accuracy, doctor productivity, and patient satisfaction in Indian healthcare.
T. S. Kushwah, 2023 [46]	To manage Electronic Health Records (EHRs) of patients and streamline the treatment process using an AI-powered EMR platform.
Wadhwani AI, 2022 [47]	To apply advanced AI solutions to tackle infectious diseases in India: USAID-supported TRACE-TB project, led by Wadhwani AI.
J. Agarwal, Wadhwani AI, 2023 [48]	To automate the disease surveillance system in India using AI.
I. Negi, JiviAI, 2024 [49]	To develop India’s healthcare language model.

Table 4. Recent developments in AI-powered healthcare innovations in India.

Study	Purpose
PR Newswire, 2023 [44,45]	Developing an AI-powered chatbot by Apollo Hospitals that provides personalized, real-time health insights to improve diagnosis accuracy, doctor productivity, and patient satisfaction in Indian healthcare.
T. S. Kushwah, 2023 [46]	To manage Electronic Health Records (EHRs) of patients and streamline the treatment process using an AI-powered EMR platform.
Wadhwani AI, 2022 [47]	To apply advanced AI solutions to tackle infectious diseases in India: USAID-supported TRACE-TB project, led by Wadhwani AI.
J. Agarwal, Wadhwani AI, 2023 [48]	To automate the disease surveillance system in India using AI.
I. Negi, JiviAI, 2024 [49]	To develop India’s healthcare language model.

The informal literature showcasing recent advances made by AI in India’s healthcare industry (see Table 4) references to privacy policies on their respective websites but lacks important ethical and legal aspects of AI, such as accountability, transparency, fairness, and compliance with laws such as GDPR or HIPAA. Their primary focus is on obtaining personally identifiable information from patients and using it with an emphasis on protecting personal data from data breaches. Patients must agree to the platform’s usage of their personal data in order to improve services, as stated in their privacy policies. It refers to data security measures to stop unauthorized access, but it skips over ethical and legal concerns specific to AI. Although it does not go into exhaustive detail about most of the ethical or legal issues unique to AI, it discusses data security procedures to prevent unauthorized access, which is in line with data privacy regulations. This study does not encompass understanding the exact procedure followed in doctor consultations, treatment procedures, and follow-ups, as it falls outside the scope of this review.

4. Discussion

4.1. Policy and Practice Implications

The World Health Organization (WHO) has released a set of guidelines [50] on ethics, governance, and applications of artificial intelligence (AI) in healthcare to its member states. In the same line, the National Health Policy (NHP) 2017 has been established in India with the objective of providing universal access to quality healthcare to all Indians regardless of their socioeconomic status. The policy focuses on various digital initiatives to deliver efficient healthcare services and has played an important role in bridging the gaps in the healthcare sector through its initiatives. It also promotes the creation of the National Digital Health Authority (NDHA) that would oversee the development, implementation, regulation, and utilization of digital health solutions. In 2018, NITI Aayog formulated a National Strategy on Artificial Intelligence (AI) titled “#AIforAll” with the aim of increasing human ability to address various economic and social challenges in five different sectors [51] viz. healthcare, agriculture, education, smart cities and infrastructure and smart mobility and transportation. To further build the national strategy on AI, NITI Aayog released its first part of the strategy titled “Towards Responsible AI for All” in the year 2021 [52], with an objective to establish broad ethics principles for the design, development, and deployment of AI in India—drawing on similar global initiatives but grounded in the Indian legal and regulatory context. The Srikrishna Committee was established in 2017 [53] with the objective of creating a framework for data protection in India, recognizing data privacy as a fundamental right of its citizens. Based on the recommendations made by the committee, the Personal Data Protection Bill (PDPB), aligned with the European Union’s GDPR, was proposed in 2019. The Information Technology Act, 2000 (IT Act) was introduced as one of India’s foundational legislations for cybersecurity, digital transactions, and data protection. Key government bodies involved in data protection, along with trustworthy AI in healthcare in India, include the Ministry of Electronics and Information Technology (MeITY), the Unique Identification Authority of India (UIDAI), and NITI Aayog among others.

India’s regulatory landscape for artificial intelligence (AI) in healthcare is fragmented, sometimes with conflicting policies. A collaborative international research and development process for AI systems may get hampered by the Personal Data Protection Bill, 2019, for instance, which places a strong emphasis on user consent and data localization for user privacy. Policies must bridge the gap between ethical AI principles and their application, upholding ethical standards, and fostering innovation. Clear guidelines are necessary to ensure that the last mile benefits of AI are achieved while incorporating AI into public health initiatives like the Ayushman Bharat initiative [54] of India. In order to facilitate interoperability and link national regulatory frameworks with international standards, policies should promote collaborations with global organizations like the WHO and the United Nations Educational, Scientific and Cultural Organization (UNESCO) [55].

4.2. Challenges Hindering the Application of Trustworthy AI in Healthcare in India

In this scoping review, we have also explored the challenges faced in the widespread adoption of trustworthy AI in the healthcare sector as reported by publications in the literature. Some of the challenges include limited access to or availability of unbiased datasets [56] that can be used to train and validate AI-based models. Other challenges include a lack of awareness in the general public about data privacy and lack of legal and ethical standards [57] to be followed in the application of AI in healthcare. In addition to these, the authors have reported the lack of proper policies and strategies and insufficient human, infrastructure, and financial resources to implement trustworthy AI in healthcare.

There are various applications of ethical AI across the sectors [58] in India, such as primary healthcare, medical research and drug development, hospital administration, public health surveillance, and mental health, to name a few. In order to comply with legal and ethical considerations, certain initiatives must be undertaken, such as developing inclusive datasets that reflect India’s diverse demographics, creating robust encryption protocols, assigning responsibility for errors made due to AI-driven systems, granting patients the right to make informed decisions, and continuous monitoring and updating of AI models. By ensuring fairness, accountability, transparency, and privacy in AI models, these sectors can adopt AI technologies that promote the well-being of the diverse population of India.

A scoping review was carried out to assess the current status of trustworthy AI in India. The review aims to understand how AI-based models are being adopted in Indian healthcare, with an emphasis on the requirement that these models comply with the legal and ethical standards for AI. To our knowledge and understanding, this kind of review is unique because it focuses on the adoption of trustworthy AI-based models in the healthcare sector of India. The number of relevant publications found is currently limited and modest, indicating that research in this area is still in a nascent stage, given the fact that there are multiple factors behind this. However, in recent years, the Indian healthcare industry has adopted AI technology, indicating a growing interest in adopting and integrating AI to improve healthcare outcomes. One can see a positive trend in the adoption of AI in the healthcare sector in India, but significant progress is still needed to ensure that AI technologies comply with guidelines regarding ethics, governance, and best practices in their applications in Indian healthcare settings. Addressing the associated challenges in aligning AI adoption in the Indian healthcare sector to comply with the transparency and explainability of AI solutions according to WHO guidelines remains a critical step in the journey.

4.3. Future Prospects and Recommendations

Adoption of AI and associated technologies has a great potential in the Indian healthcare sector. However, from the current findings, it is clear that there is a significant gap in addressing the legal and ethical considerations in the adoption of AI-based applications in the Indian healthcare sector. It shows that there is an urgent need to explore and work in this direction to design and develop trustworthy AI applications ensuring that they are used responsibly in Indian healthcare settings. To work in this direction, India must develop a robust data regulatory framework with legal support that complies with the WHO guidelines on the ethics and governance of AI for healthcare systems. Establishing guidelines for data anonymization and other measures to protect patient sensitive information should be the primary focus of data privacy and security. Simultaneously, the availability of quality healthcare data must be in place by creating a digital health infrastructure for data storage and seamless dissemination without compromising ethical and legal issues. Establishing a robust infrastructure to support the development and deployment of trustworthy AI algorithms in healthcare is essential. It is also essential that those involved in the healthcare industry are aware of and trained in the use of AI so that they can make informed treatment decisions. Collaborations between the corporate and public sectors should be promoted in order to advance research and innovation on the applicability of trustworthy AI in healthcare and its practices. Given that decisions can have a substantial impact on people’s lives, healthcare AI algorithms must be designed with interpretability in mind. Furthermore, given the socioeconomic diversity of India’s population, it must be impartial and fair.

4.4. Strengths and Limitations

This review article provides a broad assessment of the applications of AI-based technologies in the healthcare sector in India, emphasizing the legal and ethical standards currently in place. We believe that this review can play a key role in strengthening the research capacity in the area of trustworthy AI-driven healthcare service delivery and the development of future AI-based innovations that comply with ethical and legal standards. In this review, research is conducted mainly to assess research and publications in the applications of trustworthy AI and machine learning in the healthcare sector, specifically in India. The assessment covers relevant publications from the year 2015 onward, sourced from widely used electronic databases, namely Google Scholar and PubMed. In addition, an honest attempt has also been made to review the gray literature, assessing ongoing innovations and adoption of AI-based technologies by research organizations and hospitals aimed at improving healthcare facilities in India.

In this research, the study was mainly conducted on the applications of trustworthy AI-based models in the Indian healthcare sector, whereas we left out the research and publications made on the applications of trustworthy AI-based technologies in other sectors. We also acknowledge that we might have missed studies that did not use titles, abstracts, or words that were equivalent to the ones in our search query. Furthermore, we have limited our study to publications in the English language.

• Ethical Considerations:

There were no unpublished secondary data or human participants in this study, and ethical approval for human research was not needed.

5. Conclusions

A scoping review of 15 studies related to AI applications in the health sector in India and the market size of AI in Indian healthcare reveals a clear trend of growing interest in research and innovations in AI-based healthcare solutions. These technology-driven solutions are supported by the Indian government and private organizations in order to enhance healthcare outcomes nationwide. However, the study identifies significant gaps and challenges associated with the adoption of trustworthy AI-based applications in the Indian healthcare setting. The challenges include limited access or unavailability of unbiased datasets to train and validate AI-based models, lack of awareness among the general public about data privacy, lack of patient consent, and the absence of proper legal and ethical standards as per international standards. This study stresses further exploration and work to address several key legal and ethical considerations in the adoption of trustworthy AI in the healthcare sector of India, including the development of regulatory frameworks with suitable policies and strategies, the provision of proper education and training to bridge the skill gap, and investing in adequate infrastructural and financial resources. This study also emphasizes the collaborative effort that stakeholders must undertake to create a patient-centered, privacy-aware, and ethical AI ecosystem in the Indian healthcare sector.