4.1. Baseline
The current state-of-the-art (SOTA) model for information extraction, UIE [
13], was selected as the baseline model. This model was fine-tuned using the same training set, and the results after fine-tuning are shown in
Table 3. Since current entity extraction models can only label entities based on the content appearing in the text and cannot perform semantic understanding and summarization, the standards were lowered for the baseline experiment. The test set was designed based only on the content that appeared in the text.
The results indicate that in universal information extraction (UIE), the extraction of location, key items, and key actions is relatively better compared to organizations. This is because the first three categories are more general extraction tasks, for which the base model has been specifically trained, hence the better performance. However, the extraction of organization elements, which are particularly specific to police incidents, is less effective. This is primarily due to a lack of relevant pre-training and a deficiency in semantic understanding capabilities.
4.3. Base Model Fine-Tuning
After constructing the original training set, the base model was tested for the extraction of the four key elements, yielding the following results: The best results obtained in the table have been highlighted in bold.
As shown in
Table 4, for the extraction of location elements, the best-performing base model was Qwen. After fine-tuning, the best-performing models were Baichuan and Yi.
As shown in
Table 5, for the extraction of key behavior elements, the best-performing base model was Baichuan. After fine-tuning, the best-performing model remained Baichuan.
As shown in
Table 6, for the extraction of key item elements, the best-performing base model was ChatGLM. After fine-tuning, the best-performing model was Qwen.
As shown in
Table 7, for the extraction of organization elements, the best-performing base model was ChatGLM. After fine-tuning, the best-performing model remained ChatGLM.
As shown in
Table 8, considering the overall extraction performance for the aforementioned elements, the best-performing base model was ChatGLM, while after fine-tuning, the best-performing model was Baichuan.
For different base models, the initial performance on the four elements varies, and due to the lack of police incident data in the pre-training of current open-source large language models, such as definitions of police incident types and elements, the initial models perform poorly on the three types of elements other than the highly general location element, with both accuracy and recall rates below 60% overall. By using the LoRA fine-tuning method and initial training sets combined with police incident-related knowledge, all six base models showed improvement across the four categories of elements after fine-tuning, demonstrating the effectiveness of LoRA fine-tuning and enhancing the models’ understanding of police incident content.
The performance of the six base models varies, likely due to different pre-training datasets and training strategies used. After fine-tuning, Baichuan shows the best overall performance, and apart from Llama, the other open-source models are not significantly different from Baichuan, with accuracy and recall rate differences within 5%. The Llama model, mainly targeting English, performs poorly in Chinese, but after fine-tuning, both accuracy and recall rates improved by about 50%, showing significant enhancement.
Leveraging large language models for entity extraction effectively utilizes the semantic understanding capabilities of the base models. Taking the extraction of organizational elements as an example, the results can abstract various organization names, even if these names do not appear completely in the police incident text. Moreover, after fine-tuning, the models still retain their general capabilities, providing a foundation for further research and applications.
In summary, the Baichuan model, which demonstrated the best performance after fine-tuning, was selected for further data augmentation experiments.
4.4. Extraction Experiment Based on Data Augmentation
Subsequently, using the zero-shot police data augmentation method based on ChatGPT designed in this study, approximately 6000 simulated police reports were generated. The distribution of the simulated data was observed through dimensionality reduction.
The t-SNE dimensionality reduction visualization of both the generated data and the original data is shown in
Figure 4, where triangles represent the generated data and circles represent the original data. This distribution change can explain the observed performance gap between models fine-tuned on synthetic data and those fine-tuned on original data.
The training data were expanded to four times the size of the original dataset, with experiments conducted using generated datasets of 1000, 2000, 3000, 4000, 5000, and 6000 entries. These correspond to 1×, 1.5×, 2×, 2.5×, 3×, 3.5×, and 4× of the original data size, respectively. The results are shown in
Table 9, We have bolded the best results in the table.
The comprehensive performance trends as observed in the curve chart are shown in
Figure 5. To further elucidate the learning process and demonstrate the model’s convergence, we have included learning curves depicting the evolution of loss during training in
Appendix C. These curves provide valuable insights into the stabilization of the model’s performance with respect to the number of training samples, complementing the analysis presented in
Figure 5.
It can be observed from the overall upward trend in the data augmentation curve (
Figure 5) that the method for generating data is effective and improves model training. However, there is a noticeable decline in performance when the dataset is initially increased by 1.5 to 2 times the size. This may be due to the vector differences between the generated and original data distributions, which can disrupt the model’s understanding of the original data, leading to a slight decrease in performance. As the dataset is further augmented to 2 to 3 times the original size, the model gradually adapts to the new data distribution, and the performance improves and stabilizes. Beyond this ratio, increasing the data augmentation further does not significantly enhance the model’s performance due to the limited complexity of the generated data compared to real-world scenarios.
For the models using the original prompt and the designed prompt, the ablation study results before and after fine-tuning are presented as shown in
Table 10, we have bolded the best results in the table. The original prompt and the designed prompt are the prompts designed in the prompt engineering section of this paper (
Section 3.1). Specific examples of these prompts can be found in
Appendix B.
The ablation study results collectively indicate that prompt engineering and fine-tuning significantly impact the extraction outcomes. LoRA fine-tuning has the most substantial effect on improving the base models, followed by prompt design. Using the original prompts on fine-tuned models also enhances their performance, but selecting prompts consistent with the training set optimally leverages the model’s capabilities, thereby validating the effectiveness of the experimental strategies employed in this study.
4.5. Entity Alignment
Using the entity alignment method proposed in this paper with the help of ChatGPT, the entities were first vectorized by calling OpenAI’s embedding API. Elements with a similarity greater than 90% were extracted. Leveraging the large language model and using other elements as context, further alignment was performed.
After the first stage of screening, most of the entities were filtered out. The remaining entities were grouped by similarity for the second stage of screening, which effectively completed the entity alignment task. Finally, the results were verified manually.
The results of the second step are presented below:
’’’ # You will play the role of an Entity Alignment Expert. For each record, I will provide an organizational element from a police incident and other relevant elements from that incident for reference. Your task is to determine whether these organizational elements belong to the same entity and to separately output the elements that belong to the same entity, ensuring clarity and coherence.
## Record Content:
Record 1:
Organizational Element: [Civil Police]
Reference Elements: [Celebration Event]
Record 2:
Organizational Element: [Police]
Reference Elements: [Lost Phone, Gratitude, Phone]
Record 3:
Organizational Element: [Traffic Police]
Reference Elements: [Breakdown, No Power, Collision, Drunk]
Record 4:
Organizational Element: [Public Security]
Reference Elements: [Investment, Incident]
Record 5:
Organizational Element: [Public Security Bureau]
Reference Elements: [Verification, Phone, QQ Number]
## Output Format:
Same Entity Group 1: [Organizational Element 1, Organizational Element 2, …], Reason: xxx
Same Entity Group 2: [Organizational Element 1, Organizational Element 2, …], Reason: xxx
...
## Please align the entities according to the records above and output the results in the format specified.’’’
ChatGPT Output:
’’’ Entity Alignment Results:
Same Entity Group 1: [Civil Police, Police], Reason: Civil Police and Police are interchangeable terms in Mandarin, commonly used to refer to the same type of law enforcement officers.
Same Entity Group 2: [Traffic Police], Reason: Traffic Police specifically refers to traffic officers, which differ in function from general police.
Same Entity Group 3: [Public Security, Public Security Bureau], Reason: Both Public Security and Public Security Bureau refer to public safety agencies, but Public Security Bureau is a more specific institutional name.’’’
This method allows for entity alignment by modifying the prompts in a systematic manner. It fully leverages ChatGPT’s semantic understanding and contextual capabilities without compromising privacy. This approach effectively aligns the extracted key elements.