How Data Mining Can Improve Road Safety in Cities

Abstract

Traffic collisions pose a serious problem for cities due to the annually increasing number of vehicles. Information about incidents that occur on roads is important for the corresponding monitoring bodies, authorities, and emergency services. To ensure traffic safety, the data have to be visible, clean, and transparently displayed. This research was, therefore, aimed at developing a methodology for monitoring motor vehicle collision data and applying visualization techniques to evidence from New York City. The method showed that the largest number of motor vehicle traffic crashes occurred in Lower Manhattan due to its high population and traffic density. With these data, the road agencies of the city can put potentially dangerous road sections under control and make them safer for both drivers and pedestrians. Further development of the system may be associated with data analytics and visualization, resulting in new layers of heatmaps that not only provide details on car collision hotspots, which serve as the main target indicator for traffic safety authorities, but also break them down into social facilities, such as schools. This feature will enable assessment of how safe it is around a school and the evaluation of the impact of an underpass or a traffic enforcement camera on the number of collisions. The motor vehicle traffic crash (MVTC) monitoring system will help in comparing city districts and regions in terms of safety, seeing trends, realizing what exactly is happening at interchanges, and understanding the reasons behind. The methodology, in addition, can be supplemented with an analysis of risk factors for MVTCs, the efficiency of adopted measures and road renovations that are carried out, and many other functions.

1. Introduction

The high road accident rates across all countries, the ever-increasing numbers of persons who are injured and of deaths due to traffic collisions, the considerable property damage to vehicles, the harm to and loss of cargo, and the damage to traffic structures pose one of the most acute social problems in the world. Ensuring road safety is becoming a global challenge for humanity.

According to Tufts University (Boston, MA, USA), traffic collisions are some of the most common domestic emergencies1. The USA has one of the highest road densities in the world, with more than 227 million licensed drivers and 280 million vehicles in operation. This is the reason behind the large number of motor vehicle traffic crashes (MVTCs); in 2018, there were more than 12 million car collisions across the country; as the statistics for 2019 show, motor vehicle crashes took the lives of 36,120 people in the US, which resulted in 11 deaths per 100,000 inhabitants, while in 2020, more than 10 million MVTCs occurred.

Tracking how, where, and why MVTCs happen requires the application of dataset analysis. Most of the information in the modern world is obtained through visual perception, with heatmaps being one of the main data visualization techniques. The methods for their generation can also be employed when developing monitoring systems for emergencies such as MVTCs.

The purpose of this study consists in devising a methodology for monitoring MVTCs based on a heatmap drawn in Python. The MVTC heatmap uses data on the number of traffic collisions that occurred in New York City over 8 years and helps track the intensity of crashes in its different boroughs and the scenes of the incidents; as a result, a set of measures that are likely to prevent emergencies on the roads is worked out. The main task is to single out the locations with the highest MVTC rates.

As an analytical tool resting on artificial intelligence, the heatmap aids in figuring out the relationships between various functions in a dataset. It is a way of representing the data values as colors, where the redder, “hotter” shades are used to denote a strong correlation between elements, while the lighter, “colder” (or bluer) shades reflect a weak one. Thus, heatmap analysis can become one of the best ways to leverage the results from the collected data.

2. Previous Research on the Topic

A heatmap, also known as a thematic map, is a method for visualizing the intensity of a particular indicator within a specific area on the map. Data on the map can be assigned a hatching of different patterns, colors of a certain degree of saturation (choropleth map), or dots (dot distribution map) (Borland and Taylor 2007). A thematic map provides a graphical representation of statistical data that are divided into groups based on geographical factors (for example, by administrative divisions). It portrays territories on the map, which are colored (or hatched) in accordance with the value of the studied indicator observed across the area.

The history of computer-generated heatmaps originated in 1991 when C. Kinney coined the term to depict financial market information in real time (Rowell et al. 2020). Hierarchical color coding is employed in various data representation systems—for instance, in cluster analysis (Yu et al. 2020), web analytics (Kirsh 2021), and generating biological heatmaps (Chen et al. 2020).

It should be noted that, apart from colors, other graphical means can be used as a third coordinate to display 3-D data in 2-D space. For example, as shown in Lu et al. (2021), when plotting a bubble diagram, the authors expressed the third dimension through the area or size of a circle. In Friendly and Wainer (2021), the values of the variables were visualized with the height of the bars in the bar chart. Supplementary dimensions can be added by including new groups of data.

In recent years, there has been a large number of studies covering the development of visualization systems for aiding traffic safety. Let us take a closer look at the latest major scientific developments in data mining that can be applied in order to make urban traffic safer. In Bishop (2021), for example, I. Bishop considered the impact of visualization on infrastructure and changes in the landscape of the area. The author suggested utilizing data mining to examine the spatial correlations between indicators. The method showed that median visibility could be seen as the best condition for analyzing and visualizing the processed data.

In Albuquerque et al. (2021), the authors worked out an intelligent transport solution for Lisbon. They applied a cross-industry standard process, CRISP-DM, to understand problems with the traffic in the city using artificial intelligence. The methods of analysis and classification were employed to make models for calculating traffic indices that were correlated with external factors.

A visualization system based on urban traffic data mining was described in Bachechi et al. (2022). The use of such a system allows the areas of roads that are congested at specific times to be detected and enables the monitoring of traffic sensors. The authors produced a dashboard based on traffic sensor data that simulated traffic flow in two cities in Spain. The application of the dashboard was aimed at reducing traffic issues and mitigating transport emissions.

The need for increased traffic safety as a societal concern was reflected in the paper by Berres et al. (2021). The authors used both abstract and spatial visualization methods to carry out data mining of big traffic safety datasets collected in Hamilton County, Ohio, USA. The traffic safety indicators were captured in different locations: on the highways, in the city center, and in remote areas. The study stands out from all the rest precisely because of the employment of spatiotemporal patterns that characterize motor vehicle collisions in order to give a better understanding of common incident consistency in the collected data.

A system for improving traffic safety in the UK based on the incorporation of the visualization method into machine learning was devised in 2017 (Liu et al. 2022). It utilized a hybrid algorithm, LightGBM-TPE, which is claimed to perform better than other similar machine learning algorithms in terms of accuracy and recall. The algorithm established the geographic coordinates, time, and day of the week as four risk factors for traffic collisions.

The application of data mining to the visualization of MVTCs in Saudi Arabia was explored in Almajyul and Al-Ghreimil (2022). The paper introduced a software tool that represented a system for visualizing motor vehicle collisions and eased the perception of traffic data in order for the authorities to be able to limit their number. The system integrated a word-cloud, a bar chart, an interactive map, a doughnut chart, and a calendar as visualization methods.

An extensive review and analysis of scientific studies on intra-city traffic data visualization was presented in Clarinval and Dumas (2021). The authors identified mobility issues such as pollution and congested roads. Information visualization was noted as a medium for making the most out of these traffic data. As a result of the study, the authors discovered that there was insufficient attention to research on MVTC visualization and a lack of visualizations that engaged citizens in reflection on mobility-related issues.

In view of the above, creating a system for monitoring MVTCs in a major city will give a fresh look into the problem of car collisions and make it possible to find optimal solutions for reducing their number (Butsenko 2015; Shorikov and Butsenko 2021). The task of developing new tools for monitoring motor vehicle collision data and subsequent decision making aimed at handling the traffic proves to be critical and relevant. Meeting this challenge will stimulate sustainable development of the city.

3. Materials and Methods

The main techniques employed to devise the proposed methodology for monitoring MVTC data comprised data-mining methods, namely, methods for big data visualization. The solution was based on Python, a high-level programming language that was used to write the software.

PyCharm, an integrated and cross-platform environment, served as a development tool.

Since Python is a console programming language, to display the final result as an MVTC heatmap, it needs a library that enables the output of certain graphical information (Lienemann et al. 2020). This particular solution used the Jupyter Notebook library, which is an interactive shell for the Python programming language. It is especially good as a standard command shell of an operating system, particularly with regard to Windows, where the basic one has limited capabilities. The default behavior is similar to that of the shells of UNIX-like systems; however, operating in a Python environment leads to greater customization and flexibility. Thus, Jupyter Notebook is the most convenient web environment for Python in order to combine code, text, and images and to distribute them to other users.

Another software library, Pandas, which provides a means of data processing and analysis, was required to work with .CSV files. It is built on top of the NumPy package, which is considered to be a lower-level tool (Harris et al. 2020). The library contains data structures and operations for manipulating numerical tables and time series. Its main functions involve collection, cleaning, initial assessment, analysis, and modeling of data (Lemenkova 2020; Sapre and Vartak 2020).

To map heat markers, the Folium package was used, which is an open-source intuitive Python library that visualizes spatial data in an interactive manner.

4. Results

The results of the study are summarized as a description of a step-by-step methodology for monitoring and visualizing the MVTC data and their implementation.

4.1. Algorithm and Development Diagram

Setting up a system for monitoring traffic collisions starts with devising an algorithm and preparing a development diagram for it, which are required in order to have a full understanding of how the finished software will work.

The algorithm for creating an MVTC monitoring system includes the following steps:

Step 1. Initializing the dataset. Downloading the “Motor Vehicle Collisions—Crashes” dataset from data.cityofnewyork.us.

Step 2. Initializing libraries: OS (enabling work with files: deleting, saving, or modifying), Folium (for plotting a heatmap based on a clean map and heat markers), Pandas (allowing operations with a dataset: deleting, modifying, or creating data).

Step 3. Defining variables: carCrashDataset—for further work with the dataset.

Step 4. Cleaning the longitude and latitude columns of incorrect data.

Step 5. Creating the locations variable to store the clean data.

Step 6. Generating an empty (clean) black-and-white map and saving it to variable m.

Step 7. Plotting heat markers using the Heatmap function.

Step 8. Generating a heatmap by combining the results from steps 6 and 7.

Step 9. Saving the map as a Heatmap.html file.

Step 10. The final step of the algorithm: displaying the map on the screen.

Figure 1 shows a flowchart of the development of the MVTC monitoring system.

Before beginning the development process, the workspace needs to be prepared. It involves installing the required software, such as the programming language, development environment (and making adjustments to the configuration), and appropriate libraries.

The program design stage comprises coding and outputting the result. If the program does not run or operates incorrectly, the previous stage needs to be repeated until the desired result is produced (Brink et al. 2016).

4.2. Plotting the MVTC Heatmap

Generating a traffic collision heatmap entails downloading a dataset in .CSV format from the official New York City Open Data website. The dataset provided by the New York Police Department has almost 2 million registered motor vehicle collisions and is updated daily. The file size is about 300 MB (Figure 2) (NYC Open Data 2021).

According to step 2 of the aforementioned algorithm, the Jupyter environment must be properly initialized. The library may not work correctly due to the need for loading a large amount of data. To avoid this problem, the config value, iopub_data_rate_limit, must be changed by entering a specific command (Figure 3) (Lu et al. 2021).

The environment is fully ready for the development after successfully opening it up and creating a new project.

Next, the required Folium and Pandas libraries need to be initialized along with the OS system library. This will help save the final result as an .html file in order to view it without the need for an internet connection and independently of the project (Figure 4).

After that, in accordance with step 3, direct work with the MVTC dataset can be begun. The entire data array is loaded into the memory by using the read_csv function. It should be remembered that this dataset is very large, and to operate on it properly in the Pandas library, the low_memory parameter has to be adjusted.

During the processing of the dataset, it turned out that several cells of the latitude and longitude columns were missing values. To avoid errors and incorrect calculations of the coordinates of points on the map, they were temporarily assigned the value 0 with the fillna function (Figure 5).

Once this is done, all fields have the correct values and can be further transformed. Subsequent testing revealed that the set contained incorrect width and longitude data. The width and longitude vary between –180 and 180 degrees and must be non-zero. The corrections need to be made by excluding such data. Firstly, the zero values are deleted, followed by those beyond the range of –180 to 180 (Figure 6).

The set contains data since 2012. To make it relevant, we drew a heatmap for the whole year, that is, for the previous 12 months. One of the greatest advantages of the Pandas library is that it can read dates as strings. To process annual data, it is necessary to set a condition that the date in the CRASH DATE column is less than or equal to a specific value in the last year—for example, from 13 May 2020 to 13 May 2021 (Figure 7).

The data for generating the MVTC heatmap are ready. Since this dataset has other columns, those required for further use can be written to the locations variable (Figure 8). The appropriate data are now prepared for the heat overlay.

To put the heat overlay on, the map has to be initialized by utilizing the Folium library. In this paper, a stamentoner map was generated to produce a clean black-and-white image (Figure 9).

Step 7 involves uploading the ready-to-use data onto the map and applying the preset locations variable, followed by specifying values for the radius and min_opacity parameters. The radius and min_opacity are 20 and 0.1, respectively (McKinney 2017). Radius = 20 means that the point plotted on the map will be approximately 20 meters. Min_opacity refers to the arrangement of the colors in sequence from cold to hot, where 0.1 is the coldest (blue color). Max_opacity is calculated automatically. After all of the values have been loaded and processed, the data are plotted on the base layer generated above (Figure 10).

To produce the outcome, the m variable is employed, which contains a rendered map and the result of the HeatMap function (hotspots and their depth with radius) (Figure 11).

The plotted map shows water areas around the city in black, Northern Brooklyn and the South Bronx are highlighted in yellow, and Lower Manhattan is highlighted in red, which indicates a large number of car collisions in this part of the city due to the high population and traffic density.

To be able to view the output outside this environment, we shall use the save function and save the map as a heatmap file in .html format to the local storage (Figure 12).

The final heatmap can be viewed and zoomed in the browser (Figure 13).

Figure 14 shows a zoomed-in MVTC heatmap. Figure 15 demonstrates a zoomed-in heatmap of traffic collisions in the western part of the city.

This completes the description of the implementation of the methodology for monitoring and visualizing the data on MVTCs in a major city.

5. Conclusions

Traffic safety is a challenging area that needs to be studied using scientific methods. The understanding and prediction of MVTCs are key for social engineering in general and urban planning in particular. The worked-out methodology for analyzing and visualizing MVTC data will contribute to a better understanding of the traffic in major cities and give the authorities valuable insights.

The study resulted in a semi-independent MVTC heatmap that was developed to track the locations of motor vehicle collisions and their intensity in a particular area of New York City with a time lag of 1–2 days.

With these data, the road agencies of the city can identify potentially dangerous road sections and make them safer for both drivers and pedestrians or issue public service announcements to be broadcast on the TV or the internet to raise public awareness and promote responsible and attentive driving.

The application of the proposed MVTC monitoring system is associated with the processing and transmission of the heatmap data directly to the emergency services of the city. In such emergencies, the appropriate services will be able to analyze the intensity of traffic collisions throughout the city and make rational decisions when they occur. Further lines of effort will be related to enhancing the efficiency of obtaining and processing data on the MVTCs in cities. On top of this, more types of data representation and visualization will be added—for example, hierarchical (group) visualization in order to enable the study of processes at different levels of detail (Faccin 2022). Hierarchical visualization has great potential for the research and online monitoring of multidimensional dynamic data.