Monitoring Cropping Systems: From Data Collection to Cloud Database Storage Using Open Source Software ^†

Abstract

Agricultural cropping systems and experiments include complex interactions of processes and various management practices and/or treatments under a wide range of environmental and climatic conditions. The use of standardized formats to monitor and document these systems and experiments can help researchers and stakeholders to efficiently exchange data, promote interdisciplinary collaborations, and simplify modelling and analysis procedures. In the scope of the SoilCare Horizon 2020 project monitoring and assessment work package, an integrated scheme to collect, validate, store, and access cropping system information and experimental data from 16 study sites, was created. The aim of the scheme is to make the data readily available in a way that the information is useful, easy to access and download, and safe, relying only on open source software. The database design considers data and metadata required to properly and easily monitor, process, and analyse cropping systems and/or agricultural experiments. The scheme allows for the storage of data and metadata regarding the experimental set-up, associated people and institutions, information about field management operations and experimental procedures which are clearly separated for making analysis procedures faster, links between system components, and information about the environmental and climatic conditions. Raw data are entered by the users into a structured spreadsheet. The quality is checked before storing the data into the database. Providing raw data allows processing and analysing as each other user needs. A desktop import application has been created to upload the information from spreadsheet to database, which includes automated error checks of relationship tables, data types, data constraints, etc. The final component of the scheme is the database web application interface, which enables users to access and query the database across the study sites without the knowledge of query languages and to download the required data. For this system design, PostgreSQL is used for storing the data, pgAdmin 4 for database management administration, MongoDB for user management and authentication, Python for the development of the import application, Angular and Node.js/Express for the web application and spreadsheets compatible with LibreOffice Calc. The system is currently tested with data provided by the SoilCare study sites. Preliminary testing indicated that extended quality control of the spreadsheets was required from the system’s administrator to meet the standards and restrictions of the import application. Initial comments from the users indicate that the database scheme, even if it initially seems complicated, includes all the variables and details required for a complete monitoring and modelling of an agricultural cropping system.