6LWHZLVHPHDQRILQWHUREVHUYHUDJUHHPHQW

**Figure 9.** Site wise mean agreemen<sup>t</sup> ratio based on inter-observer agreements.

The results of inter-rated reliability analysis reflect a very good level of agreemen<sup>t</sup> at 83% which indicates the potential soundness of the methodology and the resulting VPA tool. It is important to highlight that out of 100 observer pairs, the minimum agreemen<sup>t</sup> ratio is 73% while the highest agreemen<sup>t</sup> ratio is 93%. The key reason behind the higher agreemen<sup>t</sup> ratio is that the tool collects data on 205 variables out of which 134 can ge<sup>t</sup> discrete answers with almost no potential of variance in observations (if observers are well trained).

From the initial testing, it is clear that the developed VPA tool tends to mitigate the previously identified limitations related to the assessment and quantification of visual pollution. The tool can deal with the subjective problem of visual pollution in a more objective way. It offers a wider coverage of 40 VPOs in local urban settings. Furthermore, it can record detailed characteristics against each VPO and quantify them by assigning weights and calculating visual pollution score. The tool can be used at both the micro and macro level scale for the measurement of visual pollution, i.e., nodes, street, neighborhood, or a city. The effective utilization of the tool requires the collection of geolocation and pictorial evidences as complementary information. In addition to the VPA scorecard, the collection of such attributes requires handling of additional gadgets (GPS device, camera or a hybrid device). From initial testing, it has been learnt that careful attention must be given to the systematic storage and labeling of these additional pieces of information so that they can be synchronized with the score card data.

### **5. Conclusions and Future Work**

In this paper, we have presented a novel methodology for the systematic development of a robust and consistent VPA tool, which provides a structured mechanism for quantification of visual pollution at any given location by measuring the presence and characteristics of various VPOs. Since the VPO characteristics have been explicitly quantified, the tool provides the resultant quantitative score representing the level of visual pollution on a scale of 1-100. The incorporation of (diverse) expert opinion makes the methodology robust and suitable for almost all urban areas of Pakistan. Moreover, given the broad coverage of the tool, it is applicable in other parts of the world, especially the developing countries. This VPA tool can help urban planners and governmen<sup>t</sup> stakeholders to better understand the prevalence of visual pollution, assess its spatial spread, identify primary contributing VPOs per location, take any corrective actions, and (most importantly) inform policy decisions in a robust, quantifiable and evidence-driven manner.

Further to the national and global application of this tool, an interesting future research dimension will be the determination of visual pollution threshold defined by means of public opinion and

statistical method. Although the tool has been primarily designed, tested and being implemented in different geographical contexts and it has wide coverage of the VPOs (local, national, international), there is a possibility to upgrade/adapt it, especially for the more developed part of the world where the frame of visual pollution is different from developing countries. In addition to this, the transformation of a paper-based VPA tool into a mobile-based VPA tool is another important area of future work, which will (in itself) lead to further research avenues.

**Author Contributions:** Conceptualization, K.W., G.A.A., M.A.N.; methodology, K.W., G.A.A., M.A.N., M.Q.H., A.W., and R.N.; software, K.W. and M.Q.H.; validation, K.W., and M.Q.H.; formal analysis, K.W., and M.Q.H.; investigation, A.W., M.J.T.; resources, M.A.N., R.N.; data curation, K.W., and M.Q.H.; writing—original draft preparation, K.W., G.A.A., M.A.N., M.Q.H.; writing—review and editing, A.W., M.J.T., and R.N.; visualization, M.Q.H.; supervision, M.A.N. and R.N.; project administration, K.W. and M.A.N.; funding acquisition, M.A.N and R.N.

**Funding:** This research received no external funding.

**Acknowledgments:** We want to thank the anonymous reviewers whose comments and suggestions contributed a lot in the refinement of research methodology and write-up.

**Conflicts of Interest:** The authors declare no conflict of interest.
