https://doi.org/10.53656/str2024-6s-9-res

2024/6s, стр. 112 - 126

RESEARCH ASSESSMENT: SOCIETAL IMPACT RELATED TO SUSTAINABILITY

Резюме: The paper focuses on selected issues regarding the societal impact assessment of research activity by taking into account possible sustainability impacts. The main challenge of research evaluation approaches here is the identification of principles to guide the design and implementation of appropriate indicators system for assessing such societal impacts. It is argued that the development of such evaluation system should take into account the specific environment, circumstances, and stakeholders related to the field of research, that is often interdisciplinary. The study utilizes expert assessment approach to validate the significance of a set of guiding principles of research impact evaluation that reflects the advancement towards sustainability and social cohesion. Directions for future research in this complex area are also outlined.

Ключови думи: societal impact assessment; research evaluation principles; sustainabilityJEL classification: I23, O30, Q01

Prof. Dr. Matilda Alexandrova
University of National and World Economy

Abstract. The paper focuses on selected issues regarding the societal impact assessment of research activity by taking into account possible sustainability impacts. The main challenge of research evaluation approaches here is the identification of principles to guide the design and implementation of appropriate indicators system for assessing such societal impacts. It is argued that the development of such evaluation system should take into account the specific environment, circumstances, and stakeholders related to the field of research, that is often interdisciplinary. The study utilizes expert assessment approach to validate the significance of a set of guiding principles of research impact evaluation that reflects the advancement towards sustainability and social cohesion. Directions for future research in this complex area are also outlined.

Keywords: societal impact assessment; research evaluation principles; sustainability

JEL classification: I23, O30, Q01

Introduction

The scientific discussion in the last decades that focuses on research evaluation and sustainability encompasses several critical issues, particularly in relation to assessing research outputs, identifying appropriate indicators, and considering the potential societal impacts of research. These themes are increasingly relevant as stakeholders demand accountability and relevance from academic research in addressing various societal challenges. New conceptual and empirical studies put in their focus the integration of societal impacts of research with the achievement of sustainable development goals (Santoyo-Castelazo & Azapagic 2014; Bornmann 2012).

Research outputs traditionally focus on quantitative metrics such as publications, citations, and funding. However, there is a growing recognition that these metrics do not fully capture the societal relevance or impact of research. The complexity of societal challenges necessitates a broader understanding of what constitutes valuable research outputs. In this respect, there has been a paradigm shift from merely measuring academic output to evaluating the societal research impact. This process acknowledges that research should contribute to societal well-being, inform policy making, and foster community engagement. Researchers are increasingly expected to demonstrate how their work addresses real-world issues and contributes to sustainable development.

The evaluation of societal impact implicitly contains significant challenges, including diculties in attributing specific outcomes to particular research activities. The long time lag between research implementation and observable societal benefits complicates this process. Additionally, traditional evaluation frameworks often lack the flexibility needed to account for the diverse contexts in which research operates. All these concerns motivate the aim of this paper to contribute to the systematic review and discussion about evaluating the societal impacts of research in the context of striving to sustainability.

The research questions here are oriented to systematization of the principles, models, and methods for adequate assessment of specific aspects of research impact, namely, the societal ones, related to sustainability. In this line of reasoning, there is a pressing need for developing appropriate frameworks and indicators that go beyond traditional metrics to assess not only environmental and economic but also social dimensions of sustainability. Effective indicators should capture qualitative aspects such as community engagement, social equity, and cultural impacts alongside quantitative measures (Aucamp et al. 2011; Zimek & Baumgartner 2024).

Another crucial issues to be considered in this respect is the involvement of key stakeholders in the process of defining and selecting assessment tools and indicators. This collaborative approach ensures that these instruments reflect the values and priorities of those affected by the research, enhancing their relevance and applicability. Moreover, the complexity of sustainability issues often requires interdisciplinary interaction to develop comprehensive approach to research evaluation. By integrating insights from various fields, e.g. environmental and social sciences, researchers should create a more holistic assessment framework that captures the multifaceted nature of sustainability impacts.

When dealing with this matter one needs to take into account the fact that societal impact encompasses not only direct benefits but also unintended consequences of research activities. Researchers must consider how their work interacts with existing social structures and contributes to social equity and justice. For example, transdisciplinary approaches are increasingly recognized as vital for achieving meaningful societal impacts (Belcher et al. 2019; Luxa et al. 2019; Newiga et al. 2019). By involving non-academic stakeholders throughout the research process, these approaches facilitate knowledge co-production that is more likely to address realworld challenges effectively. Furthermore, researchers are encouraged to adopt reflexive practices that allow them to critically assess their roles and assumptions throughout the research process. This adaptability can enhance the likelihood that findings will be relevant to specific contexts and used by policymakers to improve societal conditions.

Theoretical and methodological background

Specialized literatures in the field strives to clarify how societal impact is understood and measured across various fields. Bornmann (2012) provides a comprehensive overview of the concept of societal impact in research, examining its definitions, significance, and methods for assessment. Societal impact is defined as the complex effect which research has on society beyond academic circles, including contributions to public policy, community well-being, economic development, and cultural enrichment. Emphasis is put on the idea that societal impact is a complex phenomenon that emerges from the interaction between research outputs and societal needs. It is argued that understanding and measuring societal impact is crucial for justifying public investment in research – since funding agencies increasingly demand accountability, researchers must demonstrate how their work contributes to societal goals, such as those outlined in the UN Sustainable Development Goals (SDGs).

Previous research of Donovan (2008) formulates a basic classification of the benefits of research in a wide context. Social benefits signify the contribution of research to the formation of the “social capital” – for example, inspiring new advances of social studies, justification of public policies, and enhanced public decision making. Cultural benefits augment the understanding of the communication and interaction between cultures and societies leading to enrichment of the “cultural capital”. Environmental benefits amplify the “natural capital” via the channels of diminishing pollution, developing a circular and green economy, and preserving the biodiversity. Economic benefits indicate various effects on the “economic capital” conveyed by improved labor force skills, increased productivity, and enhanced competitiveness (Donovan 2008).

Godin and Doré (2005) explore fields where societal impact can be evaluated as well as indicators that could be implemented to assess the research impact. The authors suggest results from empirical study based on interviews conducted with: (i) scholars from research units operating with public funds; (ii) representatives of social and economic organizations as actual or potential users of research results. A typology of societal impact is defined having eleven dimensions characterized by various indicators: science, technology, economy, culture, society, policy, organization, health, environment, symbols, and training. Due to the wide range of dimensions and sets of indicators this approach was considered as dicult for implementation in practice.

In later studies attempts are made to improve the understanding of the key dimensions of societal impact of research. The latter is traditionally measured by direct economic effects, e.g. returns provided by patents, licenses, and innovation start-ups. This approach to defining indicators leads to implementation of “weak proxies” that try to assess the societal benefits of research. Nowadays, the academic communities have commenced substantial endeavors to to assess the impact in a more accurate way (Muhonen et al. 2020). In this respect, complexities of measuring the social impacts of research is addressed by Spaapen & van Drooge (2011) highlighting the multifaceted nature of such impacts, which often arise from various causes. They note a lack of robust measurement tools for assessing social impact and propose a shift in the evaluation focus “from judgment to learning”. This approach emphasizes on the understanding of the interactions between the key actors (researchers and stakeholders) which can facilitate the clarification of relationship between research and its impacts.

The idea for “productive interactions” is put in the basis of an overall model for evaluating research impacts that includes: (i) direct interactions – face-to-face communications between researchers and stakeholders; (ii) indirect interactions – dissemination of materials or artefacts that facilitate an improved communication; (iii) financial interactions – reflecting the monetary contributions that support collaborative research efforts. The proposed framework aims to enhance the evaluation of social impacts by illuminating the pathways through which research influences societal change (Spaapen & van Drooge 2011).

Some contemporary authors note that research impact is inherently subjective, often perceived differently by various stakeholders depending on their contexts and interests (Reed & Rudman 2023). Three critical considerations are expected to enhance the likelihood that research outcomes are beneficial. Firstly, “sensitivity to context” appears crucial for researchers as much as it involves recognizing local processes, cultural norms, and community dynamics that influence how research is received and utilized. By engaging stakeholders early in the research process, researchers can tailor their approaches to better align with community needs and expectations. Second, “representation of diverse voices” requires effective representation of affected groups which is vital for achieving sustainable and socially desirable outcomes. In particular, this may involve addressing barriers to engagement of relevant actors that originate from scarcity of resources and insucient communication. Third, “management of power dynamics” acknowledges power imbalances which hinder the fostering of equitable research environments. For example, knowledge creation and dissemination are influenced by existing power relations which can lead to marginalization of certain actors. By recognizing these dynamics, researchers can work towards creating more inclusive processes that elevate traditionally underrepresented voices (Reed & Rudman 2023). Academic entrepreneurship is identified as such an inclusive process as far as it involves actors from the research communities and targets various benefits defined as academic-business results, e.g. applied research publications, licenses, patents, operations of technology transfer oces, etc. (Sterev 2023).

Vutsova et al. (2023) pay attention to the multifaceted nature of evaluating academic research that induces the importance of a holistic approach. It transcends the traditional metrics, advocating for an assessment framework which incorporates diverse indicators of research quality and impact. The inclusion of qualitative assessments and the consideration of societal and economic impacts of research highlights the limitations of citation counts and journal impact factors as sole indicators of research excellence. This approach points out that the traditional metrics can incentivize quantity over quality and may overlook significant research that does not conform to mainstream trends. Some adverse effects related to research evaluation are discussed in Lambovska (2023) where a structured conceptual framework proposed to mitigate them. The “dark side” to research evaluation is characterized by negative consequences that can arise within the evaluation process. These adverse effects can manifest in various forms, impacting not only the individual researchers but also the institutions and the broader academic community. One such effect is the pressure on researchers to publish in high-impact journals, which can lead to a “publish or perish” culture, often at the expense of research quality and integrity. Additionally, the emphasis on quantitative metrics, such as citation counts, can overshadow the actual impact and relevance of the research to society. The proposed framework offers possible practical applications for university and government research evaluation systems oriented to improving academic governance (Lambovska 2023).

A comprehensive analysis of the current state of social impact measurement is provided by Feor et al. (2023) identifying common practices, challenges, and future research directions in the field. This review indicates various definitions of social impact, emphasizing that it encompasses all social and cultural consequences of actions that alter how individuals and communities live and interact. This broad understanding is asserted to be crucial for developing effective measurement frameworks. Several established models for measuring social impact are discussed, e.g. “Social Return on Investment” (SROI) that quantifies social value related to the investment; “Impact Reporting and Investment Standards” (IRIS+) which provides a standardized framework for measuring social, environmental, and financial performance; “Global Reporting Initiative” (GRI) that offers guidelines for sustainability reporting; “Social Accounting and Audit” (SAA) striving to evaluate the organizational social performance through stakeholder engagement. The analysis identifies several challenges in measuring social impact, including the lack of consensus on indicators and measurement methods; diculties in attributing specific impacts to particular interventions (due to complex causal relationships); data availability issues, particularly regarding longitudinal data that captures changes over time.

The impact of interdisciplinary sustainability research is discussed by Rau et al. (2018) exploring various challenges and opportunities. The authors emphasize that despite the growing importance of these issues, sustainability research often struggles to achieve visibility and recognition in both scientific and societal contexts. Sustainability research inherently requires collaboration across various disciplines, however, the integration of knowledge from environmental, economic, and social sciences is often insuciently realized in practice. Interdisciplinary sustainability research frequently remains “invisible”, meaning its contributions are not adequately acknowledged or utilized by policymakers and practitioners. This invisibility can lead to missed opportunities for translating research findings into actionable policies or practices. Furthermore, understanding the specific context in which research operates is crucial. Recognizing local conditions, stakeholder needs, and existing power dynamics can enhance the relevance and applicability of research outcomes. Contextual awareness allows researchers to tailor their work to better meet societal needs (Rau et al. 2018). Particular case in this respect is the boost of R&D in Artificial Intelligence for which specifically focused EU policies towards the development of “Common European Data Spaces” (e.g. Digital Europe Programme) aim in the digital transformation of Europe which directly induces societal impacts of contemporary ICT research (Molhova & Biolcheva 2023).

A more pragmatic strand in the literature on research impact assessment in the context of sustainability often focus on the evolving landscape of sustainability indicators and their role in assessing progress toward sustainable development. Laedre et al. (2015) focus on identifying and establishing effective indicators for sustainability impact assessments. It is argued that relevant indicators are crucial for evaluating the environmental, social, and economic impacts of projects and policies aimed at promoting sustainability. They should provide quantifiable data that helps stakeholders understand the effectiveness of sustainability initiatives and inform decision making processes. Specific criteria for selecting indicators are outlined, including relevance, measurability, clarity, and ability to reflect changes over time. Particular emphasis is put on engaging the stakeholders in the indicator development process. Involving diverse groups ensures that the selected indicators reflect the needs and priorities of those affected by sustainability initiatives, enhancing their legitimacy and acceptance.

In the same direction of study, Ramos (2019) identifies key challenges and opportunities associated with current indicator frameworks, proposing new paradigm to enhance effectiveness. It is argued that existing sustainability indicators often face issues related to standardization, context specificity, and integration across different spatial dimensions (international, national, or local). A more integrated and holistic perspective in developing sustainability indicators is advocated which involves developing sets of indicators that encompass multiple areas (environmental, economic, social, cultural, etc.) allowing for a comprehensive assessment of sustainability. Moreover, non-traditional aspects of sustainability, such as ethics, culture, and community engagement, are proposed as far as such intangible dimensions are often overlooked but are critical for understanding the broader implications of sustainability (Ramos 2019).

Some studies present assessment frameworks aimed at enhancing the societal relevance and effectiveness of research processes in sustainability transformations. Daedlow et al. (2016) address the need for research to not only produce knowledge but also actively engage with societal challenges to drive meaningful change. Socially responsible research is defined as a process that integrates ethical considerations, stakeholder engagement, and reflexivity into the research design – an approach ensuring that research outcomes are aligned with societal needs and contribute positively to sustainability goals. An integrated assessment framework is introduced combining the dimensions of sustainability into a cohesive evaluation process. This framework aims to facilitate an understanding of the research impacts on sustainability transformations. Specific evaluation metrics are proposed for assessing the effectiveness of socially responsible research processes. These metrics focus on reflecting not only scientific outputs but also societal impacts, stakeholder satisfaction, and the extent of knowledge co-production (Daedlow et al. 2016).

Acomprehensive review of existing social impact assessment models is provided by Corvo et al. (2021) by mapping of their strengths, weaknesses, and potential areas for future research. The authors analyze various methodologies and frameworks used in social impact assessment to propose a cohesive agenda that enhances the understanding and application of social impact measurements. The review reveals a wide array of such models based on quantitative, qualitative, and mixed-method approaches. Each model has unique features that cater to different contexts and objectives, highlighting the need for flexibility in choosing appropriate assessment tools. In the same time, limitations are outlined including challenges related to data availability, measurement consistency, and the complexity of attributing social changes directly to specific interventions. An appeal for greater integration of social impact assessment with other fields, e.g. environmental and economic evaluation, is proclaimed as far as such an interdisciplinary approach leads to more inclusive assessments that consider multiple dimensions of sustainability (Corvo et al. 2021).

Based on the specialized literature on evaluating social impact of research, the current study suggests a review that summarizes the major aspects of selected frameworks, their characteristics and potential implications (Table 1).

Major principles for the assessment of societal impacts of research

The assessment of societal impacts of research is increasingly recognized as a critical component of evaluating research effectiveness and relevance. Several major principles guide this assessment process, focusing on the multifaceted relationship between research outputs and societal benefits. Vanclay (2003) identifies some key concepts of social impact assessment. Core values are defined as “fundamental, ideal-typical, enduring, statements of belief that are strongly held and accepted as

premises”. Guidelines are “statements by which to plan a specific course of action and which clarify how it should be done”. Principles are identified as “general statements of either a common understanding or an indication as to a course of action about what ought to be done”. Several key principles derived from recent literature are presented in Table 2 (ICGPSIA 1994; Vanclay 2003; Hansson & Polk 2018; Lauronen 2020; Kny et al. 2023).

The assessment of societal impacts of research is guided by principles that emphasize inclusivity, contextual understanding, theoretical rigor, dynamic evaluation, and practical relevance. By adhering to these principles, researchers can enhance the effectiveness of their work in addressing societal challenges while ensuring that their contributions are meaningful and impactful. As the landscape of research continues to evolve, these principles are expected to play a key role in shaping future evaluations that prioritize societal benefits alongside traditional academic metrics.

Method and results from a survey of research expert opinion

In order to contribute to the current discussion about societal impact assessment of research in the context of sustainability, this paper presents selected results from expert opinion survey conducted in January-May 2024. It focused on the identification of areas of agreement among the experts about the importance of different principles identified in the literature. 27 national experts representing the scientific areas in Bulgaria (according to the National Classification of Scientific Research: Natural Sciences, Medical Sciences, Earth Sciences, Technical Sciences, Social Sciences and Humanities, etc.) have been selected purposively to participate in the survey under the condition of possessing evaluation experience and minimum of 10 years of research practice. The survey was conducted by inviting the experts to fill an online questionnaire covering a variety of state-of-art issues of higher education and research activities.

Among other issues, respondents have been asked – on the basis of their expertise in fundamental or applied research – to evaluate the importance of each one within a set of guided principles for assessing societal impacts of research. A 5-point Likert scale was adopted to capture the assessment grade indicated by each respondent, where 1 stands for “No important at all” and 5 for “Highly important”, respectively. Table 3 contains the summarized results presented by the average scores acquired by each suggested principle (in descending order).

The highest ranks on the adopted evaluation scale have been received by “Inclusivity and stakeholder engagement” principle (a score of 4.67 on average). The next ordered by the same criterion (average score) are “Utilization-focused evaluation” (4.63) and “Realist evaluation” (4.37). Apparently, the pool of respondents identified the three most important principles, according to their expert opinion, which must be taken into account when indicators of societal impacts of research are defined. Two other principles – namely, “Dynamic assessment over time” and “Contextual relevance” – are lower ranked but the divergence from the top three principles is not substantial. These results show that the pool of experts evaluate core indicators with practical role, clarity, and direct focus on perceived societal effects. Evaluation of research impacts on society has to take into account the interests of various stakeholders, to provide realistic assessment, and time resilient. It is yet important that indicators should implement the principle of contextual relevance which can additionally contribute for a high degree of measurement reliability.

Conclusion

Utilizing the review of theoretical and conceptual consideration as well as recent survey results, the current study accentuates on the need for adoption and implementation of principles, models, and relevant indicators that can achieve an adequate reflection of societal impact of research. The assessment of the latter is essential for the advancement of an effective system for evaluating research activities taking into account the dimensions of sustainability and social cohesion. By improving assessment methodologies and fostering stakeholder engagement, an evaluation system can induce sustainability relevance of researchers’ operations that effectively contribute to the societal well-being.

The review identifies several avenues for advancing research evaluation related to sustainability, e.g. developing standardized frameworks for assessing societal impact; encouraging interdisciplinary collaboration to enhance indicator development; fostering a culture of accountability within academic institutions regarding societal impacts; promoting capacity building initiatives that empower researchers and stakeholders alike. However, further empirical studies are needed to enhance the understanding of how societal impact emerges from research activities. In order to effectively assess societal impacts, researchers are encouraged to: engage with stakeholders at an earlier stage of the research process to align objectives; adopt mixed-method approaches that allow capturing both quantitative and qualitative impacts; reflect on their roles in facilitating societal change, considering the ethical implications (Bornmann 2013).

Addressing the major issues related to research evaluation and sustainability requires a fundamental rethinking of how we define success in academic research. By prioritizing societal impact, developing appropriate indicators, and engaging stakeholders throughout the research process, researchers can enhance the relevance and effectiveness of their work in contributing to sustainable development goals. Such a shift can not only benefits society but also to enrich the academic community by fostering innovation and collaboration across disciplines. By establishing a clear framework for indicator selection and emphasizing stakeholder engagement, an evaluation system can improve the relevance and effectiveness of societal impact assessment of research in order to drive meaningful change towards sustainability practices.

Acknowledgements and Funding

This work was financially supported by the UNWE Research Programme (Research Grant № 25/2023).

REFERENCES

AUCAMP, I.C.; WOODBORNE, S.; PEROLD, J.J.; BRON, A. & AUCAMP, S.M., 2011. Looking beyond impact assessment to social sustainability. In: VANCLAY, F., & ESTEVES, A. M. (Eds). New Directions in Social Impact Assessment. Edward Elgar Publishing.

BELCHER, B. M.; CLAUS, R.; DAVEL, R. & RAMIREZ, L. F. 2019. Linking transdisciplinary research characteristics and quality to effectiveness: A comparative analysis of five research-for-development projects. Environmental Science & Policy, vol. 101, pp. 192 – 203.

BORNMANN, L., 2013. What Is Societal Impact of Research and How Can It Be Assessed? A Literature Survey. Journal of the American Society for Information Science and Technology, vol. 64, no. 2, pp. 217 – 233.

BUCHMAYR, A.; VERHOFSTADT E.; VAN OOTEGEM, L.; THOMASSEN, G.; TAELMAN, S.E. & DEWULF, A., 2022. Exploring the global and local social sustainability of wind energy technologies: An application of a social impact assessment framework. Applied Energy, vol. 312, https://doi.org/10.1016/j.apenergy.2022.118808.

CORVO, L.; PASTORE, L.; MANTI, A. & IANNACI, D., 2021. Mapping Social Impact Assessment Models: A Literature Overview for a Future Research Agenda. Sustainability, vol. 13, no. 9, pp. 1 – 16.

DAEDLOW, K., PODHORA, A., WINKELMANN, M., KOPFMÜLLER, J., WALZ, R., & HELMING, K., 2016. Socially responsible research processes for sustainability transformation: an integrated assessment framework. Current Opinion in Environmental Sustainability, vol. 23, pp. 1 – 11.

DONOVAN, C., 2008. The Australian Research Quality Framework: A live experiment in capturing the social, economic, environmental, and cultural returns of publicly funded research. New Directions for Evaluation, vol.118, pp. 47 – 60.

EU, 2010. Assessing Europe’s university-based research. Expert group on assessment of university-based research. Brussels: Publications Oce of the European Union.

FEOR, L.; CLARKE, A. & DOUGHERTY, I., 2023. Social Impact Measurement: A Systematic Literature Review and Future Research Directions. World, vol. 4, pp. 816 – 837.

GODIN, B. & DORE, C., 2005. Measuring the Impacts of Science: Beyond the Economic Dimension. Montreal: INRS Urbanisation, Culture et Société.

HANSSON, S., & POLK, M., 2018. Assessing the impact of transdisciplinary research: The usefulness of relevance, credibility, and legitimacy for understanding the link between process and impact. Research Evaluation, vol. 27, no. 2, pp. 132 – 144.

Interorganizational Committee on Guidelines and Principles for Social Impact Assessment (ICGPSIA), 1994. Guidelines and principles for social impact assessment. Impact Assessment, vol. 12, no. 2, pp. 107 – 152.

KNY, J., CLAUS, R., HARRIS, J., & SCHÄFER, M., 2023. Assessing societal effects: Lessons from evaluation approaches in transdisciplinary research fields. GAIA – Ecological Perspectives for Science and Society, vol. 32, no. 1, pp. 178 – 185.

KURUVILLA, S.; MAYS, N.; PLEASVANT, A. & WALT, G., 2006. Describing the impact of health research: AResearch Impact Framework. BMC Health Services Research, vol.6, Article 134.

LAEDRE, O.; HAAVALDSEN, T.; ANDRÉ BOHNE, R. KALLAOS, J. & LOHNE, J., 2015. Determining sustainability impact assessment indicators. Impact Assessment and Project Appraisal, vol. 33, no. 2, pp. 98 – 107. DOI: 10.1080/14615517.2014.981037.

LAMBOVSKA, M., 2023. What is behind the shine? The dark side of research evaluation: a conceptual framework. TEM Journal, vol. 12, no. 4, pp. 2552 – 2563.

LAURONEN, J.-P., 2020. The dilemmas and uncertainties in assessing the societal impact of research. Science and Public Policy, vol. 47, no. 2, pp. 207 – 218.

LUX, A.; SCHÄFERB, M.; BERGMANN, M.; JAHNA, T.; MARGA, O.; NAGYB, E.; RANSIEKB, A. C. & THEILERA, L., 2019. Societal effects of transdisciplinary sustainability research – How can they be strengthened during the research process? Environmental Science and Policy, vol. 101, pp. 183 – 191. DOI:10.1016/j.envsci.2019.08.012.

MOLAS-GALLART, J.; SALTER, A.; PATEL, P.; SCOTT, A. & DURAN, X., 2002. Measuring Third Stream activities. Final Report to the Russell Group of Universities. SPRU Science and Technology Policy Research Unity, University of Sussex, Brighton, UK.

MOLHOVA, M., & BIOLCHEVA, P., 2023. Strategies and policies to support the development of AI technologies in Europe. Strategies for policy in science & education-Strategii na Obrazovatelnata i Nauchnata Politika, vol. 31, no. 3s, pp. 69 – 79. https://doi.org/10.53656/str20233s -5-str.

MUHONEN, R.; BENNEWORTH, P. & OLMOS-PENUELA, J., 2020. From Productive Interactions to Impact Pathways: Understanding the Key Dimensions in Developing SSH Research Societal Impact. Research Evaluation, vol. 29, pp. 34 – 47.

NEWIGA, J.; JAHNA, S.; LANGA, D. J. KAHLEA, J. & BERGMANN, M., 2019. Linking modes of research to their scientific and societal outcomes. Evidence from 81 sustainability-oriented research projects. Environmental Science and Policy, vol. 101, pp. 147 – 155.

RAMOS, T. B. 2019. Sustainability Assessment: Exploring the Frontiers and Paradigms of Indicator Approaches. Sustainability, vol. 11, no. 3:824. https://doi.org/10.3390/su11030824.

RAU, H, GOGGINS, G., & FAHY, F., 2018. From invisibility to impact: Recognising the scientific and societal relevance of interdisciplinary sustainability research. Research Policy, vol. 47, no. 1, pp. 266 – 276.

REED, M. S. & RUDMAN, H., 2023. Re-thinking research impact: voice, context and power at the interface of science, policy and practice. Sustainability Science, vol. 18, pp. 967 – 981.

SANTOYO-CASTELAZO, E., & AZAPAGIC, A., 2014. Sustainability assessment of energy systems: integrating environmental, economic and social aspects. Journal of Cleaner Production, vol. 80, pp. 119 – 138.

SPAAPEN, J., & VAN DROOGE, L., 2011. Introducing 'productive interactions' in social impact assessment. Research Evaluation, vol. 20, no. 3, pp. 211 – 218.

SPAAPEN, J., DIJSTELBLOEM, H., & WAMELINK, F., 2007. Evaluating research in context. A method for comprehensive assessment. 2nd Edition, The Hague: COS.

SPAAPEN, J., et al., 2011. Social Impact Assessment Methods for Research and Funding Instruments Through the Study of Productive Interactions (SIAMPI): Final report on social impacts of research.Amsterdam: Royal Netherlands Academy of Arts and Sciences.

STEREV, N., 2023. Pre-incubation toolkits for academic entrepreneurship fostering: Bulgarian case. Strategies for policy in science & educationStrategii na Obrazovatelnata i Nauchnata Politika, vol. 31, no. 3s, pp. 90 – 103. DOI: 10.53656/str2023-3s-7-pre.

VANCLAY, F., 2003. SIA principles. Impact Assessment Project Appraisal, vol. 21, pp. 5 – 11.

VUTSOVA, A.; YALAMOV, T. & ARABADZHIEVA, M., 2023. In search of excellent research assessment. Series: Wissenschaftsund Technikforschung, Neue Folge, no. 24. Baden-Baden: Nomos Verlagsgesellschaft mbH & Co. KG. ISBN 978-3-8487-7587-3.

WILLIAMS, S. & ROBINSON, J., 2020. Measuring sustainability: An evaluation framework for sustainability transition experiments. Environmental Science and Policy, vol. 103, pp. 58 – 66.

WOOD, T. & WILNER, A., 2024. Research impact assessment: Developing and applying a viable model for the social sciences. Research Evaluation, rvae022. https://doi.org/10.1093/reseval/rvae022.

ZIMEK, М., & BAUMGARTNER, R. J., 2024. Systemic sustainability assessment: Analyzing environmental and social impacts of actions on sustainable development. Cleaner Production Letters, vol. 7, Article 100064. DOI: 10.1016/j.clpl.2024.100064.

 Prof. Matilda Alexandrova
ORCID iD: 0000-0001-7235-2433
Department of Management
University of National and World Economy
19, December 8th St.

1700 Sofia, Bulgaria
E-mail: matildaa@unwe.bg