INDICATORS FOR DETERMINING THE EFFECTIVE LEVEL OF DIGITALIZATION IN HIGHER EDUCATION *

. With the rapid advent of the new information and communication technologies, including fast development of the Artificial Intelligence, the digitalization process could be considered as a basic key element of the transition process, securing the sustainability of an organization. This is also relevant for the Higher Education Institutions that have very significant and at the same time responsible role to prepare well the next generation for the challenges of the labor market. Therefore, the effectiveness of the digitalization at the universities is critical for two major aspects: to secure proper education based on the cutting-edge technologies and to provide good quality of all accompanying administrative services. This article presents the potential indicators determining the effective level of digitalization in higher education institutions. These indicators are developed based on the empirical study among 360 universities students in Master and Bachelor degrees in leading Bulgarian universities. Our findings indicate that students generally positively evaluate the efforts of universities to digitize the educational process. At the same time, some aspects of the educational process - such as the assessment in a digital environment, or the provision of administrative services - such as the application for dorm accommodation, which should be improved, are outlined. However, the need to make constant efforts to protect personal data and provide quality educational digital content is considered. We propose 9 complex indicators for assessment the effectiveness of digitalization in university.


Introduction
One of the most important objectives in front of every Higher Education Institution (HEI) is to provide quality education in order to prepare well next talented young people for the challenges of the labor market.At the same time, this educational service must be consistent on the one hand with the requirements of the business, and on the other -with the educational standards and the maintenance of high-quality education.In the era of information and communication technologies (ICT) and the extremely strong development of Artificial Intelligence (AI), it is impossible to provide quality education if a high level of digitization is not sufficiently ensured.The process of digitalization in HEIs has to be considered as multifactorial and complex and respectively the determination of indicators for its effectiveness is the main object of the current study.
From the very first universities to our days, it can be said that they are not just institutions of education where some people try to transfer their knowledge and skills to other people.Universities create an entire environment, worldview, form critical thinking, encourage their students to discover and develop their potential and opportunities.In this sense, every single university can be seen precisely as an opportunity for young people -to prove themselves that they know, that they can, and that they are ready for the challenges of tomorrowequipped with knowledge skills and abilities required by business organizations.
For any serious university nowadays, digitization and the digitization process are a key to guaranteeing a quality education recognized by academy community, young people and businesses.Of course, digitization can be found in everything -from a quality university website that is easy to navigate, search for information and provide access to any educational platforms, through online application, inclusion in certain courses and access to digital educational resources to, for example, submitting documents for student accommodation.
The key understanding of digitalization, shared by the authors of the current article, should be sought not in the exact reproduction of the traditional process in a digital environment and their implementation with digital tools, which simply copy the currently existing traditional process, but in the way to improve its efficiency -less time, less resources, more options, etc.Therefore, just having a digitized process is not enough prerequisite to ensure efficiency of organization -it is necessary to consider these processes in their complexity and systems and in the opportunities for improvement.The present article defines namely the indicators for securing effectiveness of the digitalization in HEIs.The proposed indicators are identified on the base of the empirical survey among 360 students in leading Bulgarian universities.

Theoretical background
Digitization, digital services and their applicability in all aspects of our modern life are in the top focus of the scientific discussion and interest for the last few years.This interest is logical and easily explicable due to the fact that digitization and the provision of digital services can bring the development of any organization to a completely new level.The business organizations are the most adaptable and flexible to the rapidly changing demands of the external environment, but the public authorities and organizations from other social systems have also changed following the digital trends, albeit at a slower pace.
The interaction between universities and industries in terms of digital transformation and digitization is one of the most vivid scientific discussions at the moment.For some researchers this collaboration is considered as a driver needed both for industries and universities for achieving sustainable transition to the new economic development. Evans, Miklosik and Du (2023) consider that the advantages of the partnership between businesses and universities encompass gaining valuable resources, validating work, facilitating learning and teaching opportunities, financial gains, enhancing reputation, and advancing career prospects.Albats, Alexander and Cunningham claim that intermediaries in relations universities and industries are shifting from physical to virtual and digital formats.Some interesting researches in the relation line between industry and universities but in more general context are explored by different scholars: Meissner et al. (2022) focus on the role of the labs, created as a result of partnership between industry and universities; Roncancio-Marin et al. (2022) consider the social impact of this collaboration; Alpkan and Gemici (2023) examine the synergy effect from this partnership for the product innovations; Zhang et al. (2021) explore the role of this collaboration for the creation of new technology companies.Silva et al. (2021) made comprehensive research to identify the milestones of successful collaboration and propose a model for sustainable partnership.
Based on a comprehensive research and data analysis, Ma and Li (2022) revealed an interesting correlation between the level of digitalization and from the other siderelations between enterprises and universities.They concluded that the digitization process facilitates the transfer of knowledge from universities to businesses and identified indicators to assess digitalization in different dimensions.The scholars selected indicators covering 5 different aspects of digitalization which are: digital facilities, digital carriers, digital communications, digital applications, and digital finance (ibid).
The possibilities of digital learning, as well as its limitations, is also one of the main topics when it comes to digitization in education system, especially after COVID-19 pandemic.Liu et al. (2023) made an interesting research on the digital capability of university students, and concluded that for students, attributes such as innovative capability, the application of big data, and employability skills are crucial factors connecting digital capability with sustainable behavior.Similar research was made by Reinhold et al. (2021) whose major conclusion is that positive attitude of students towards e-learning is the key element for success in digital education.Different aspects of e-learning coming from different countries around the world and its impact on both on students and teaching staff is explored by Zaharia et al. (2022), Weerarathna et al. (2023), Liu (2023), Stoykov and Petrova (2023); Angelova (2020), Salahshouri et al. (2022), Manjeese (2022), Al-Okaily et al. (2020) etc. Tautz, Sprenger and Schwaninger (2021) focus their scientific interest on the digital tools that could improve digital learning such as Virtual reality, question tools, classroom response system, lecture recording etc., in order to understand their impact for the active learning of students.Their main finding is that the students are willing to use digital tools in learning and consider this as a form of diversity (something new), which enrich the lecture (ibid).
Liu, Lee and Huang (2023) explore the possibility how could be boosting creative confidence, motivation for learning, and the collaborative creative performance of university students in design thinking through the utilization of a digital visual collaborative environment.Elnadi and Gheith (2023) on the other side pay attention on the role of individual characteristics of students for their digital entrepreneurial skills and intention to do digital business.They found out that digital competence as well as passion and curiosity are key factors for digital business intention.In this line comes the research of Demsash, Emanu and Walle (2023) who explores the digital technology and its level of exploration from university students.Scholars made empirical research among university students for their digital skills and usage of technologies and outlined several factors for good level of digital exploration such as: previous computer experience, favorable attitudes, good internet access (ibid).Almost the same subject is explored by Alferaih (2022), who also study the intentions of the university students for doing digital entrepreneurship.Scholar admit that factors such as attitude, subjective norm, perceived feasibility, perceived desirability, propensity to act, digital entrepreneurial education, and innovativeness play a substantial role in influencing entrepreneurial intentions.Additionally, perceived behavioral control and self-efficacy significantly impact actual entrepreneurial behavior (ibid).Chan, Krishnamurthy and Sadreddin (2022) investigate the ways in which digital tools, particularly technologies encompassing social, mobile, analytics, and cloud (SMAC), enable both internal and external interactions within university incubators and various participants in the entrepreneurial innovation ecosystem.Some of researchers explore the specific digital services provided by universities.For instance, Litoussi et al. (2022) consider the practice of universities for providing digital certifications (such as official academic transcript, registration certificate, diploma) and propose a model for decentralization of digital certification using blockchain technology.Ahmad and Rafiq (2022) assess the level of digitalization readiness of the university libraries and secure the digital preservation of the available literature.The results of their research reveal that libraries demonstrated a high level of readiness only in terms of having sufficient funds for acquiring technological infrastructure, while lack of adequate policy and procedure for digital preservation, training opportunities and skilled human resources are pointed out as insufficient (ibid).Zeqiri et al (2023) admit that students' satisfaction nowadays is a complex magnitude, but on the other side, the quality e-services provided by university, including trough web-site and intranet pages of the universities are one of the factors that are directly connected and impact the students' satisfaction.Guerrero, Heaton and Urbano (2021) explore the real possibility of new level of development of universities through the Massive Open Online Courses (MOOCs) which they consider as a chance for university to regain competitive advantage in digital era and digital economy.The main benefit that MOOCs provided, identified by Ogunyemi, Quaicoe and Bauters (2022) is the huge potential for massive audience which is geographically dispersed and absolutely heterogenous in its basic characteristics.On the other side, Dang, Khanra and Kagzi (2022) study the limitation in front of the MOOCs and reveal the following major groupsusage barriers, value barriers (including poor content quality etc.), tradition barriers and image barriers (including accent of the lecturer etc.).Interesting researches for different aspects of MOOCs are developed by Wong, Baars, de Koning and Paas (2021), Aparicio et al. (2019), Weinhardt and Sitzmann (2019), Janelli and Lipnevich (2021) and others.
On the other hand, McLean, Maalsen and Lake (2022) pay attention to something very importanthow university can secure the digital sustainability in long terms period.They conducted a study at the University of Sydney and one of the conclusions they reached was that digital sustainability is not yet at the center of strategic sustainability planning efforts.
In summary, the process of digitization in universities represents a transformative journey that extends beyond mere technological integration.Embracing digitization opens avenues for enhanced learning experiences, improved collaboration, and increased access to information.The benefits extend to both students and educators, fostering a dynamic and interactive educational environment.Additionally, digitization enables universities to streamline administrative processes, optimize resource utilization, and stay abreast of evolving educational trends.

Methodology of the research
Our main objective of the current study is to identify workable indicators for determining the effective level of digitalization in the higher education institutions, considering the two major aspects in any university: the core educational process itself (with all levels and forms) and all accompanied servicesincluding different administrative services and securing access digital record and online educational platforms and resources.This study is a part of the entire scientific project title, financed by Bulgarian Science Fund.
In order to achieve the above-described objective, we implemented the followed methodological steps, presented in Table 1.Discussion and determination of the most thorough and effective way to achieve the research objective 2.
Preparation of questions for semi-structured interviews and task allocation between authors 3.
Identification of the major target groups and their concrete representatives for the interviews 4.
Collecting initial information through semi-structured interviews 5.
Developing an initial set of questions to be used in the questionnaire 6.
Pilot testing the quality of questions and their order in the questionnaire 7.
Finetuning of the questionnaire according to the result of the pilot test 8.
Distribution the questionnaire 9.
Analysis of the collected information 11.
Identification of workable indicators for the assessment of the effective level of digitalization in HEIs Our first step was to discuss and agree on the most appropriate and effective approach for the overall methodology of the research.In order to obtain initial information about the process of digitalization in university, we decided to conduct several semi-structured interviews (21 in total) with different representatives of our major target groups -academic staff, management body, administration and students.Although different groups can be presented in more comprehensively, for the purposes of our study we have identified the above four groups, with their main roles in the digitization process of a university shown in Table 2. Decision makers They lead the entire process of digitizing the university, including making decisions about the different systems to be used, ways to ensure connectivity and security.They are also the people who make decisions about the overall strategic development of digitization in the university.

Academic Staff
Users with different level for access and activities Regarding the different platforms used for distance learning, the academic staff should have more rights than the average user -including managing different groups, being able to create groups, assign tasks and monitor deadlines for their completion, etc.At the same time, in purely administrative aspect, academic staff has access to the systems for entering and signing grades -i.e. to student files etc.

Administration
Service and maintenance Administration representatives at each university are responsible for maintaining systems and providing administrative services, from different documents to applying for scholarships.At the same time, the administration works on a daily level with these systems, enters information and their opinion is very important in relation to the overall digitalization approach of the university.

Students
Users The most important target group, as universities are devoted to create communities of active, intelligent young people.In the sense of digitalization, they are the main users of university systems, especially in terms of access to platforms with educational materials and platforms on which distance online learning takes place.At the same time, however, students are also users of all accompanying administrative services, provided by universities.All this means that the final evaluation of the effectiveness of the level of digitization should be done precisely by this target group, since they are also the final beneficiaries.
The information collected as a result of these interviews we considered as very important for proper understanding of the overall digitalization of one university and not only the point of view of some of the stakeholders.
However, even at this early stage of the research, differences in the focus of our target groups emerged.For example, for the representatives of the management structures, the emphasis was placed on ensuring the security of data and personal information of students and staff, for the representatives of the administration themselves, the most important thing was the easy maintenance and service of the various systems, as well as their connectivity, so as not to be required repeatedly or several times entering the same information into different systems.
At the same time, academic staff emphasizes the preservation of the quality of the educational process and the need to keep the attention of students in a digital environment, including by using various digital new tools for this purpose.In turn, for students, one of the most important advantages is the possibility of easy access to educational resources and the possibility of a facilitated / distance educational process.
Based on the information, collected and analyzed from the semi-structured interviews, we developed an initial set of questions to be used in the questionnaire of our main research.Given the depth of the research, the initial version of the questionnaire was pilot-tested among students (as focus group), who were asked to say whether they understood the questions, whether there were questions with pre-formulated answers that did not fully exhaust the possible range of answers, whether they thought the questions themselves were correctly asked, lack of Ambiguity, etc.In general, the questions were relatively well developed, but it was necessary to reformulate three questions in order to achieve greater clarity.
After finetuning of the questionnaire, we distributed the information for survey among students in different universities.As a result of our efforts, we received 360 answers from university students both in Bachelor and Master degree.

Survey Results and Discussion
First section of the questionnaire collects information for the students' profile (Table 3).The profile of our responders indicates serious predominance or representatives coming from the public university (91,1%), while the responders of the private universities remain modest (8,9%).This situation is easily explicable considering the fact that there are 52 universities in Bulgaria in total and the majority of them are public.According to data from the National Statistical Institute, approximately 12% of the university students in Bulgaria are in private universities.On the other hand, the administrative fee of the private universities could be considered as some kind of barrier in front of all young people who desired to study.Therefore, the number of students in private universities is in general smaller than the number of students in public universities, which fact is visible in our collected data.In terms of form of education, the majority of our responders are in regular form (91,4%) and only 8,6% are students in Distance Learning form.This result does not really represent the correct proportion between students in the different forms of education.It is clear that our questionnaire was taken more seriously and more responsibly by the students in a regular form of education, while it did not arouse any interest at all in terms of students in distance learning.This can also be explained by the fact that distance learning students usually choose this because they work and study at the same time, which also explains their less commitment to a voluntary university study.
The vast majority of respondents are studying at a bachelor's degree, which is also understandable considering that a bachelor's degree is four years long, while a master's degree is between 1 and 2 years.
The second part of the questionnaire contains the questions on the assessment of the different aspect of digitalization in university, starting with those concerning education process.The next two questions are focused on the assessment of the learning process in digital environment.On the question "Did you like that the study was flexible -you could study from different places and at different times?" the accumulated results show that a large part of the responders rated positively the flexibility of the digital learning.A convincing 70,3% of responders declare their positive attitude to the flexibility which is one of the leading strengths of the digital learning process, versus only 25% whose assessment is negative.This question collects also almost 5% of hesitant students, who find some aspects of digital learning as positive and someas negative ones (Fig. 1)."Did the new type of learning lead to higher personalization (to have emphasis / attention on the needs and problems of each individual learner)" is the following question from the questionnaire.Here there is no such predominant preferred answer as was the previous question.However, 43,6% from responders admit that this type of learning meets individual needs of student and other 33,6% claim that these needs were met only sometimes.Here the share of responders with direct negative assessment is 22,8% which fact is indicative of the level of satisfaction or lack of satisfaction with regard to personalization achieved in digital learning (Fig. 2).The next question "Was remote assessment good and effective?"presents an interesting distribution of answers (Fig. 3).Here the major part of responders claims positive assessment.
The completely opposite opinion is shared by 12,2 % of the respondents, who strongly disagree that remote assessment is good and effective.Here we have two other answers, the first of which (preferred for 11,1% of the responders) contains negative assessment for the start of the period, but at the same timerecognition that the assessment was improved during the time.However, 18,9% claimed to be rather positive about the current state of assessment now, but at the same time they admitted that they expect further improvement and development.The next question "What is the degree of digitization of the process of application and admission of students at the university" collects information for the overall process and for specific typical services conducted for this purpose: Document application; Acceptance of application documents; Admission to exam; Conducting competition exam; Admission of students based on their grades; Enrollment of newly admitted students (Fig. 4).

Fig.4. Assessment of the application and admission process of students at the university
As it can be seen on the figure above, the students predominant assess the level of digitalization of the process for application and admission in universities as high, with slight differences between different operations.According to their opinion, the admission of students based on their grades is the operation with highest level of digitalization, while the conducting of competition exam for admission is the operation with lowest level of digitalization (but this assessment still remains positive for the serious majority of responders).The next question measures the level of digitalization of traditional administrative services for students.Here the results, especially compared to the results of the previous question, show that there are specific activities where the digitalization has to be drastically improved (Fig. 5).
Results indicate that service for Dorm Accommodation achieves the lowest assessment in terms of digitalization.This finding is clear evidence that students' satisfaction on the digitalization level of the procedure for dorm accommodation remains insufficient and has to be further developed.At the same time this procedure is the only one from four, where the number of responders choosing assessment "medium" is highest than the number of responders with answer "high" for the level of digitalization.Apparently, the digitalization of the Study Schedule in the majority of the Bulgarian universities achieved the highest level of recognition from the students.Next question measures the level of digitalization in different forms of educationregular (full-time), part-time and distance learning.The results are explicable, considering the general idea behind these three forms of education.According to our responders, the digitalization in distance form of learning accumulates the highest assessment, while the digitalization in part-time learning remains the lowest (Fig. 6).The results obtained for the next question reveal a clear picture for further improvements and development of the level of digitalization, developed so far by the universities.Interesting finding is the fact that students assess the level of digitalization during lectures higher than the level of digitalization during seminars.On the other hand, depending on the subject of the education itself, the lecture usually is more theoretically focused rather than the seminars, where it is expected to developed some skills and abilities more closely connected to the practice.In this regard sometimes, the digitalization could be difficult where the practical exercises are required (for engineering students, medicines students etc.).At the same time, the digitalization during the exams gains the highest level of assessment, comparing to the other activities.This is also logical, considering the fact that in almost all universities there are built test centers, used for this purpose (which support also the effort of the academic staff in assessment of students).

Fig. 7. Assessment the level of digitalization in different educational activities
One of the major problems when we explore the quality of educational process in digital environment is the content of educational material.The next two questions focus attention namely in this serious aspect of digitalization in HEIs.The question "Should educational institutions ensure/improve their online content and resources to be accessible to people with disabilities" shows the understanding of students on the accessibility of materials and resources for disable people (Fig. 8).The majority of responders claim that there is a clear necessity of improving the applicability and accessibility of digital materials and resources to be adapted for students with disabilities (78,6%).At the same time, another almost 20% (17,8%) felt that the materials available were of good quality, but could still be further improved and only 3,6% consider that the current quality of materials is good and therefore, it does not need to be upgraded.
The opinion of students on the quality of the educational materials and its adaptation is collected through the following question "Is it necessary to develop high-quality digital educational content that is brand new and aligned with curriculum standards?".The obtained results indicate unambiguously that university students in Bulgaria found the necessity of developing new content appropriate for digital learning and digital environment (Fig. 9).When we consider digitalization, the problem for protection of personal data remains one of the important topics that has to be payed special attention, considering the speed of development of ICT, including the possibility of emergence of new threats and sophistication of hacker attacks.The answers of the question "Is it important to increase measures to protect students' personal data and ensure the security of online platforms?"presents the understanding of students for further development and guaranteeing protection of the personal data (Fig. 10).As it can be seen from the figure, the majority of responders (60,6%) admit that additional steps for better security measures in digital environment could be taken.This result is in line with the responsible understanding No -it's fine now Yes -now it's not good Yes -additional steps may be taken for better security measures that cyber security must be seen as a continuous improvement process, parallel to the development and advancement of ICT technologies.At the same time 26,4% from our responders claim that the universities put enough effort into securing personal data and the current state of the systems is good enough.
Results, obtained through the question "Is it necessary to develop and improve hybrid learning models that combine both face-to-face and online elements?"clearly draw the expectation of university students (Fig. 11).The majority of them (almost 80%) find the future development namely in hybrid learning, combining the advantages of different forms to achieve high quality education in compliance with the new technologies and business requirement.Next question is the following: "Do student assessment methods need to be revised and refined to align with the positives and negatives of online learning?"and it is not surprising the fact, that the majority of responded students answer with "yes" (76,1%) versus only 23,9% whose answer is "no".Therefore, it would be useful to come up with a comprehensive toolkit related to ensuring a fairer assessment of knowledge in digital evaluation process.The final question collects information on the students` opinion for the further implementation of innovations in digital learning process.The majority of responders answered positively (almost 80% in total) to the question "Is there a need to focus on additional innovations in educational technology, including gamification, virtual reality, etc.?".For almost half of the responders (47,5%) the innovations in digital learning process are very important aspect of the education.At the same time 31,7% of responders consider that these new technologies are poorly implemented at the moment and have to be further integrated in the learning process.Only 20,8% think that current state of play is fine and there is no need to be improved (Fig. 12).
In summary of the presented results of the empirical research, it should be said that students generally positively evaluate the efforts of universities to digitize the educational process.At the same time, some aspects of the educational process -such as the assessment in a digital environment, or the provision of administrative services -such as the application for dorm accommodation, which should be improved, are outlined.However, the need to make constant efforts to protect personal data and provide quality educational digital content is considered.

Identification of workable indicators for the assessment of the effective level of digitalization in HEIs
Based on the results and analysis of the research as well as authors study of different processes and systems of the universities in terms of digitalization, it could be identifying the following indicators for the assessment of the effectiveness of digitalization in HEI:

Online learning platforms adoption
The extent to which the university has adopted online learning platforms like Learning Management Systems (LMS).Online Learning Platforms Adoption refers to the incorporation and utilization of digital platforms designed to facilitate teaching and learning activities in higher education institutions.Common examples of these platforms include Moodle, Canvas, Blackboard, Sakai, and others.These platforms serve as centralized hubs where it can be organized course materials, facilitate communication, and administer assessments.Higher adoption of online learning platforms can streamline administrative tasks for instructors, making it easier to manage course materials, communicate with students, and assess learning outcomes.At the same time online platforms offer students greater flexibility in accessing course materials and participating in learning activities, potentially accommodating diverse learning styles and schedules.By tracking usage metrics and engagement levels on the online platform, university staff and managers can make data-driven decisions to improve teaching practices, allocate resources effectively, and enhance the overall educational experience for students.
The usage metrics such as course enrollment, content creation, student engagement, and assessment activities can indicate the level of digital integration in the teaching and learning process.Course Enrollment: This metric measures the number of students enrolled in courses offered through the online learning platform.Higher enrollment numbers typically indicate greater adoption and utilization of digital platforms across the university's curriculum.Content Creation: It evaluates the extent to which instructors are utilizing the platform to create and distribute course materials such as lecture slides, readings, videos, quizzes, and assignments.Metrics might include the number of new content uploads, types of content shared, and frequency of updates.Student Engagement: This metric assesses the level of student interaction and participation within the online learning environment.It could include indicators such as discussion forum activity, participation in virtual classrooms, and submission of assignments.Assessment Activities: It evaluates how extensively assessments, such as quizzes, tests, projects, and assignments, are administered and graded through the online platform.Metrics might include the number of assessments conducted online, variety in assessment types, and timeliness of feedback provided to students.

Digital resources availability
Availability and accessibility of digital resources such as e-books, online journals, databases, and multimedia materials and other digital assets, which are accessible to students, academic staff, and researchers through the university's library and online repositories.
Metrics for assessment of this indicator could be the following: quantityby measuring the total number of digital resources available within the university's library and online repositories.It includes the total count of ebooks, online journals, databases, and multimedia materials accessible to users; Coverage of digital materials: It evaluates the breadth and depth of subject coverage across different disciplines and fields of study.The richness and diversity of these resources can reflect the institution's commitment to digitalization in supporting research and academic activities; Accessibility and usability: the ease of access and usability of digital resources, including user interface design, search functionality, navigation tools, and availability of user support services.
Here have to be added materials adopted for the needs of university students with disabilities; Statistics: Tracking usage statistics such as downloads, views, citations, and interactions with digital resources provides insights into the popularity and relevance of specific materials among users.

Available technology infrastructure
Available technology infrastructure refers to the physical and virtual resources, including hardware, software, networking capabilities, and support services, that are accessible to students, faculty, and staff within the university ecosystem to enable effective utilization of digital technologies.This indicator assesses the adequacy and accessibility of technological resources and support systems within a university to facilitate teaching, learning, research, and supportive administrative activities.It measures the quality and robustness of the technology infrastructure including network bandwidth, Wi-Fi coverage, computer labs, and availability of digital tools for students and faculty.A reliable and up-to-date infrastructure is essential for facilitating digital learning, research, and administrative tasks.
The indicator could be measured by the following metrics: Coverage and Accessibility: assessment the accessibility of technology infrastructure across different campus locations, including classrooms, laboratories, libraries, study areas, administrative offices, and remote learning environment; Reliability: it assesses the reliability, speed, and performance of technology infrastructure components such as network connections, hardware devices, and software applications to ensure uninterrupted access to digital resources and services; Scalability and Future Development: This metric examines the scalability and capacity of technology infrastructure to accommodate growing user demands, technological advancements, and emerging trends in digitalization, ensuring long-term sustainability and future readiness; User satisfaction and feedback: gathering feedback from students, both academic and administrative staff through surveys, focus groups, and feedback mechanisms helps in assessing user satisfaction levels, identifying areas for improvement, and addressing specific needs and preferences related to technology infrastructure.

Virtual collaboration tools usage
Utilization of virtual collaboration tools such as video conferencing platforms, online project management tools, and communication platforms.These tools enable remote collaboration among students and staff, promoting teamwork, communication, and productivity.Virtual collaboration could be implemented using

Integration of data analytics
This indicator assesses the extent to which a university utilizes data analytics techniques and tools to gather, analyze, and derive insights from various sources of data across academic, administrative, and operational domains.Adoption of data analytics tools and techniques for decision-making processes such as student performance analysis, course optimization, resource allocation, and strategic planning.Leveraging data analytics can enhance operational efficiency and effectiveness in various aspects of university management.
Integration of Data Analytics refers to the incorporation of data analytics methodologies, technologies, and practices into the university's decision-making processes, academic programs, research endeavors, student support services, and administrative operations to drive insights, improve performance, and enhance outcomes.This involves collecting, organizing, and managing diverse sources of data generated within the university ecosystem, including student information, academic records, research data, financial transactions, institutional operations data, and external data sources.This indicator includes Data analysis and modeling, Data Reporting, Predictive Analytics and Machine Learning.

Electronic Document Management
Implementation of electronic document management systems (EDMS) or digital repositories for storing, organizing, and managing administrative documents and records electronically.It assesses the extent to which a university has implemented digital systems and processes for the creation, storage, organization, retrieval, sharing, and management of electronic documents and records across various academic, administrative, and operational functions.EDM refers to the systematic management of electronic documents and records throughout their lifecycle, from creation or receipt to disposal or archiving, using digital technologies and tools to improve efficiency, accessibility, security, and compliance.This includes digitizing paper-based documents, enabling electronic signatures, and ensuring document version control and access permissions.
Metrics for assessment could be the following: adoption and utilization of electronic document management systems and tools across academic departments, administrative units, and operational functions within the university; Accessibility and usability: It assesses the ease of access, navigation, and usability of electronic document management systems for users, including students, administrative and academic staff, and external stakeholders, through user satisfaction surveys, feedback mechanisms, and usability testing; Efficiency and productivity -the impact of electronic document management on workflow efficiency, productivity gains, time savings, reduction in paper-based processes, and elimination of manual tasks such as printing, copying, filing, and physical document storage; Compliance and Risk management -the degree to which EDMs comply with relevant legal, regulatory, and institutional requirements, including data protection laws, records management policies, retention schedules, and audit trails; Cost Savings and ROI -cost-effectiveness and return on investment (ROI) of electronic document management initiatives, including savings in paper, printing, storage, and administrative overhead costs, as well as improvements in operational efficiency and productivity.

Digital marketing and recruitment efforts
The university's digital presence and activities in marketing, recruitment, and student engagement through social media, websites, online advertisements, and virtual events.This indicator assesses the effectiveness of a university's digital strategies and initiatives aimed at attracting, engaging, and converting prospective students and stakeholders through online channels and platforms.Digital Marketing and Recruitment Efforts refer to the strategic use of digital channels, technologies, and tactics to promote the university's brand, academic programs, campus culture, and unique value proposition to prospective students, parents, alumni, business, partners, and other stakeholders.
Components of the digital marketing could be: Online Presence and Brandingestablishing a strong and cohesive online presence through the university website, social media profiles, blog posts, videos, virtual tours, and other digital platforms to showcase the institution's brand identity, values, and offerings; Search Engine Optimizationto optimize the university's website and digital content for search engines, improving visibility, organic traffic, and search engine rankings for relevant keywords and phrases related to academic programs, campus facilities, and student life; Content Marketing and Social Mediacreating and sharing valuable, informative, and engaging content across social media platforms, blogs, podcasts, webinars, and other channels to attract and engage prospective students, parents, influencers, and other stakeholders; Email Marketing and Automation to nurture relationships with prospective students, provide personalized communication, deliver targeted messaging, and guide them through the admissions funnel from inquiry to enrollment using automation tools and drip campaigns; Paid Advertising -such as pay-per-click (PPC) ads, display ads, social media ads, and remarketing campaigns are deployed to reach specific audience segments, retarget website visitors, and drive traffic, leads, and conversions for key recruitment goals; Data Analytics and Performance Trackingto monitor and analyze the performance of digital marketing campaigns, track key performance indicators (KPIs), measure ROI, and optimize strategies based on data-driven insights and actionable analytics.Effective digital marketing strategies can help attract prospective students, engage current students, and strengthen the university's brand reputation.

Cybersecurity policy
Implementation of robust cybersecurity measures to safeguard digital assets, sensitive information, and personal data against cyber threats and breaches.This indicator assesses the effectiveness of a university's strategies, policies, and practices in protecting digital assets, information systems, and sensitive data from cybersecurity threats, vulnerabilities, and breaches.
Metrics of assessment under this indicator could be the following: Security posture and maturity -assessing the university's overall cybersecurity posture and maturity level based on cybersecurity frameworks such as NIST Cybersecurity Framework, ISO/IEC 27001, CIS Controls, and maturity models like Cybersecurity Capability Maturity Model (C2M2); Incident Response Time and Effectivenessmeasuring the average time to detect, respond to, contain, and recover from cybersecurity incidents, breaches, and data breaches, and evaluating the effectiveness of incident response procedures and mitigation efforts; Compliance and regulatory adherence: Ensuring compliance with relevant Bulgarian laws and European regulations, and industry standards such as GDPR and other, and assessing adherence to compliance requirements through audits and assessments; Security Controls: assessing the effectiveness of security controls, patches, updates, and vulnerability management processes in mitigating known vulnerabilities, patching critical vulnerabilities, and reducing the attack surface; User Awareness and Training: Measuring the effectiveness of cybersecurity awareness training programs, phishing simulations, security quizzes, and user feedback surveys in improving user awareness, behavior, and adherence to security policies and best practices.Regular security audits, compliance with data protection regulations, and cybersecurity awareness programs are crucial for maintaining a secure digital environment.

Feedback mechanisms and continuous improvement
Availability of feedback mechanisms such as online surveys, evaluation forms, and digital suggestion boxes to gather input from stakeholders including students, academic and administrative staff.It evaluates the effectiveness of mechanisms and processes implemented by a university to gather, analyze, and act upon feedback from stakeholders regarding various aspects of digitalization efforts.
Metrics for its evaluation could be the following: Feedback response ratesthe percentage of stakeholders who provide feedback in response to feedback requests or surveys.Higher response rates indicate greater engagement and participation in feedback mechanisms; Feedback satisfaction scores: the satisfaction levels admit by users in terms of accessibility, ease of use, responsiveness, transparency, and perceived impact on addressing their concerns and suggestions; Improvement Culture -the development and promotion of a culture of continuous improvement within the university, where feedback is valued, encouraged, and integrated into decision-making processes at all levels of the organization.Monitoring of feedback and subsequent improvements demonstrate a commitment to enhancing the digital experience and addressing emerging needs.

Conclusion and recommendations
Universities should by default be institutions where knowledge is preserved during centuries and shared by new generation of young people, eager for study.In this sense, higher education institutions are the guardians of traditions.At the same time, however, they are called to prepare the new generation for the new challenges of the economy and the demands of business.Which means that in addition to preserving traditions, they must also be open to the latest technologies.
The digitization of higher education and, in particular, education process at a university is an extremely complex process, which, among other things, should guarantee the provision of quality education.From this point of view, digitization in universities should not be seen only as the transfer of activities and services from a traditional to a digital environment.Here comes precisely the moment for the importance of evaluating the effectiveness of the level of digitization in universities.For this purpose, we have developed 9 complex indicators with their evaluation metrics, through which a clear idea can be given of the level of effectiveness of digitization in higher education institutions.
Based on an empirical survey among 360 students, we established their assessment of the current state of the system in Bulgarian higher education institutions, and after a thorough analysis of the collected information and additional research, we developed and presented the following indicators for evaluating effectiveness, which have to be assessed at the same time in order to analyze properly the achieved effective level of digitalization: Available online platform and their adoption in university activities; Availability and accessibility of digital resources; Available technology infrastructure; Usage of virtual collaboration tools; Integration of Data analytics; Electronic document management; Digital marketing and recruitment efforts; Cybersecurity policy and Feedback mechanisms and continuous improvement.

Fig. 1 .
Fig. 1.Assessment the flexibility of digital learning, in %

Fig. 5 .
Fig. 5. Assessment of level of digitalization of traditional administrative services for students

Fig. 6 .
Fig.6.Assessment the level of digitalization in different forms of education

Fig. 8 .
Fig.8.Assessment the quality and applicability of digital resources and materials for students with disabilities, in %

Fig. 9 .
Fig. 9. Assessment the necessity of the development of brand-new digital content of educational materials, in %

Fig. 10 .
Fig. 10.Assessment the level of personal data protection of university students in digital environment, in %

Fig. 11 .
Fig.11.The attitude of students to hybrid learning process (both in traditional and digital environment), in %

Fig. 12 .
Fig.12.Further adoption of innovations in education process, in %

Table 1 .
Methodology, used for the achieving the main objective of the research

Table 2 .
Major target groups, their role and characteristics in the digitalization process of a university

Table 3 .
Combined data for the responders' profile (in %) : Video Conferencing Platforms: Examples include Zoom, Microsoft Teams, Google Meet, and Cisco Webex.These platforms enable real-time audio and video conferencing, screen sharing, chat, and collaboration on documents and presentations; Collaborative Document Editing Tools: Platforms like Google Docs, Microsoft Office Online, and Dropbox Paper allow multiple users to collaborate on documents, spreadsheets, presentations, and other files simultaneously, facilitating collaborative editing and version control; Instant Messaging and Chat Applications: Tools such as Slack, Microsoft Teams, Discord, and WhatsApp provide instant messaging, group chat, file sharing, and integration with other productivity apps, enhancing communication and collaboration among team members; Project Management Software: Platforms like Trello, Asana, Basecamp, and Jira enable teams to plan, organize, track progress, assign tasks, and collaborate on projects using visual boards, task lists, timelines, and workflow automation feature; Virtual Whiteboarding Tools: Applications like Miro, MURAL, and Jamboard allow users to create and collaborate on digital whiteboards, diagrams, mind maps, and visual presentations, facilitating brainstorming, ideation, and collaborative problem-solving.