The art of evaluation lies in ensuring that the measurable does not drive out the immeasurable.
UK Audit Commission
Evaluation and review are vital components of performance improvement. They help you understand how you are performing, whether you are reaching your goals, and they inform your decisions about what to do next.
ACIG has almost twenty years experience helping its clients from all sectors measure and assess all aspects of performance – from the process level, to the business unit level, to organisation-wide.
We deliver practical, results-focused evaluations and reviews.
- Organisational, Program and Project Evaluation
- Best Value and Best Practice Reviews
- Evaluation Frameworks
- Benchmarking and Measurement
Evaluation and review should not occur in isolation – sustainable improvement usually requires an integrated approach of planning, implementation and review. Some clients implement the whole approach, others need only certain aspects – we can tailor our assistance accordingly.
Some of our recent evaluation and review assignments are listed below.
Evaluations in Government
The Australian Government Department of Broadband, Communications and the Digital Economy engaged ACIG to evaluate the Australian Music Radio Airplay Project (AMRAP). The AMRAP is a community radio initiative that works with musicians and community broadcasters to increase radio airplay for contemporary Australian music through the community radio sector.
ACIG is working with the Australian Government Department of Regional Australia, Regional Development and Local Government undertake a comprehensive review of the Norfolk Island public service. Our review recommends actions that the Norfolk Island Government and the Australian Government can take, if required, over the short, medium and long term to:
- More efficiently and effectively provide an appropriate and sustainable range of government services to the highest ethical standards;
- Improve the capacity and the skills of Norfolk Island Adminstration to deliver government services;
- Deliver government services to a similar standard as those provided by a local or state level government in mainland Australia;
- Improve resilience, change management and innovation across the public sector; and
- Improve performance and accountability mechanisms.
The NT Department of Health has engaged ACIG to review the organisation’s perfomance management systems and processes. We were subsequently engaged to undertake an audit of the Department’s orientation and workplace induction activities.
For the Victorian Department of Business and Innovation ACIG, leading a team including Roy Morgan Research, delivered an evaluation of the Industry Capability Network (ICN) industry support program. ICN Victoria assists its clients to maximise Victorian, Australian and New Zealand content in their procurement activities under the Government’s VIPP policy and matches local suppliers to government buyers acting on behalf of significant Victorian projects. After ACIG developed the Program Logic Model and Evaluation Framework, more than 250 clients and suppliers were surveyed, and thirty-three stakeholders interviewed in person. The resultant 200 page evaluation report was fully accepted in late 2010 and contained agreed recommendations for program improvement.
The Australian Government Department of Sustainability, Environment, Water, Population and Community recently engaged ACIG to conduct an evaluation of the of Caring for Country program. This complex program is a large initiative that funds a wide range of integrated projects that will deliver “an environment that is healthy, better protected, well-managed, resilient and provides essential ecosystem services in a changing environment.”
The Victorian Department of Human Services wanted to develop best business practices within each division and region and thereby move DHS as a whole towards service excellence. The strategy was to initiate a sub-program of improvement activity within each division and region. ACIG was engaged by DHS to assist in the implementation of DHS’s Service Excellence Framework, an organisational evaluation framework based on the Australian Business Excellence Model. ACIG assisted each division and region to undertake evaluation against the SEF framework and advised on coordination of the various sub-programs so that the overall program objectives were achieved.
For Sustainability Victoria ACIG reviewed processes and procedures throughout the organisation, with a view to improving performance and productivity.
ACIG completed an evaluation of the Australian Red Cross Blood Service (ARCBS) governance arrangements. Our analysis included the examination of relevant documents, policies and processes as well as face to face interviews with key stakeholders. ACIG was then able to make comprehensive recommendations about governance improvements.
ACIG has also assisted the Asian Development Bank develop its own Project Performance Monitoring System – a monitoring and evaluation framework for projects in developing countries. ACIG carried out several projects, including:
- Development of indicators for measuring longer term impact, short term objectives, and outputs of their loan projects which involve multiple sectors such as education, water, environment, energy and social development.
- Development of a process for managing and monitoring projects to achieve greater focus on development impacts and outcomes.
- A comprehensive training course to teach ADB staff about: defining a service/program /project, using the monitoring framework to identify inputs, outputs, outcomes and impacts, developing performance indicator definitions, sourcing performance data, reporting performance data and considering alternative solutions.
- An e-learning toolkit. This project included the design, development, field testing and operationalisation of an interactive CD based learning toolkit for the deployment of Continuous Improvement and Benchmarking in governments of developing member countries. A key module was on evaluation techniques, including the design, development and use of key performance measures that focus on service outcomes and outputs.
Building Evaluation Capacity
From time to time ACIG runs evaluation workshops and seminars. Contact us to discuss your training needs.
ACIG is currently working with the State Library of Victoria to develop and pilot a best practices framework against which public libraries will evaluate themselves, to determine their level of compliance with the best practices and identify areas for improvement. We helped SLV develop a comprehensive best practice framework based on Australian and international quality and business excellence frameworks, international library best practices and the SLV’s research project Libraries Building Communities. We are now implementing evaluations against the framework, called Being the Best We Can, in Victorian public libraries. The participating libraries are a mixture of metropolitan and country, single-municipality and regional corporation services. Being the Best We Can won the 2011 LGPro Award for Innovative Management.
ACIG worked with the Victorian Department of Justice to assess the extent to which the regulators within the Justice portfolio comply with the regulator practices contained within the Better Business Regulation (BBR) Best Practice Framework. ACIG implemented a series of self-evaluation reviews in each of nine DoJ regulators, testing and modifying the self-evaluation process and the framework itself through continual assessment of the BBR process.
ACIG helped the regulators to evaluate their regulatory processes from end to end, starting with policy development, strategy selection, regulatory scheme design and implementation planning, through to full-scale operation of the regulatory scheme, and its review and improvement. One of the most important outcomes was a strategic improvement action plan that each regulator was able to use to improve their regulatory performance and better manage their legislative obligations.
ACIG was subsequently engaged by a portfolio agency within the Department of Justice – the Office of the Emergency Services Commissioner – to develop and evaluation framework and plan for thier project Climate Change and the Emergency Management Sector: Building Research Capacity.
Department of Primary Industries Victoria also engaged ACIG to undertake Better Business Regulation reviews with their own regulators, similar to the DoJ reviews. Once again, the reviews covered end to end processes of regulatory development, operation, review and coordination.
The Department of Primary Industries subsequently engaged ACIG to conduct a high level assessment of its internal evaluation cacpacity.
For the Victorian Department of Sustainability and Environment ACIG evaluated the current state of knowledge management practices, systems and culture within the organisation. We then articulated a ‘theory of action’ that describes a pragmatic way forward for achieving DSE’s knowledge management aims.
ACIG assisted the Government of Tajikistan to develop the capacity for monitoring and evaluation of national development programs and projects. Over two years we advised on institutional development and capacity building as well as system design and implementation for the monitoring system for the poverty reduction strategy. ACIG also developed and delivered training courses for monitoring and evaluation, results-based management, data management and statistical analysis.
ACIG assisted the Government of the People’s Republic of China develop a Result-Based National Monitoring and Evaluation System for Key Projects. We worked in 2005-2007 with the Key Projects Inspectorate Office of the National Development and Reform Commission to:
- Review the monitoring and evaluation system under development, and assess the institutional framework and capacities of the key project executing agencies, project implementation enterprises, and key projects inspectorate offices at the national and provincial levels.
- Identify a set of simple, measurable outcome indicators for key sectors, in consultation with key stakeholders.
- Work with staff members of the project executing agencies, project implementation enterprises, and responsible provincial and central Key Projects Inspectorate Offices to study the selected pilot projects and determine measurements for the key indicators.
- Prepare and deliver training to explain the concepts and principles of result-based project monitoring and evaluation, methodologies for devising result indicators, and use of such indicators and the monitoring and evaluation system for policy and decision making. Use the pilot case studies to supplement the training package.
Local Government Reviews
LGPro is the peak industry association for local government professionals in Victoria. ACIG was engaged by LGPro to conduct benchmarking with thirty-one Victorian councils across eight services to identify improvements and to meet their obligations under the Best Value requirements. ACIG helped participants to develop service analyses, map processes, define practical performance indicators, benchmark performance, identify best practices and develop specific improvement action plans to help councils meet their Best Value obligations.
Banyule City Council – Review of engineering services.
Bayside City Council – Review of performance management framework.
Bayside City Council – Review of options fro delivering golf facilities.
Bayside City Council – Benchmarking and efficiency review.
Cardinia Shire Council – HR reporting review.
City of Boroondara – Best value review of waste management services.
City of Boroondara – Review of Building and Planning Services Unit.
City of Boroondara – Review of City’s approach to best value and continuous improvement.
City of Boroondara – Cost analysis and benchmarking review of corporate services.
City of Brimbank – Review of the Arts and Culture Unit.
City of Greater Geelong – Planning process review.
City of Greater Geelong – Best value review training.
City of Greater Geelong – Public broadcasters review.
City of Maribyrnong – Best value review training.
City of Melbourne – Review of business planning rating scheme.
City of Melbourne – Corporate learning and development KPI review.
City of Melbourne – Contract management review.
City of Melbourne – Mapping and review of Council resolutions.
City of Melbourne – Mailroom review.
City of Melbourne – Review of Melbourne Certification Group.
City of Melbourne – Parking and traffic benchmarking and process review.
City of Melbourne – Review of metered parking business processes and data accuracy.
City of Melbourne – Review of Permits and Approvals Branch.
City of Melbourne – Review of printing and copying services.
City of Monash – Contract procedures review.
City of Monash – PABX requirements review.
City of Monash – Business planning review.
City of Ryde – Best value review: Parks
City of Ryde – Best value review: Urban Planning
City of Ryde – Best value review: Human Resources
City of Ryde – Best value review: Environmental Health & Building Services
Hobsons Bay City Council – various best value reviews.
Kingston City Council – Review of building management services.
Kingston City Council – Review of Council business processes.
Nillumbik Shire Council – Review of Environmental Building Services
Parramatta City Council – Centralising services review.
Port Macquarie-Hastings Shire Council – Strategic plan review
The above list is indicative of the kind of evaluation and review work we’ve done. A full list of our clients is available here.