Explaining Housing Policy Change through Discursive Institutionalism

Abstract

Explaining how and why housing policies change is an ongoing theoretical challenge for housing scholars. A key approach is the ‘housing regimes’ framework (Kemeny 2006), drawing from Esping-Andersen’s work on the role of labour/capital struggles in shaping welfare states. However, this framework has been criticised (Stephens 2020; Clapham 2020) for inadequately explaining housing system changes, including neoliberal shifts and financialization. In response, scholars have turned to political science and sociology theories on policy change, such as historical institutionalism (Ruonavaara 2020) and discursive theories focusing on interactions between policy actors (Clapham 2018). This article builds on Clapham’s discursive turn in housing studies by incorporating concepts from ‘discursive institutionalism’ (DI) (Schmidt 2008). DI explains policy change by examining the interplay of ideas, interactions, and power dynamics in a given policy field. DI provides a methodological framework for understanding how policy actors develop and use ideas to shape policies, while considering the influence of the institutional context and power relations. The aim of the article is to highlight the utility of DI as a framework for examining housing policy change. As a vehicle for doing so, an analysis of social housing policy change in New Zealand employing DI is provided for empirical reference. The article builds on Clapham’s (2018) focus on discourse in housing studies, adding DI to the repertoire of conceptual frameworks available to researchers interested in the causal role of ideas and discourse in policy change processes.

1. Introduction

Housing scholarship has evolved significantly since Kemeny’s (1992) critique regarding the lack of theoretical grounding in the field. The formation and transformation of housing policies now constitute a vibrant area of theoretical debate, not only within housing studies but also in broader disciplines like political science and sociology. Understanding the drivers of policy change is crucial for several reasons. First, it aligns with the fundamental social science objective of explaining how and why the empirical world functions as it does. Second, this understanding can serve normative purposes by identifying opportunities to influence housing systems and propose alternative agendas. Third, examining change processes can have a critical role in revealing power dynamics and expose discrepancies between the rhetoric and actions of political elites. Additionally, comprehending the logic and dynamics of change enables comparisons between contexts, illuminating the impact of various variables on housing and facilitating the development of broader theorisation. Such comparisons can also highlight how ideas can cross-pollinate between contexts, and cause “family resemblances” to develop (Peck 2013). Finally, analysing change can unveil the pull and influence of past decisions on current housing systems and future proposals. Investigating the reasons behind, and mechanisms of, policy change is a live critical theoretical question for housing studies. Bengtsson (2015) has characterised the methodological approaches in this field as a tension between ‘structure’ and ‘Thatcher’, signifying a divide between analyses focused on macro-level institutional structures, often employed comparatively, and those centred on ‘descriptive narratives’, emphasising the role of actors and institutional context in political dynamics.

Bengtsson (2012) further challenges housing policy scholars to expand their theoretical toolkit by analysing the interplay of political actors, institutions, and discourses to identify ‘logics, patterns, and mechanisms’ that can inform analysis beyond a single empirical context. This call for theoretical advancement has prompted some housing scholars to integrate institutionalist and discursive approaches. Buitelaar and De Kam’s (2012) synthesis of sociological institutionalism and discursive theory to examine policy change in land provision for social housing in the Netherlands exemplifies this trend. However, an ample scope remains to incorporate additional insights and explanatory frameworks into the disciplinary toolkit to further enrich our understanding of housing policy change. The aim of the article is therefore to highlight the utility of ‘discursive institutionalism’ as a framework for explaining housing policy change. In discursive institutionalism ideas, discursive interactions between actors in an institutional context and power relations shape both the content and timing of policy change (Schmidt 2008; Schmidt 2011). The article first summarises and discusses existing explanatory theories of housing policy change, contrasting new institutionalist and discursive approaches. The article then introduces the core tenets of discursive institutionalism, setting out theories of ideas, institutions, actors and agency, and power relations. Discursive institutionalism is critically contrasted with other new institutionalist theories, including historical institutionalism. Lastly, an analysis of social housing policy change in New Zealand employing DI is provided as an empirical reference, highlighting the utility of discursive institutionalism as a framework for understanding the drivers of policy change. While a range of qualitative and quantitative research methods can be used to advance a DI analysis (Schmidt 2022), the empirical case in this article employed qualitative interviews with key actors in the policy field and documentary analysis.

2. Theories of Housing Policy Change

Housing policy scholars have deployed a range of theoretical frameworks and methods to describe and explain how and why housing systems develop and how and why they change. Influential within housing studies has been ‘new institutionalist’ scholarship emerging out of political science and sociology. New institutionalism emerged as a response to the perceived shortcomings of previous approaches, including ‘behaviouralist’ and ‘old’, largely descriptive institutional analysis (Schmidt 2006, p. 98). New institutionalism is an overarching term for several theoretical perspectives, with the divergent strands offering different views on institutions, their impact on political life, and processes of change (Fischer 2003, pp. 27–32; Schmidt 2006, p. 98). The major strands of NI include rational choice institutionalism, historical institutionalism, and sociological institutionalism, each with unique ontological, epistemological, and methodological characteristics (Hall and Taylor 1996; Steinmo and Thelen 1992).

Historical institutionalist (HI) accounts of policy change have featured prominently in housing research. HI scholars argue that institutions (including policies) emerge from past choices and unique conditions, leading to path dependence, where past decisions constrain future options (Pierson 2000, p. 252). In housing studies, HI is used to explain shifts in policy trajectories and institutional resilience (Bengtsson and Ruonavaara 2010). Kemeny’s (1995, 2006) ‘housing regimes’ framework also draws on HI. Kemeny’s approach aligns with Esping-Andersen’s (1990) ‘welfare regimes’, in which the form and function of welfare state institutions are theorised as the outcomes of contestation between groups with different interests in a specific social structure who seek to better or maintain their relative material position. Kemeny (2006) posits that the structure of rental markets reflects a site of struggle, whereby the institutional context supplied in a particular nation reflects the comparative power of social groupings and the institutional setting in which political conflicts are mediated. Stephens (2016, pp. 24–25) considers housing/welfare regime frameworks to reflect their moment in time, suggesting they ‘require reassessment’ in light of changes to welfare and housing systems as a result of significant social, cultural, and political changes. Stephens (2020, p. 545) further suggests that ‘middle range’ theorisation needs to both look ‘upwards’ to account for the impact of transnational forces and ‘downwards’ to interrogate country specific policymaking to make sense of changing housing systems.

Ruonavaara (2020) builds on the concept of a ‘housing regime’ in two ways: extending the housing regime analysis beyond the constitution of rental markets and by incorporating more recent HI insights into the housing regime framework to explain stability and change. Institutional change is held to occur through ‘critical junctures’ (exogenous moments of crisis or challenge) or through the gradual ‘layering’ of new policies or institutions on top of old, gradual shifts in policy focus (‘policy drift’) and ‘conversion’ of an existing institution’s purpose (Hacker et al. 2015, pp. 180–84). While HI remains prominent, scholars have critiqued the approach for struggling to articulate an explanation of human agency within institutions (Schmidt 2006, p. 106) and for offering description rather than explanation when assessing institutional change (Schmidt 2022). According to Schmidt (2006, p. 105) this is because HI lacks a ‘micro-foundational’ logic or a theory of how actors and why actors act and thus ‘it still has difficulty explaining what brings about the crisis that spurs change’, i.e., the process by which the ‘punctuating’ of policy equilibrium occurs or the processes by which actors pursue a ‘layering’ project to modify existing policies (Schmidt 2006, p. 105).

Social constructivist approaches, including those focused on the role of ideas and discourse in policy, also have an established presence in housing research. Hochstenbach (2015) examines how groups of actors organise and coalesce within policy fields, following Hajer (1997) and Sabatier and Jenkins-Smith (1993). Other scholars have analysed how actors develop and deploy ideological perspectives through narratives in policy development processes or via housing policy debates (e.g., Bierre and Howden-Chapman 2020; Dodson 2007; White and Nandedkar 2019). Hierarchies of power and authority within policy fields have also been examined, demonstrating a link between the social power of actors and the policy direction adopted by governments (e.g., Marston 2004). Gramsci’s (1971) concept of ‘hegemony’, according to which elite groupings exercise ideational power to shape perceptions of social life, is deployed by Ivanova (2011) to explain the construction and continuity of post-war housing expansion in the United States.

New institutionalist and ideational or discursive scholarship has tended to operate separately, with new institutionalism geared towards investigating ‘causal regularities’ in policy while discursive or ideational analysis is concerned with ‘understanding meaning’ (Kulawik 2009, pp. 262–63). There is value in ‘bringing together’ a focus on meaning making and institutional dynamics, to explain how housing policies are constructed and remain or change. Clapham’s (2018, p. 3) view of housing regimes as ‘the set of discourses and social, economic and political practices that influence the provision, allocation, consumption and housing outcomes in a given country’ represents a step forward by focusing on discourse, agents, and structure. For Clapham (2018, pp. 34–35) ‘housing policy is influenced by the policy-making discursive games that are reified in the institutional structure, as well as the welfare ideology and the finance system’. Clapham’s (2018, p. 37) analysis of six institutionally and economically different countries describes a growing influence of neoliberal ideas. However, Clapham (2020) articulates an expansive (‘holistic’) rather than specific approach to operationalising a focus on discourse.

Discursive institutionalism (DI) offers a focused methodological framework that integrates new institutionalist, ideational, and discursive insights to show how actors undertake meaning making, coordinate to (re)make institutions (specific ‘policies’ are understood as institutions), and experience power relations (Schmidt 2010, p. 3). Schmidt (2022) frames DI as an ‘umbrella concept’, drawing together scholarship focused on the role of ideas and discourse in shaping policymaking.

3. Discursive Institutionalism

DI focuses on the content of ideas, how actors convey ideas through discourse in institutional settings, and how power relations shape the ascendency or decline of ideas.

3.1. Ideas in Discursive Institutionalism

Ideas are crucial for understanding policy change, as they provide the substantive content of discourse that actors use to enact or resist change (Schmidt 2008). Discourse comprises both ideational content and the interactive processes through which ideas are communicated and implemented. Ideational scholars (e.g., Mehta 2010; Schmidt 2008) identify three ‘levels’ of ideas at play in public policymaking processes: first-level policy ideas, second-level programmatic ideas, and third-level public philosophies. Additionally, there are two types of ideas: cognitive and normative.

First-level ideas are specific policy solutions, such as ‘smaller class sizes’ or ‘broken windows policing’, that address particular problems (Mehta 2010). According to Mehta (2010, p. 28), policy solutions entail ‘the narrowest conceptualisation of the role ideas plays in politics and the most theoretically developed’. Policy solutions are underpinned by a broader framework, by second-level ‘programmatic ideas’, providing a broader ideational resource that defines problem definitions and gestures towards appropriate solutions (Schmidt 2008). Second-level ideas sensitise policy actors as to the issues to be considered important and the ‘norms, methods and instruments’ to be applied (Schmidt 2008, p. 306). Thus, they are termed ‘programmatic ideas’ for the way they guide the generation of first-level ideas (Schmidt 2008, p. 306).

Hall’s (1993) concept of policy paradigms illustrates how programmatic ideas can shift, for example, the transition from the acceptance of Keynesian economic ideas to the adoption of neoliberal ideas in the United Kingdom and elsewhere. Hall (1993) explains the paradigm shift as resulting from the failure of the Keynesian paradigm to generate policy solutions deemed sufficient and appropriate to tackle the United Kingdom’s economic challenges in the 1970s, and due to the organised propagation of neoliberal ideas within interlocking networks of think tanks, political actors, and civil servants. Policymakers embraced neoliberalism as it represented a generative ideational framework for new policy solutions and because actors in important institutional locations organised and promoted an epistemic shift in favour of neoliberal ideas. Programmatic ideas used by policymakers to guide decision making can be sites of struggle for ideational hegemony by elites (Jobert 1989; Schmidt 2008, p. 306), with actors holding competing ‘cognitive maps’ (Axelrod 1976; Jobert 1989, p. 377) or views on the cause of policy problems and the effect of policy solutions.

Drawing on Lakatos (1976), Sabatier and Jenkins-Smith (1993; Sabatier et al. 2014) extend the discussion of ideas by differentiating between ‘deep core’ beliefs, ‘policy core’ beliefs, and ‘secondary beliefs’. Deep core beliefs consist of broad normative values and ontological assumptions that are not specific to any policy but involve general conceptions of human nature and social priorities. Policy core beliefs, on the other hand, relate to specific policies and include both normative and empirical elements. These beliefs guide actors in assessing the seriousness of problems, identifying causes, determining preferred solutions, and making value judgements on whose welfare to prioritise. For example, policy core beliefs will guide actors on the balance between state or market responsibility on a given issue or on the ‘relative importance of economic development vs. environmental protection’ (Sabatier et al. 2014, p. 3). Secondary beliefs are more flexible and context-specific, addressing narrower issues (e.g., context specific policy or regulatory preferences, institutional design, budgetary allocations, etc.) and allowing for adjustments based on new evidence or changing circumstances.

Third-level ideas, or public philosophies, underpin policies and programs with fundamental values and principles (Schmidt 2008). These are ‘underlying assumptions’ which ‘sit in the background’ and are frequently only debated in crisis (Schmidt 2008, p. 306). Schmidt (2008, p. 308) notes that the scrutiny of public philosophies has ‘often been the domain of macro-sociologists’. Schmidt (2008, p. 308) points to Bourdieu (1998), Foucault (2000), Gramsci (1971), and Weber as theorists “who present public philosophies as the ideas of the powerful who dominate society”. Foucault’s (1971, 1979, 1992, 2000) notion of ‘truth regimes’; the language–power nexus; and ‘governmentality’ have influenced the scholarly understanding of how ideas emerge and come to permeate and organise social life at a deep level. Analytically, third-level ideas can be the hardest to pin down and point to, according to Schmidt (2008, p. 308); this is because ‘it is often the case in a given society that, at a very basic level, “everyone knows” what the basic philosophy of worldview is, even if they may not be able to define it precisely or describe how it developed or changed’. Nevertheless, it is clear that public philosophies do organise ideational reality at a deep level, including shaping the norms of housing systems (e.g., Ivanova 2011; Kemeny 1981, 1992).

3.2. Cognitive and Normative Ideas

DI holds that ideas at each of the three levels are held to contain two ‘types’ of ideas: ‘cognitive ideas’ and ‘normative ideas’. Cognitive or ‘causal’ ideas speak to ‘what is and what to do’; thus, cognitive ideas serve as ‘recipes, guidelines, and maps for political action and serve to justify policies and programs by speaking to their interest-based logic and necessity’ (Schmidt 2008, p. 306). By contrast, normative ideas ‘attach values to political action and serve to legitimate the policies in a program through reference to their appropriateness’ (Schmidt 2008, p. 306). Put simply, cognitive ideas set out the logic of why an idea will achieve an outcome, while normative ideas convey the value of why a policy is appropriate and should be adopted. These two types of ideas operate across the three levels of ideas (Schmidt 2008, p. 307). Cognitive ideas manifest in first-level policy solutions in terms of their claims to offer solutions to problems. Cognitive ideas within second-level programmatic ideas define problems and identify methods to solve them. At the third level of ideas, cognitive ideas manifest as principles and norms of technical or scientific practice. Normative ideas are at play in the claims of first and second-level ideas to align or represent ‘aspirations and ideals’ of the public, and to tie first and second-level ideas to third-level accepted norms and social values.

3.3. Discourse and Institutions in Discursive Institutionalism

‘Discourse’ is a multifaceted term in social sciences. Schmidt (2008) notes the term often evokes images of postmodernist and poststructuralist interpretations of texts devoid of context. Within DI, discourse is understood as ‘whatever policy actors say to one another and to the public in their efforts to generate and legitimise a policy programme’ (Schmidt 2002b, p. 210). Discourse consists of a substantive ideational element (‘policy ideas and values’) and an interactive element in the form of the communicative practices through which ideas are put forward by actors (Schmidt 2002b, p. 210; 2008, p. 305). The ‘ideational dimension’ of discourse ‘performs both a cognitive function, by elaborating on the logic and necessity of a policy programme, and a normative function, by demonstrating a policy’s appropriateness through appeal to national values’ (Schmidt 2002b, pp. 210–11). However, Schmidt (2008, p. 305) is clear that discourse is ‘not just ideas or text (e.g., ‘what is said’) but also includes context (e.g., ‘where, when, how, and why it was said’). Thus, discourse refers ‘not only to structure (what is said, or where and how) but also to agency (who said what to whom)’ (Schmidt 2008, p. 305). This view sets Schmidt’s approach to discourse apart from approaches that focus primarily on text or language alone (e.g., Fairclough 2000; Wodak and Meyer 2001).

Schmidt (2008, 2012, 2015, p. 175) sets out a schema for considering discourse as a social practice. The connection between ‘discourse as ideas’ and ‘discourse as social practice’ is found in the capabilities of agents to interact and organise. Actors have ‘background ideational abilities’ and ‘foreground discursive abilities’. A background ideational ability coincides with Bourdieu’s notion of ‘habitus’, whereby agents are shaped by the structure or context in which they are operating, yet Schmidt (2008) views actors as capable of possessing the ability to competently navigate structures. This means actors are never completely subject to or constrained by ‘path dependence’. Rather, actors have the ability to ‘communicate, argue, and deliberate about taking action collectively to change their institutions’ (Schmidt 2015, p. 176). Discourse is therefore integral to institutional or policy change as agents produce change by creating and holding ideas and persuading others. This is a point of difference to HI approaches; HI approaches neglect the role of actors’ motivations and ideas in driving system change, focusing instead on describing moments of disruption or continuity.

3.4. Coordinative and Communicative Discourse

Discourse in policy change serves both a coordinative function (providing a common language for policy construction) and a communicative function (public presentation and deliberation) (Schmidt 2002b, 2008, 2015). Schmidt identifies two modes of discursive activity: coordinative discourse (involving the creation, deliberation, and agreement on policies among actors) and communicative discourse (geared towards policy legitimation in the public sphere) (Schmidt 2015). Coordinative discourse often occurs among bureaucratic and political actors, but can also involve communities, civil society, academia, and business. Communicative discourse involves presenting, deliberating, and legitimising policy ideas in the political sphere, i.e., attempting to justify and legitimise policies with citizens (Schmidt 2015, pp. 180–183). As with coordinative discourse, communicative discourse generally involves a range of actors and can be ‘top down’ (e.g., communications by governing actors) or ‘bottom up’ (e.g., public communications by citizens, civil society, or protest movements) (Avigur-Eshel 2019).

3.5. Institutional Structures

Within DI, institutions shape how actors engage in coordinative and communicative discourse. For example, states with a ‘single actor’ with strong executives (as in some Westminster parliamentary systems) may see less ‘coordinative discourse’ amongst political actors (Schmidt 2002a). This is because coordinative discourse may be closed to all but a small set of actors located within the executive group and bureaucracy as a result of no strong institutional requirement to include a broader range of actors. By contrast, states with ‘multi-actor’ political institutions may require greater levels of consultation and negotiation in a more inclusive coordinative discourse for a policy to proceed. This is due to institutional factors such as the necessity of coalition arrangements; power-sharing agreements; dynamics in bicameral parliaments; and dynamics between levels of government. Single actor systems tend to have a ‘thin’ coordinative discourse while ‘multi-actor’ political systems have a ‘thicker’ or more elaborate coordinative discourse, reflecting the need to involve (and convince) a wider range of actors (Schmidt 2002a, p. 172).

3.6. Ideational Power and Discursive Institutionalism

Power relations impact the trajectory of ideas in the world. Carstensen and Schmidt (2016, p. 320) define ‘ideational power’ as ‘the capacity of actors (whether individual or collective) to influence normative and cognitive beliefs through the use of ideational elements’ and set out three ways that ideational power shapes policy development. ‘Power through ideas’ refers to the capacity of actors ‘to persuade other actors to accept and adopt their views of what to think and do through the use of ideational elements’ (Carstensen and Schmidt 2016, p. 321). Actors use cognitive and normative ideas in an attempt to persuade others. Secondly, ‘power over ideas’ refers to actors imposing or resisting ideas through the imprimatur or their institutional location, maintaining the dominance of their own ideas and warding off ideational challenge (Carstensen and Schmidt 2016, p. 321). This theorisation aligns with van Dijk (1993, 1995) who views ‘access to discourse’ or the ability to be heard and taken seriously in institutional contexts as a critical factor in influencing decisions. ‘Power over ideas’ is not the sole preserve of elites. Other actors, including those in less powerful institutional locations (e.g., civil society, advocacy coalitions, protest movements) are able to exert pressure on powerful actors to adopt ideas or take action they would otherwise decline (Carstensen and Schmidt 2016, p. 326). In this mode, power is deployed via communicative discourse to appeal to social norms or public opinion to cajole powerful actors to pursue an idea (Carstensen and Schmidt 2016, p. 326).

The third form, ‘power in ideas’, refers to the ‘hegemony’ of background ideas and assumptions, i.e., ‘third-level ideas’, that may stay hidden as accepted ‘common sense’ (Carstensen and Schmidt 2016). Whereas the previous two forms focus on interaction within a policy field, ‘power in ideas’ operates in the background as ‘ideational processes—constituted by systems of knowledge, discursive practices and institutional setups—that in important ways affect which ideas enjoy authority at the expense of others’ (Carstensen and Schmidt 2016, p. 329). Powerful background ideas shape broader public norms of acceptability and appropriateness; thus, actors may appeal to such ideas in order to ‘depoliticise’ their actions or to explicitly couch their own goals in a publicly acceptable manner (Carstensen and Schmidt 2016, p. 329). If certain ideas become ‘taken-for-granted’ and ‘fade into the background’, they possess powerful qualities as norms. Third-level ideas are not necessarily deployed explicitly or deliberately, although this is possible; nevertheless, such background ideas impact how actors undertake their second and first-level ideational development, particularly as they work to moderate ‘the range of alternatives that elites are likely to perceive’ as well as providing guidance on what ‘could’ or ‘should’ be done (Carstensen and Schmidt 2016, p. 329).

4. Discursive Institutionalism and Housing Policy Change

Applying discursive institutionalism to understand housing policy change requires a focus on ideas, actors who wield ideas and interact, the institutional environs discourse occur within, and ideational power dynamics. Processes of interaction occur within a ‘policy field’, whereby ‘policies’ are understood as centres of social activity in which policies create threads of relations, semantic spaces, subjectivities, and forms of knowledge (Shore et al. 2011). Certainly, a policy field in a given context will include political and bureaucratic actors, as well as interest groups (e.g., tenants associations, commercial interests, housing justice movements, financial institutions etc.), media, think tanks, and academics. Ideas are the currency of housing policy change and policy stability. Where actors perceive current policy settings to be ‘working’, i.e., aligning with the cognitive and normative content of key ideas or producing desired outcomes, stability is likely to permeate housing policy. Where actors perceive a policy to be delivering less than satisfactory outcomes, to be out of step with actors’ preferred ideas, or to be a challenge to public values, change may occur. Alternatively, exogenous events may introduce political, economic, and social obstacles to the sustaining of a policy, yet far from a response to such circumstances being automatic, actors will draw on the cognitive and normative content of ideas to generate their approach to navigating through these obstacles. Here, it is worth pointing to Milton Friedman’s oft quoted observation that in a crisis, actors reach for ‘the ideas that are lying around’.

Not all actors are created equal within a policy field. Actors in particular institutional locations may, by virtue of that location, have greater agency to take decisions and exercise ideational power. An interactive coordinative discourse can socialise new ideas and build support for their implementation or provide continuing support for the status quo. Attempted legitimation through communicative discourse can occur, and may involve opposition from other actors attempting to challenge, reframe, or replace policy agendas. The three levels of ideational power may operate to facilitate or stymie actors pursuing their ideas.

A key consequence of discursive institutionalism is therefore that the ideational landscape underpinning housing policies in a given jurisdiction is not fixed, but dynamic. Furthermore, these discursive institutional processes are the causal drivers of historical institutionalist descriptions of change processes, i.e., ‘conversion’, ‘layering’, and ‘drift’ (Streeck and Thelen 2005). Programmatic ideas, e.g., neoliberal ideas, hit the ground and become embedded in policy settings as a result of discursive interactions within a policy field, contoured by institutional structures and ideational power relations. According to (Birch and Siemiatycki 2016, p. 172) ‘neoliberalisation does not, and indeed, cannot be considered as a singular process; it is, rather, a varied and variegated process of geographical transformation’. Thus, examining the ideational interplay occurring amongst actors within a given policy field is critical to understanding how policies develop and impact a particular context.

5. Explaining Housing Policy Change through Discursive Institutionalism: An Empirical Case Study of Social Housing Reform in New Zealand

New Zealand’s conservative National Party-led government undertook social housing policy reforms between 2010 and 2017. The research focused on questions of causation, policy design, and efficacy, i.e., why did the National government embark on social housing reform, why and how did the government develop the policy model it did, and what were the outcomes of its policy decisions? DI was operationalised as a methodological framework to reveal the causal interplay of ideas, actors, and institutions that generated the impetus for reform, shaped the policy model, and produced social and political outcomes. The next section describes the methods used. Subsequent sections provide a general overview of the policy reforms, before moving to examine three specific moments in the reform: reform initiation, the construction of a new policy model, and legitimation challenges and the subsequent establishment of a new emergency housing policy.

5.1. Methods

The research employed a qualitative research design that combined semi-structured interviews with documentary analysis, allowing a comprehensive analysis of the policymaking process. Following an initial period of document analysis, potential participants were selected on the basis that they had played a role as an actor in the policy field in some capacity between 2010 and 2017. The selection approach was designed to allow for “studying through” the policy field (Reinhold 1994; Shore and Wright 1997) or selecting participants in different locations within the policy field to understand how they relate and connect to the locations and experiences of other actors. This approach dovetailed with Schmidt’s (2008) view that multiple actors engage in any policy field, and methods to illuminate the ideas and actions of actors are required to undertake discursive institutionalist analysis. The potential sample for the research therefore included political actors, bureaucratic actors, community housing sector actors, opposition political actors, and members of the government’s external advisory group on housing, and a ‘purposeful sample’ (Curtis and Curtis 2011) of participants was justified based on documentary analysis, indicating an individual had clear involvement in or knowledge of the reform process.

In total, 15 semi-structured interviews were conducted with actors in the policy field, including cabinet ministers (including the Minister of Finance, Bill English), opposition politicians, senior officials from key departments, community housing sector representatives, and members of the government’s advisory group. The interviews gathered data on participants’ ideas, roles, institutional location, power profile, and experience of interacting in the policy field. Several serving public officials and politicians declined to be interviewed, representing a limitation to the study. Transcripts were coded to identify insights on ideas, coordinative discourse, communicative discourse, power dynamics, and views on the success or failure (depending on the participant) of the reform programme. These discursive institutionalist elements also acted as the primary coding framework for the analysis of documents. New Zealand’s freedom of information regime provides liberal access to official information for citizens compared to other jurisdictions (i.e., Australia, Canada) (Treadwell 2020), allowing for an in-depth analysis of policy documents and mitigating participant access challenges. Freedom of information requests enabled access to internal government documents, such as briefing notes, memoranda, and cabinet papers. A range of documents in the public domain, including media reports, parliamentary speeches, and publications by civil society organisations were also analysed. Document analysis allowed the policy field to be mapped and revealed how actors developed and wielded ideas and interacted to shape the policy development process. Key documents are cited in this article, with a wider set of documents (over 500 documents were analysed in total to achieve ‘analytic saturation’ (see Seale 1999, p. 92), informing the research and this article. Triangulation was achieved by cross-referencing interview data with documentary evidence, to address potential issues of misinformation, bias, or memory limitations.

5.2. An Overview of New Zealand’s Social Housing Reform Experience 2010–2017

New Zealand’s conservative National Party-led government took office in November 2008, commencing work on social housing policy reform in earnest shortly thereafter, in 2010. The policy reform process continued throughout the remainder of the government’s term until the election of the Labour-led government of Jacinda Ardern in 2017. Section 5.3 illuminates how the National government’s reforms were driven by senior political figures and supported by influential bureaucratic agencies. These key actors developed their agenda in line with neoliberal ideas, including New Public Management (NPM), and exercised ‘power over ideas’ by virtue of their central location within New Zealand’s policymaking apparatus. Their approach sought to marketise social housing provisions by encouraging non-government community housing providers (CHPs) to compete with the national, unitary public housing corporation, Housing New Zealand (HNZ). The National-led government articulated a new discourse that elevated the role of non-government organisations in the provision of housing. ‘Social housing’, instead of ‘state housing’ (the established term for publicly owned stock), became the preferred nomenclature of the government, signalling a broadening of who was expected to provide and be responsible for housing support. Section 5.3 further details how a government-appointed advisory panel (the Housing Shareholders Advisory Group or HSAG) initially recommended a greater state provision of funding, including providing capital grants and subsidies to CHPs and implementing active measures to enhance low-cost private rental availability and affordable homeownership, recognising the interconnection between a ‘spectrum’ of housing tenures. The government and Treasury (the finance ministry) exercised power and narrowed the scope of the proposals, limiting CHP funding to the access of weekly rent subsidies (Income Related Rent Subsidies or IRRS) but not the substantial capital investment required for significant sector growth. This policy was contradictory; while CHPs were expected to play a larger role, the government—focused on minimising costs—declined to invest in expanding the small sector.

Section 5.4 examines the process by which a new social housing policy model was designed. This came in the form of a ‘transfer’ scheme, gradually developed and implemented through a cloistered coordinative discursive process between cabinet ministers and officials in the Treasury. The new model, by now termed the Social Housing Reform Programme (SHRP), reflected the ideational prism of these central actors. Their cognitive and normative commitment to neoliberal ideas on the role of the state underpinned plans for the financialisation of New Zealand’s social housing stock. The transfer model consisted of the auctioning of parcels of “state houses” (a New Zealand-specific term for public housing) to the consortia of CHPs financially underwritten by commercial investment banks. Concurrently, a material ‘housing crisis’, impacted by the stagnant social housing provision and controversy over the ‘transfer’ of 1100 houses in the city of Tauranga, led to public dissatisfaction, and a political crisis pressuring the government into rapid and costly policy measures in an attempt shape communicative discourse and maintain legitimacy. Consequently, and ironically given its longstanding housing funding parsimony, substantial funds were allocated to emergency housing, including the use of motel accommodations, a feature of the New Zealand housing system that persists. These matters are the focus of Section 5.5.

5.3. Initiating Social Housing Reform

The impetus for reform stemmed from the ideas and actions of key political actors in the National government, primarily Bill English, the finance minister and a ‘shareholding minister’ in Housing New Zealand (then New Zealand’s public housing agency), and Phil Heatley, the housing minister. Prior to entering Parliament, English had served as a Treasury official at the height of the agency’s advocacy for neoliberal reform (see Kelsey 1995). English was elevated to cabinet in the fourth National government (1990–1999), leading health system reforms underpinned by NPM ideas. English also served for a period as the opposition leader (2001–2003), expressing a normative belief that “important decisions and choices are made within families and within communities. Any government should be careful about interfering too much” (English 2001). English (2003) also expressed dissatisfaction at the operation of New Zealand’s state housing system, framing tenure security and rent subsidy for public tenants as both disadvantaging lower income citizens in private rental accommodation and dampening ‘motivation’ for state housing tenants who were framed as at risk of ‘dependency’ by the Clark-led Labour government (1999–2008). Appeals to normative ideas of individual choice and personal responsibility underpinned the justificatory discourse of both the fourth and fifth National governments (Humpage 2015, p. 332). In terms of cognitive ideas, English (2001) framed non-government actors and communities as ‘dynamic’—better able to organise responses to social ills in more efficient and effective ways than the state. However, during this period of opposition, the National Party abandoned its 1990s policy of setting market rents for state housing tenants and providing a voucher system for eligible citizens regardless of tenure (see Murphy 2004). This policy had been contested in communicative discourse and had been rolled back by the Clark-led Labour government. National Party housing spokesperson Tony Ryall stated “…we’re not committed to restoring what happened in the 1990s, we’re working on a policy for the next decade” (Radio New Zealand 2000). The National Party’s 2008 election manifesto contained no hint that the National government would pursue social housing policy reform; however, English would play a central role in driving endogenous change.

Three factors mark English as a pivotal figure in the social housing reform process pursued by the Fifth National government. Firstly, his institutional location as a senior and experienced minister provided him with a powerful position to direct activities and processes, and to make ideational judgements. Secondly, English held both normative and cognitive ‘second-level’ ideas on how social service provision should be organised and operated in New Zealand, with a strong interest in reshaping the organisation and in the performance of the state housing system. In a research interview, Bill English elaborated on his interest in reform. English stated the following (Bill English, interview, 2019):

I would say there were two drivers. The first was that I had a long term interest in the interaction of interaction housing subsidy with welfare and work. And we [National in government] had been through an experience in the 1990s of trying to change the subsidy system in order to equivalise the accommodation benefit [AS] with the Housing NZ Corp subsidised income related rents [IRRS]. That had turned out to be politically unsaleable and was undone. I was interested in finding other ways. I was pretty interested in the way that the state housing system worked to trap people into living in particular areas as part of a particular long term culture that may have meant that it was very difficult for them to make the transition from being on a benefit in a state house to getting into work. So that was one driver. So I was keen to find other ways of looking at it—in the knowledge that aligning the subsidies was not an option.

Thirdly, English played a crucial role in commissioning a ministerial advisory group (the Housing Shareholders Advisory group or HSAG) to provide ideas for reforming state housing, a process which visibly marked the beginning of a reform. English therefore played the role of the key ‘policy entrepreneur’ (Béland 2016; Kingdon 1984) in the development of a state housing reform agenda. The period following the 2008 general election provided a ‘window of opportunity’ (Kingdon 1984) for considering policy change. Policy entrepreneurs must perceive that the ‘window’ is open and make decisions which set processes in motion (Béland 2016, p. 234). English used his background ideational abilities and foreground discursive abilities (Schmidt 2008, p. 316) to navigate through the opportunity window, bringing in other actors in a coordinative discourse on policy change.

English’s interventions in the coordinative discourse on housing policy frequently deployed a ‘third-level’ or underlaying assumption that individual agency and the non-government organisation of social services generated preferable social outcomes, in contrast to state social service provisions, which could remove agency from individuals, and foster ‘dependency’. The new housing minister, Phil Heatley, shared English’s views and envisaged both a curtailment of direct state provision with a concomitant increase in the role of non-government social housing provision:

We came from a National Party perspective where we put focus on independence, and independence in housing rather than to grow the state housing stock…So we didn’t come in thinking we’d be increasing the number of houses on the ground…it was never our intention as a government to end up through Housing New Zealand owning more state houses in number. But it was always our intention to have more social houses or social housing places in number between us [government] which might stabilise or even decrease…and whatever community organisations and all the others can get out of it would see an overall increase. But we [the government] didn’t want to be a bigger landlord… we didn’t want to be landlord of more houses ourselves.

(Phil Heatley, interview, 2018)

The convergence of their ideas, coupled with the political power they wielded as ministers, established the path for a significant shift in social housing policy. The policy field during the government’s initial period comprised various actors. The Treasury (finance ministry), with its authoritative position in the bureaucracy (Goldfinch 2009, p. 185) and strong and ongoing adherence to neoliberal ideas (including New Public Management (NPM) principles, public choice theory, transaction cost economics (see Kelsey 1995, 2015)), and commitment to limiting fiscal policy, played a critical role in shaping the coordinative discourse on reform. A Treasury (New Zealand Treasury 2009) report to English ahead of the 2009 Budget criticised HNZ for failing to operate ‘efficiently’ and questioned whether HNZ’s approach was ‘the most effective way of addressing state housing need’. The Treasury further suggested a regular assessment of tenant ‘need’ with accompanying ‘encouragement’ into the private rental market for some tenants, and for the government to articulate an ‘expected level of return’ on capital investment, as HNZ did “not provide a commercial rate of return on the Crown’s capital investment”.

The Treasury’s focus on achieving ‘efficiency’, ‘value for money’, and a scepticism of HNZ’s performance resonated with English’s ideational outlook and bolstered the government’s interest in pursuing reform. New Zealand’s nascent community housing sector had gained some momentum during the previous Labour government and actively advocated for an expanded role in social housing provisions through its national peak body. This advocacy, grounded in the idea of community empowerment and local responsiveness, found some resonance with the National government’s interest in leveraging non-government actors to address the challenges facing the sector. However, as the reform process unfolded, the power dynamics within the policy field became increasingly evident, with the Treasury’s influence overshadowing that of other actors. The sector had limited power within the coordinative discourse on policymaking. The establishment of HSAG in February 2010 marked a significant step in the reform process. Tasked with providing advice on social housing reform, the group’s membership represented various sectors, including community housing, social services, building, and finance. Notably absent were tenant representatives and housing academics, reflecting the government’s preference for a particular ideational lens. The HSAG report, Home and Housed, offered a vision for social housing that encompassed the broader housing spectrum and pushed beyond the government’s more restrictive terms of reference (see Murphy 2019). While the report affirmed a turn to greater provisions from non-government providers, contestability, and the potential use of non-government capital, the report also advocated for increased government funding and support for community housing providers, recognising the limitations of a purely market-based approach and the need for state intervention beyond social housing and into private rental and ownership markets. An ‘Affordable Housing Agency’ (AHA) was mooted to drive an affordable housing policy agenda (HSAG 2010, pp. 77–82). The government’s response to the HSAG report was selective, shaped by the dominant ideational framework of the Treasury and ministers, and power dynamics within the policy field, i.e., HSAG had limited ‘power over ideas’ as an actor in its own right. The Treasury’s (New Zealand Treasury and Department of Building and Housing 2010) scepticism about certain HSAG recommendations, particularly those related to demand-side interventions, the breadth of HSAGs vision, and the establishment of an AHA, led to their exclusion from the final policy model. However, a capital fund was made available to community housing providers to increase development, with the fund oversubscribed from demand. The Treasury’s dominance in this phase, driven by its institutional power and expertise in economic policy, resulted in a narrowing of the reform agenda envisaged by HSAG and a focus on measures that aligned with its ideational framework. The Treasury’s (New Zealand Treasury 2012a) concerns relating to ‘fiscal sustainability’ and ‘value for money’ led ministers to suspend capital funding for community housing providers, a decision that was met with criticism from the sector and one that set the scene for exploring alternative policy options.

5.4. Constructing a New Policy Model

With a narrowed agenda and an austere approach to public funding in place, the reform process moved into a deliberative phase of further coordinative discourse, a protracted period marked by extensive interaction among various actors in the field. This phase, which extended beyond the initial timeframes allocated by the Cabinet, was instrumental in shaping the direction of the reform and ultimately led to the adoption of a controversial financial model for the housing reform now known as the Social Housing Reform Programme (SHRP). The deliberative phase was dominated by the Treasury, which, despite its limited expertise in housing policy, continued as the central bureaucratic force driving the reform process. The sustained centrality can be attributed to several factors, including the Treasury’s institutional position within the bureaucracy, its perceived expertise in financial matters, and its close relationship with the Minister of Finance. This concentration of power in the Treasury’s hands had significant implications for the reform process. Other agencies, such as the Ministry of Social Development (MSD), found their perspectives marginalized as the Treasury’s economic rationalist viewpoint prevailed. MSD, with its focus on the social aspects of housing and the needs of tenants, struggled to counter the Treasury’s dominant narrative, which prioritised financial considerations and cost control. According to one senior MSD official (MSD official, interview, 2018):

…there was a real tension between Treasury advisors, Treasury staff and everyone else I guess. But particularly with MSD…. They were at almost opposite ends of the spectrum. I kind of got the sense that there was this ‘New Right’ perspective from Treasury at play...”this is just an economic problem about how you deal with assets and how you provide assistance to people” which can be number crunched. Where of course, the Ministry of Social Development had much more nuanced view about people and their trajectories.

Officials and ministers struggled to clearly articulate a “desired end-state” for the SHRP policy. Officials sought guidance from ministers on fundamental questions, such as the optimal structure of the sector, whether third-party providers were to complement or compete with HNZ, and the desired housing outcomes sought by the government. A restrictive fiscal policy supported by both the Cabinet and the Treasury that had presided over the removal of capital grants further manifested in the ruling out of no or low-cost transfers of stock to increase the scale of community housing providers. With these options removed, the government on the one hand maintained that it wished to see the community sector grow provision, while on the other hand refused to inject capital to allow providers to scale up. Dykes (2016, p. 68) notes the mismatch between the stated aims of the SHRP and the community housing sector: ‘represent an inherent contradiction in the policy logic, given that one of the original primary drivers of the policy was the purported ability of community housing providers to access capital from ‘multiple public and private philanthropic sources’. The apparent contradiction is explained by a preference, amongst the central actors, for muting the government’s fiscal contribution and for involving the private sector, including financial institutions, in the financing of stock transfers.

The finalised model involved a stock transfer process that encouraged community housing providers to ‘partner’ with financial institutions to acquire state housing stock and claim IRRS subsidy. Prioritising ‘value for money’, the government enacted a policy to transfer stock at below market rates while retaining a portion of equity and a share of future capital gains through a “Crown retained interest” encumbrance instrument (New Zealand Treasury 2015). This approach aimed to balance SHRP policy goals with the Treasury’s concerns about the impact of low-cost transfers on the government’s balance sheet and New Zealand’s credit rating, as state housing stock represented a significant ‘asset’ in the government’s books (see New Zealand Treasury 2012b).

5.5. Legitimation Challenges and the Establishment of the ‘Emergency Housing’ System

Where coordinative discursive processes focus on policymaking, actors may engage in communicative discourse in the public sphere to attempt to justify or challenge policies. Beginning in 2014, a mounting public discourse surrounding housing affordability and supply placed increasing pressure on the government, compelling it to adopt a defensive stance in communicative discourse. The public sphere became increasingly dominated by concerns over social housing shortages, escalating rents, rising homelessness, and increasing property values and visible homelessness. By 2015, public opinion polling suggested a record level of public concern over housing affordability and supply (Roy Morgan Research 2015). This pressure was further exacerbated by the government’s Social Housing Reform Programme (SHRP), a policy initiative that had ultimately resulted in the stagnation of available social housing places (Johnson 2017, p. 9). Initially, the SHRP was met with a muted public response. However, following the 2014 election, the programme faced heightened scrutiny. The government’s attempts to engage charitable organisations and private developers in the transfer of state housing stock were met with increasing resistance and public criticism. This resistance, coupled with the broader narrative of a “housing crisis”, served to undermine the legitimacy of the government’s housing policies. Despite initially championing the SHRP as a policy equipped with cognitive sense and normative appropriateness, pressure in the communicative discourse pushed the government to adapt its strategy in response to mounting public pressure and worsening housing conditions. This shift culminated in a significant change in policy direction, marked by the creation of a new ‘emergency housing policy’: renting rooms in motel accommodations to house individuals and families in housing need at a significant fiscal cost. The government’s handling of the housing crisis was further complicated by its attempts to involve private capital and overseas providers in the SHRP, a decision that drew criticism from some potential community housing providers (Butcher 2015), other civil society groups, and the media (The Press 2016). This criticism, amplified by media coverage and support from opposition political actors in the context of an upcoming general election, presented a substantial challenge to the government’s legitimacy in the eyes of the public. The Ardern Labour-led government, successful at the 2017 election, ended the SHRP in late 2017 and promised to increase the number of government-funded state houses.

6. Discussion

This case study underscores the utility of discursive institutionalism in analysing housing policy change. It highlights how ideas, actors, and the institutional context interact to shape policy outcomes, and how power dynamics influence the ascendency or decline of ideas. This case study of New Zealand’s social housing reforms illustrates how DI reveals causal factors behind policy shifts and the complexities of policy formulation. The study also demonstrates the importance of considering all three levels of ideas—policy ideas, programmatic ideas, and public philosophies—to understand the full range of factors influencing policy change. For instance, the centrality of neoliberal ideas, particularly New Public Management principles, in shaping the reform agenda underlines the power of programmatic ideas in guiding policy solutions. Similarly, the influence of deep-seated public philosophies regarding the role of the state in relation to the individual underscores the importance of considering the broader ideational landscape in which policies are formed. This case study also highlights the role of actors as both agents of change and subjects who must navigate institutional contests. The actions of key policy entrepreneurs like Bill English and Phil Heatley, driven by their ideological commitments and institutional positions, were pivotal in initiating and shaping the reform process. However, their agency was not absolute, as they had to navigate the constraints imposed by the power dynamics within the policy field—namely, the ‘feedback loop’ in the form of the communicative discourse that delegitimised the government’s housing policy record. The institutional context, particularly the dominance of the Treasury, played a crucial role in shaping the reform process. The Treasury’s economic rationalist perspective and focus on fiscal constraint significantly influenced the direction of the reform. This highlights the importance of understanding the institutional landscape in which policy change occurs and the power dynamics that shape the discourse: some actors are able to exercise ideational power and shape coordinative discourse in ways others are not. Finally, the case study illustrates the importance of communicative discourse in influencing policy outcomes. The growing public discourse of the ‘housing crisis’, advanced by media coverage and political opposition, exerted significant pressure on the government, ultimately forcing it to adapt its policy stance and inject public funding.

7. Conclusions

This article has highlighted discursive institutionalism as a framework for understanding housing policy developments and change. By focusing on the interplay of ideas, actors, and institutions, and by considering the role of power dynamics in shaping discourse, DI provides an analytic framework for understanding of the complex processes of ideation and interaction that drive policy change. The empirical case of New Zealand’s social housing reforms demonstrates the practical application of this framework and its use in analysing real-world policy changes. While historical institutionalism has been influential in housing policy research, it struggles to explain the agency of actors and the role of ideas in driving change. Discursive institutionalism, by contrast, explicitly addresses these factors. Policy change is not merely a product of institutional inertia or external shocks, but a dynamic process fuelled by competing discourses, strategic interaction and manoeuvring, and the interplay of power relations. By spotlighting the role of discourse and the power dynamics inherent in policy fields, discursive institutionalism transcends descriptive accounts and offers a more incisive explanation of how and why policies change over time. Furthermore, this article contributes to the broader debate on policy change in housing studies by highlighting the importance of understanding the ideational landscape in which policies are formed, building on the work of other scholars interested in the role that discourse and ideas play in driving forward change.