Abstract: E-government evolution has been described as a government's internal process of digital development which eventually transforms its ability to respond to the public. As time goes by and these promised benefits have yet to fully materialize, civic technology – online tools that aim to achieve improved online interaction between governments and the public – is sometimes placed in the gap. This study provides findings from 38 interviews across five US municipal civic technology implementations, answering the question of whether US cities which have adopted civic technology tools enjoy improved two-way interactions between governments and the public, and also whether an "interaction-first" approach to government digitization appears to spur additional e-government development. By selecting five very different tool implementations, the research design employs Mill's Method of Difference to isolate commonalities springing simply from a municipality's implementation of a civic technology tool. Interviews reveal a range of common effects beyond the simple improvement of the service-delivery experience.
Acknowledgement: Research and initial writing were conducted while the author served as a US Civic Tech researcher for mySociety with support from a grant from Microsoft Technology and Civic Engagement. A preliminary version of this article was presented at the Chicago Community Trust's On the Table event in May 2017. The author would like to thank Rebecca Rumbul and Christopher Wilson for helpful early feedback and all of the interviewees across the five cities who gave so generously of their time and thoughts.
Introduction
Can civic technology projects help governments use technology to fundamentally improve the public's experience of government? This question, coming decades after governments' first steps towards embracing e-government, speaks to the failure of most administrations to achieve e-government's early promise to use technology to make government fully accessible and democratically responsive. However, it also speaks to the popularity of this goal. Many people inside and outside of government still share the common aspirations of e-government, particularly with regard to the aim of achieving improved two-way interactions between government and the public.
Into this space of unfulfilled interest, independent groups both outside and in partnership with government have begun launching civic technology tools: technology "platforms designed to amplify citizen voices in order to improve public service delivery." (Peixoto & Sifry 2017) While these tools do not achieve the full spectrum of e-government benefits, since they are focussed explicitly on the point of public interaction, and not the internal digitization or internal digital integration that governments are also envisioned to achieve, they do nonetheless serve to provide that interactive aspect of e-government which benefits the public most directly.
Notes ():
1: Peixoto, T. & Sifry, M. (2017) Civic Tech: New Solutions and Persisting Challenges. In T. Peixoto & M. Sifry, (Eds.), Civic Tech in the Global South (pp. 25-46). Washington, DC: International Bank for Reconstruction & Development / The World Bank.
We can think of this approach as reordering the process of government digitization to focus on the public-interaction elements first – or, from the public's perspective, "skipping ahead to the good part" of e-government. This study evaluates whether a selected set of civic technology implementations in five US cities have transformed government service delivery in their areas of service provision. By providing the findings from thirty-eight interviews with government staff and members of the public, this study seeks to better understand the implementation of civic technology and its relationship to the achievement of fully established e-government.
An Alternate Path to Achieving the Benefits of E-Government
While people in cities around the world use smart phones to obtain desired goods and services, many of their local governments are still trying to figure out how to provide the services of government online. In the US, the challenge local governments faced in achieving the transition to full digital accessibility was seriously complicated by the financial impact of the 2007-2009 Great Recession. At the same time President-elect Barack Obama announced he would "use all available technologies and methods to open up the federal government, creating a new level of transparency...and giving Americans the chance to participate in government deliberations and decision-making," O'Reilly 2009), governments across the country discovered they would have to make due with significantly lower levels of revenue. This new financial reality, together with the traditional challenge of modifying existing regulations and practices for the digital age, slowed progress towards having Americans broadly experience "Government 2.0."
Notes ():
2: O'Reilly, T. (2009, August 10) Gov 2.0: The Promise of Innovation. Forbes. https://www.forbes.com/2009/08/10/government-internet-software-technology-breakthroughs-oreilly.html#736714b73b7b
This gap between aspiration and reality has been particularly visible at the local government level, where governments are charged with providing services (policing, education, transportation, utilities, and sanitation) that affect all Americans' daily lives. Meanwhile, local government's slow pace towards full online presence produced a clear tension because the promises associated with achieving "e-government" were manifold. As US Government Accounting Office Associate Director David McClure described it:
Electronic government refers to government's use of technology…to enhance the access to and delivery of government information and service to citizens, business partners, employees, other agencies, and government entities. It has the potential to help build better relationships between government and the public by making interaction with citizens smoother, easier, and more efficient. Indeed, government agencies report using electronic commerce to improve core business operations and deliver information and services faster, cheaper, and to wider groups of customers. (Layne & Lee 2001)
Notes ():
3: Layne, K. & Lee, J. (2001) Developing fully functional E-government: A four stage model. Government Information Quarterly, 18(2), 122-136.
To the extent that local government workers may have been sorry to miss the benefits of the greater ease and efficiency inherent in digitized services, surely no smaller portion of disappointment lay with citizens who desired greater access to their governments. Those outside of government who wanted increased access to government functions and decisions were waiting for a revolution that would alter not just their ease of access to government materials, but also the country's very orientation to government. Described in the e-government movement's characteristically optimistic terms, Darryl West (2004) documented the expectation that e-government would produce the "transformation of service delivery and citizen attitudes," achieved through a flattening of the power dynamic between governmental service providers and citizen recipients:
Notes ():
4: West, D. M. (2004). E-government and the transformation of service delivery and citizen attitudes. Public administration review, 64(1), 15-27.
The nonhierarchical character of internet delivery frees citizens to seek information at their own convenience, not just when a government office is open. The interactive aspects of e-government allow both citizens and bureaucrats to send and receive information. By facilitating two-way interaction, electronic governance has been hailed as a way to improve service delivery and responsiveness to citizens, in the long run generating greater public confidence in government (West 2004).
Notes ():
5: West, D. M. (2004). E-government and the transformation of service delivery and citizen attitudes. Public administration review, 64(1), 15-27.
Indeed, scholars made many optimistic pronouncements in the early years of the 21st century about the ways that e-government would transform the very nature of governance. "eGovernment," a European research consortium declared, "is to become a meaningful agent of transformation embedded in the culture of the public sector. Its potential goes far beyond early achievements, enabling qualitative gains in work processes, results and efficiency. If implemented properly, it will help develop and consolidate principles of good governance such as democratization, coherence, effectiveness, transparency and accountability" (Leitner 2003). Through the power of increased public input into government service provision, theorists augured "a consumer-led revolution bringing with it more efficient government, more transparent ways of doing business with the different branches of government; a two-way path of consultation and collaboration; a new level of accountability for elected and unelected officials; and more open and responsive politics" (Silcock 2001).
Notes ():
6: Leitner, C. (2003). eGovernment in Europe: the state of affairs. EPIAScope,2003(3), 37-39.
7: Silcock, R. (2001). What is e-government. Parliamentary affairs, 54(1), 88-101.
About a decade later, researchers began publishing less optimistic observations. Donald Norris and Christopher Reddick (Norris 2010, Norris & Reddick 2012), for example, argued that there was very little change in the nature of services provided since the beginning of the 21st century. Instead, they argue that e-government still mainly delivers information and existing services without transforming the public-government power balance. Furthermore, they claim that theorists have persistently been over-optimistic in arguments about the transformative nature of technology because they fail to ground their work in existing studies of public administration. (They note that perspectives on e-government as "transformational" miss the routine public administration observation that government change is almost always gradual and incremental, not revolutionary.)
Notes ():
8: Norris, D. F. (2010). E-Government 2020: Plus ça change, plus c'est la meme chose. Public Administration Review, 70(s1), s180-s181.
9: Norris, D. F., & Reddick, C. G. (2013). Local e-government in the United States: Transformation or incremental change?. Public Administration Review, 73(1), 165-175.
Governments' failure to "transform" either service delivery or public confidence did not mean that the belief in the power of technology to achieve that transformation faded away, however. Instead, it created a clear opportunity for people outside of government to step in and try to effect the transformation themselves. Particularly desirable to non-government individuals and groups was the goal of achieving the promised e-government function of "two-way interaction…as a way to improve service delivery and responsiveness to citizens." This aspect of e-government, sometimes identified as "e-participation" (Macintosh 2004) – or more aspirationally, as "we-government," Dennis Linders' (2012) term for digitally-enabled citizens' co-production of government decisions and services – offers the most direct benefit to the public through giving them additional opportunities to provide input into government processes. Meanwhile, this specific e-government function of increased two-way interaction was also the most lagging element of all the aspects of local e-government implementation. The International City/County Managers' Association (ICMA) surveyed US local governments in 2011 and found that while local governments had greatly increased the amount of information they had published on their websites, online services involving true back-and-forth interaction between government employees and citizens were present only in a very small percentage of local governments (Norris & Reddick 2012).
Notes ():
10: Macintosh, A. (2004, January). Characterizing e-participation in policy-making. In System Sciences, 2004. Proceedings of the 37th Annual Hawaii International Conference on (pp. 10-pp). IEEE.
11: Linders, D. (2012). From e-government to we-government: Defining a typology for citizen coproduction in the age of social media. Government Information Quarterly, 29(4), 446-454.
12: Norris, D. F., & Reddick, C. G. (2013). Local e-government in the United States: Transformation or incremental change?. Public Administration Review, 73(1), 165-175.
Into this space of frustrated heightened expectations stepped a group of civically-minded computer programmers. Rebecca Rumbul (2015) described the origins of civic technology in terms of a movement of people developing online tools to improve two-way citizen-government interactions:
Notes ():
13: Rumbul, R. (2015). Who Benefits from Public Technology? Demographic and public attitudes research into the users of civic technologies. London, UK: mySociety. Retrieved August 31, 2018 from https://research.mysociety.org/media/outputs/demographics-report.pdf.
The rise of civic technology in this new millennium has been organic and profound. It has not been led by politicians or corporations, nor by powerful knowledge-rich institutions or NGO's, but by individuals and loosely constituted groups with specific digital expertise and an interest in getting things done. Such individuals are not normally considered to be on the cutting edge of political and practical behaviour change. Across the world, small pockets of coders and developers have independently, and occasionally with a little peer support, created a range of online platforms to help citizens like themselves get government working for them in one small way or another (Rumbul 2015).
Notes ():
14: Rumbul, R. (2015). Who Benefits from Public Technology? Demographic and public attitudes research into the users of civic technologies. London, UK: mySociety. Retrieved August 31, 2018 from https://research.mysociety.org/media/outputs/demographics-report.pdf.
The fact that independent individuals and groups began developing civic technology tools outside of government seems to solve at least part of the challenge facing local governments: the problem of how to pay for new, exploratory technology under conditions of scarcity. Many civic technology projects are developed by volunteers or externally funded, requiring relatively little financial commitment from governments (Wood 2016). Governments can potentially use civic technology created outside of government to achieve the collective, ultimate goal of implementing important aspects of e-government: transformed citizen-facing services, and hopefully the increased public confidence that was said to come with them.
Notes ():
15: Wood, C. (2016, August 16). What is Civic Tech? Government Technology. Retrieved August 31, 2018, fromhttp://www.govtech.com/civic/What-is-Civic-Tech.html).
Meanwhile, though governments benefit from not having to fund the new technology projects, they do not necessarily benefit from the way that external civic technology projects work, setting problem agendas that may not match their preferences. Rumbul's description of the civic technologist as hoping to "get government working for them in one small way or another" highlights the fact that civic technology projects respond to problems in existing government service delivery. To merit public attention (and attract volunteer coders), citizen-led civic technology projects often focus on problems that affect citizens in relatively significant ways. However, simply adding an externally-developed online communication tool does not necessarily get at the root of these problems. A range of obstacles – from government staff opposition, to perceived political risk to leaders, to countervailing pressure from other public groups – can impede the ability of a civic technology project to have its intended effect.
Indeed, if governments are not full and willing partners, citizen-led implementations of two-way e-government functions demonstrate unsatisfying rates of success. In her study of civic technology tools aimed at enhancing public access to government information in Latin America, Rumbul (2016) documents the regular obstruction of the tools' function through bureaucratic countermeasures. Similarly, Tiago Peixoto and Jonathan Fox (2016)evaluate the effectiveness of 23 civic technology tools on improving governmental responsiveness and find that where government institutions are not actively seeking out online interaction with the public, the tools by themselves do not increase government responsiveness. Scholars documenting the effects of civic technology have described this problem as the gap between "transparency" and "accountability." Katharina Welle, Jennifer Williams and Joseph Pearce (2016) summed up the findings of their research on service delivery complaints by noting: "[P]utting the user's reporting preferences at the centre of the ICT [information and communication technology] design may be missing the point unless the wider design supports a more responsive service delivery model." Even the best civic technology platform will not increase responsiveness if a government cannot or does not want to respond.
Notes ():
16: Rumbul, R. (2016). Developing Transparency through Digital Means? Examining Institutional Responses to Civic Technology in Latin America. Journal of e-Democracy, 8(3), 12-31.
17: Peixoto, T. & Fox, J. (2016) When Does ICT-Enabled Citizen Voice Lead to Government Responsiveness? IDS Bulletin, 47(1), 23-40.
18: Welle, K., Williams, J. & Pearce, J. (2016) ICTs Help Citizens Voice Concerns over Water – Or Do They? IDS Bulletin, 47(1), 41-54.
On a more positive note, all of these studies also suggest that under some conditions, civic technology genuinely does enhance two-way interaction between citizens and governments. While Peixoto and Fox (2016) observe the failure of many civic technology projects to achieve the transformation of citizen-government interaction, they simultaneously find that civic technology initiatives which are either led by governments or just achieved in partnership with governments do attain the goal of improving two-way communication. This suggests that in situations where governmental actors have an interest in increasing their public responsiveness, they can take advantage of technology created outside of government to improve two-way interactions. With sufficient internal interest, they may even overcome bureaucratic inertia and funding challenges to create those avenues themselves.
Notes ():
19: Peixoto, T. & Fox, J. (2016) When Does ICT-Enabled Citizen Voice Lead to Government Responsiveness? IDS Bulletin, 47(1), 23-40.
A similar dynamic appears to be at play in the case of US local governments. While the overwhelming majority of US local governments lack tools for significant two-way interaction, some do make use of online, two-way communication tools for the specific purpose of augmenting a citizen-facing government service. In a number of these cases, these tools have been developed by groups outside of government, provided at a great discount or through non-governmental funding. The achievement of the full promise of e-government, with the opportunity "to build better relationships between government and the public by making interaction with citizens smoother, easier, and more efficient" is something that civic technology can indeed help facilitate when governments choose to embrace it.
However, adopting ad hoc, externally-developed tools to perform particular e-government functions represents a very different way to accomplish e-government than the rational, internally-driven path suggested by Obama's description. In order to establish a way to support sustained online interaction, it seems logical to first require that governments digitize – providing necessary records available in digital form – and that they sufficiently integrate their data so that it is accessible to everyone within the organization who needs it. Indeed, many e-government scholars suggest that the development of new modes of government through technology occurs through a gradual, progressive evolution, in which publicly interactive systems are the final stage (Feeney and Welch 2012). In many e-government models, the final stage of evolution includes the realization of both internal and external goals: a system which is fully integrated internally, allowing seamless interaction between all parts of government, and interactive externally, allowing the public to participate in as many processes as possible.
Notes ():
20: Feeney, M. K., & Welch, E. W. (2012). Electronic participation technologies and perceived outcomes for local government managers. Public Management Review, 14(6), 815-833.
The concept of a rational order for e-government development which begins internally and ends externally can be observed repeatedly across the literature. Tomasz Janowski (2015) analyzed twenty-two years of "digital government" articles to synthesize the essence of these hypothesized trajectories. First comes the digitization of government, including the representation of all previously analog information in digital formats. A second stage features full digital integration of internal functions. A third stage – which is where government launches two-way interactive tools– digitizes government-public interactions, and in the final stage government and the public use these new digital relationships to improve policy.
Notes ():
21: Janowski, T. (2015). Digital government evolution: From transformation to contextualization. Government Information Quarterly, 32(3), 221-236.
When a non-government actor launches a civic technology project, meanwhile, those projects – which are fundamentally about including the public – ultimately always aim at the last two of these stages. While this has the attraction of "skipping ahead to the good part" and moving directly to the desired place of greater citizen-government interactivity, it can lead to functional challenges when the first two stages have not yet been realized. Kirsten Boehner and Carl Disalvo (2016)studied the implementation of civic technology in the US city of Atlanta and found that a common concern across both governmental and non-governmental participants was that they lacked data, or lacked access to relevant data – an issue that would not have come up if e-government had developed in line with the theorized trajectory, and had they begun their projects in the wake of full governmental digitization. On the other hand, civic technology projects which prioritize citizen-government interaction may, rather than resulting from internal government tranformation, end up sparking internal transformation to support the project. While we know this is unlikely to happen in cases where the government does not embrace the technology project, we can also hypothesize that it might occur when government is supportive of its aims.
Notes ():
22: Boehner, K. & DiSalvo, C. (2016). Data, Design and Civics: An Exploratory Study of Civic Tech. San Jose, CA: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, 2970-2981.
This project sought to discover the impact of civic technology projects on US local governments under conditions where full e-government development has yet to be realized, but many people still have interest in achieving its aims. Preliminarily, the research sought to explore how municipalities hosting civic technology projects experienced those projects, both from the government and from the citizen side. In other words, how do projects change perceptions of government service delivery? Secondarily, the research sought to discover what happened in situations where civic technology was effectively stepping in to replace a yet-incomplete process of e-government evolution. What happens when you simply "skip ahead to the good part" and start a government project government-citizen "interaction-side" first? In other words, when the start of the e-government improvement process lies at the public endpoint, and not in internal digitization or internal digital integration, do the public and government still benefit? Do these projects by themselves manage to change the nature of government service delivery to become more responsive? Or do the implementations point to a more complicated role for civic technology in governments' path to achieving comprehensive e-government?
Research Method
To explore the impact of government-led civic technology projects, the research consisted of structured interviews with government employees and members of the public from five US cities. Interviews were conducted with a standardized set of internal and external stakeholders and aimed to assess their perception of the city's identified civic technology tool and how it affected service delivery and public sentiment. Interview transcripts were coded using RQDA to identify clusters of common sentiment, then analyzed for frequency, similarities and differences.
The five cases included in this study fulfilled a list of criteria that aimed to produce comparable information across different settings. The first criterion was that a department providing a city service had launched a civic technology tool – which, for the purpose of this study, means a named online software function, located on a single website or mobile app, which helps to improve the public's experience through increasing opportunities to participate in governmental decision-making or service delivery. To ensure the project's authentic support of government (thereby avoiding governments with no clear interest in improving service delivery), selections were restricted to online sites which, even if developed outside of government, were now hosted or owned by the governments themselves.
The selected cases also met a set of specific benchmarks. All cases are online tools on a city-owned website which incorporate avenues for resident feedback. All cases feature tools which accomplish a single function and are managed by a single department, rather than being multi-departmental or multi-functional tools. This criterion, which was simply put in place to ensure clarity regarding project goals and the entities responsible for its management, meant excluding significant government-led technology projects like open data portals, which consist of multiple components and serve many functions and audiences. In line with the definition of "civic technology" chosen for the purpose of this study, sites were also chosen on the basis of having a tool which enabled or furthered two-way interaction between the governmental department and the public. Finally, in order to ensure that tools were viewed with an equivalent level of commitment, the civic technology tools at all study sites had been in place for at least one year.
While the municipal civic technology cases were similar in many respects, they were different in one fundamental way: their substantive subject matter. This measure was taken in order to ensure that any results which emerged didn't fundamentally stem from the kind of service that was provided, but rather from the fact that they have all implemented a civic technology tool. According to Mill's Method of Difference, if interviews about five different kinds of public services produce similarities, those similarities likely stem from the single element they share in common: the civic technology tool.
On the basis of these criteria, the following five cases were selected:
- SpeakUpAustin (www.speakupaustin.org), in Austin, Texas, is a public forum tool which allows Austin residents both to respond to questions and surveys from the city, as well as to freely offer ideas about ways to make Austin a better place to live. It was developed (and initially provided for free) by local market research company Sentient Services, first launched in 2008, and is managed by Austin's Public Information Office.
- LargeLots (www.largelots.org), in Chicago, Illinois, is a tool developed to support a new city program where residents of some of Chicago's most disadvantaged neighborhoods could purchase nearby vacant lots for $1. It was developed by local civic technology company DataMade with the support of community development organization LISC, launched in 2014, and is managed by Chicago's Department of Zoning and Land Use Planning.
- RecordTrac (records.oaklandnet.com), in Oakland, California, is a tool for making requests to the city under California's Public Record Act (PRA). RecordTrac also serves as a platform for delivering responses for these requests and a publicly-searchable database of requests and responses. It was developed by fellows from civic technology organization Code for America, launched in 2013 and is managed by Oakland's Communications Department.
- DC 311 (311.dc.gov), in Washington, DC, is a tool which serves as part of a comprehensive intake platform for residents wanting to obtain non-emergency city services. Initially conceived of as a way to receive reports of city maintenance problems like potholes and overflowing garbage cans, DC 311 now provides and receives information across a wide variety of city services. The first version of DC 311's online component was launched in collaboration with civic technology company SeeClickFix in 2011. The tool is now managed by the city's Office of Unified Communications.
- Office of Professional Accountability (OPA) Complaint Tracker (www.seattle.gov/opa/file-a-complaint-about-the-seattle-police), in Seattle, Washington, is a website which allows Seattle residents to register complaints about Seattle police practice (including anonymously) and then track the status of that complaint. The site also provides closed case summaries. The tool was developed by the city of Seattle's consolidated Information Technology Department, launched in 2015 and is managed by the City of Seattle's Office of Professional Accountability.
Qualitative rather than quantitative methods were selected for this project because of the exploratory nature of the research question and the desire to capture and illuminate new patterns. Interview methods allowed for specifically targeting the limited number of respondents knowledgeable about each implementation inside government and identifiable civic tech tool users outside of government. Interviews were conducted between January and May of 2016 and a total of 38 respondents were interviewed from the five study sites. Interviews included project managers, department heads, internal users, external users, and the tool's software developer from nearly every city. Respondents were interviewed using a standard group of questions which aimed to determine, from the interviewee's perspective, the impact of the tool's introduction on the delivery of the relevant public service. Interviewees were asked how the introduction of these additional sources of public input affected the departments where they've been introduced, whether the department had noted internal benefits (such as increased efficiency, or more effective service provision), and whether external stakeholders perceived increased effectiveness or had changed their feelings about the service.
Since the focus of this research was to explore trends in government and public experience across study sites, it was preferable to find ways to look across all cases rather than to evaluate cases separately. Interview statements were coded by aspect of service-delivery process (the interactive software tool, enabling or related government policy, government service-delivery) and affective valence (does the respondent describe an aspect of the tool's implementation as problematic, beneficial, or neutral) and then clustered to identify similarities and differences across the cases (Miles, Huberman and Saldana 2013).
Notes ():
23: Miles, M. B., & Huberman, A. M. (1994). Qualitative data analysis: An expanded sourcebook. Sage.
Findings
Clusters of observations from the interviews fell into two general categories: observations relating to the ways that people may have benefited from the civic technology tool, and observations relating to the challenges that the tool created or highlighted. The following table summarizes commonalities from interviews across all study sites about the changes interviewees observed resulting from the tool's implementation, separated to indicate whether the observed change affected people on the government side or on the citizen side of the relationship.
Table 1: Clustered observations on the effects of civic technology implementations
| New Benefits | New Challenges | |
| For Citizens |
|
|
| For government |
|
|
The following sections review each of the above-listed benefits and challenges created by civic technology implementations across all five sites. The sections situate each of these clusters of observation in the language of the respondents themselves, in order to provide context and clarity about respondents' observations.
First, tools did produce a number of positive points of feedback from and about external users, who described an increased ease of access and efficiency.
Users of LargeLots.org found significant improvements over the existing process, and appreciated the role of the tool in achieving service goals.
I think it's a great tool – it's great for me to have a website to point to. It really helps having it there. I know it's really easy to navigate, applicants find the applications relatively easy to complete, there's a good FAQ that's been built over time. - Partnering community organization, LargeLots
DC 311 staff reported that they were able to reduce the number of abandoned requests, likely creating more satisfied users, by adding online functions to their existing phone-based maintenance request service.
Was there a big change when we went online? It did open up the venue for more requests to come in. Our abandonment rate has gone down because people didn't have to wait online - Program manager, DC 311
RecordTrac, which has some dedicated users in the journalism community, saw increased dimensions of services now regularly available online. The regular service function of requesting government records was also perceived to be simpler.
RecordTrac is also useful because you can check the timeliness of past requests, or see how long someone was not responsive. It becomes a de facto public log of public record requests. With other agencies you have to request the log.
Before RecordTrac [the records request process] was pretty confusing. If you knew the Public Records Act, then you could do it [otherwise, no.]
– External user, RecordTrac
In addition to appreciating the advantages of digital tools, respondents pointed to the way that civic technology development processes empowered the public. In three of the studied cases, relevant policy development occurred alongside the development of the civic technology tool. The intertwining of these processes meant that information-sharing possibilities described by developers or uncovered during technology development ultimately shaped policy outcomes. In other words, official city policy was fundamentally altered by the design options developed by civic technologists.
The development of LargeLots, for example, represented a policy development process where a substantive policy change could not have been easily effected without digital technology. In this case, several community organizations worked with the City of Chicago's planning and zoning department in order to develop a better way for the city to sell vacant city-owned lots in low-income neighborhoods. The policy development process included technologists, who showed how creating a comprehensive online information and application process made the program easier for both city staff and community applicants.
[The technology organizer] said we've developed a website to see what the city owns. I looked at it and said, "this is great! Can I send it to people?" and he said, "Sure."
And then it just exploded.
When we started with a plan we talked about keeping it to a smaller area, so that's what [the website] had in the map. I said, "What about the rest of it?" and so he's populated it with the whole Green Healthy Neighborhood area…Once you receive green healthy neighborhood applications that are such a positive response—400 applications for 500 lots just in one month and thousands of phone calls – I realized there is interest in expanding so we launched again in the summer in East Garfield Park. Many aldermen asked for their areas to be included.
Program manager, LargeLots
The RecordTrac tool development process also happened in concert with a revamp of city policy. In this case, technologists and policymakers worked together to optimize the process for producing requested public records.
We also got involved in the policy side-- this project was interesting because an internal working group was working on updating the administrative policy and instructions on delivering public records. I got to work with this group, and so both policy and technology got to affect each other through this process.
For example, we have ten days in California to deliver a response [to a public records request.] So we got to work on the question: when is a request received? When it is opened? What if it's a Saturday? Is it opened the Monday after it's received? When something is time-stamped, it's particularly hard.
In addition to having the city update its internal procedures, it needed to think about the technology. We had to balance the right of people to see requests - because that would be public for the first time as well. But we wanted to balance that against the fear that we would chill speech through that policy. We decided to leave requesters anonymous because we didn't want to have a chilling effect on speech. If the experience is hard for people, they might want to avoid using it - and bypass the system or not make requests.
Developer, RecordTrac
An interesting feature which emerged from discussions of the co-production of technology and policy is that policies developed in concert with technology tools appear simplified, with fewer restrictions and requirements. This may be due to developers' greater focus on user experience than legal issues. Civic technology, with its explicit focus on improving government-community communication, may frame the question of "policy effectiveness" differently than either governmental or community representatives would.
[Regarding the decision to make all requests public on the website,] prior to [the project], I didn't know what FOIA was, so I didn't come in with preconceptions about it. I thought of it from the perspective of being the simplest most obvious way to put information out. Everything else is pretty much out in the open on the website, you can see the history of who's been assigned the request, and notes. I didn't want to deal with different views for different people, and I didn't want to deal with logins. I wanted to do the most simple design possible, so that's why everything is public.
Developer, RecordTrac
One major issue in the application for vacant city property in Chicago was a particularly complicated economic disclosure form (EDS). Over the course of the tool development process, software developers and community members worked with the city to reduce its impact on the application process.
We had a lot of feedback on how the application went. We worked with [the program manager] on ways to delay the pains of the EDS… The EDS is a set of disclosures for ethics purposes, it's just a bad form. Citizens and city employees get the same form [for all economic transactions.] We wanted to figure out if we could auto-generate PDF from the website, but instead got them to shift it to further along in the application process. People don't have to deal with it until they're through the first hurdle.
The general strategy is just to push further down the line if we can't change things.
Developer, LargeLots
Meanwhile, the OPA Complaint Tracker represents an example of how a reform-minded director can take advantage of the technology development process to empower the public without creating formal new policy. Developed as part of a broader push for transparency under a new departmental director (who was fresh from working as a long-time Community Ombudsman in a different city), the OPA tracker changed rules about how much complainants needed to disclose about themselves when submitting complaints. While the Seattle police department had accepted e-mailed complaints before their new website, the development of a dedicated website made it technically possible to establish an anonymous webform for submitting complaints, adding a new level of protection for people worried about facing repercussions from submitting a police complaint.
OPA is committed to as much transparency as possible, figuring out what works.…[He] wanted the OPA to be as independent as possible - moved it physically out of the SPD, and then moved the website too…. The anonymous complaint form was launched in March 2015. The complaint tracker was launched in May 2015… He has been vigilant about protecting OPA's independence.
– Program manager, OPA Complaint Tracker
On the government side, program staff also experienced the tools positively, finding that they helped improve the way government services functioned. Government staff described this in terms of a range of ways that the tool improved the efficiency and effectiveness of their work.
Chicago LargeLots was able to develop a significantly more efficient process for service delivery as a result of the civic technology implementation.
We did the first sales in eight months which is really fast for the city. It includes from city council approval to disposition that's just eight weeks. The real estate folks say that's very fast. The ANLAP process [a program LargeLots replaces] requires a year to 18 months. And that's only if you can get a letter [from a city alderman].
Program manager, LargeLots
Users of SpeakUpAustin identified several specific benefits from their online interactive forum, including the ability to get feedback from people who wouldn't come to in-person meetings:
SpeakUp Austin is an added tool that helps us to connect - I've used it when I'm working with a client department that needs input. They're having one or two public meetings, but they also want a broader perspective.
For example, the Ethics Review Commission wanted public input and I put their proposal on SpeakUpAustin even though it was a complicated issue. It got several hundred responses - government nerds responded to that.
It's been effective when people don't have time to attend the meeting. The question about library hours, too - the response was very good on that, people love libraries. Where department or policymakers want broader input we would use it - for example, a decision on a budget, or on something that affects the community as a whole like library hours. [It's good for] the broader-affecting projects, to get the pulse of the community.
Internal user, SpeakUpAustin
For Seattle's OPA, the Complaint Tracker emerged as an aspect of the department's digitization of its complaint management process. The change to using a comprehensive digital case management system significantly altered the way that cases could be tracked both inside and outside of government.
The way we manage cases is different. Our contact log used to be an Excel spreadsheet, and the auditor would ...examine it at the end of the month. Now that we're processing in real time, the auditor can also review in real time… Media are now able to ask for more specific cases as result of information coming out more regularly. The standing media request for all "sustained" findings has gone away, and now asking for specific cases. In that respect, it's better.
Program manager, OPA Complaint Tracker
Oakland government staff using RecordTrac saw similar benefits from simplifying information access and putting it online.
This [requested information on RecordTrac] can be information that's useful to us internally too – so it makes it easier for us to access. At one point - with the economy crashing and people were getting hired - there were lots of requests for public records on salaries. I would ask managers to run this [query] 12 times a year. And then an administrator allowed us to put it online - we haven't gotten a single request for that record since.
Department head, RecordTrac
In addition to providing users with direct benefits, civic technology tools demonstrated to government staff how additional digitization and integration processes could even further improve the quality of their services.
Internal users of one tool (DC 311) found their tool became more functional and flexible when the city adopted a new digital content management system (CMS). Other internal DC 311 users described an interest in further internal digitization in order to make the tool even more effective.
Middle part of last year we shifted to Open 311 customer service system leveraged by [the new CMS]…It's much better because we don't have to have our own separate system and link it. With the original 311 system there were a lot of third party integrations which connected in their own way and bypassed our work order management system. This was a big problem whenever there was any change. Now that [the new CMS] is in place, it doesn't need the third party apps and it's easier to use.
Internal user, DC 311
Across cities with 311, half the time is spent figuring out where the [object of the request] is. Locating the thing can often be pretty problematic, if you don't have a great address validation system, location validation…The challenge is figuring out essentially what [requesters are] looking for – it's an Easter egg hunt. The 311 system is not able to tag the exact location. It needs to be like that, or we often can't find the sidewalk defect exactly. We get a block, that's it. Same could be said for potholes, a lot of things. It's like a 311 service request last mile problem – it's the hardest part. We can get to the block – but can we get to the asset? The tech of 311 helps – but in order for the 311 system to be leveraged in the best way, it's important that we maintain an asset inventory that helps us not waste administrative time connecting the dots.
Internal user, DC 311
Users of SpeakUpAustin described the value of being able to present multiple kinds of city information to the public through a single platform. One user expressed an interest in using the tool to capture in-person interaction as well in order to unify data collection.
On SpeakUpAustin there were around 150 responses on the discussion. It was very passionate. [The platform also] lets people see the documents that were involved so they had detailed information. You get back-up, you don't have to drag and hunt, everyone has the same platform for sharing information - the Council, the public and staff all have the same information."
Internal user, SpeakUpAustin
We'd like to add more canvassing…Taking canvassing a step further, we'd like to go out with survey iPads and canvassers. That would be great to capture data in the same place. [In town hall meetings, it] would be great to have laptops there with a survey. Engaging them and giving users resources but also integrating the information.
Internal user, SpeakUpAustin
Meanwhile, in the case of two tools (RecordTrac and LargeLots), the tool preceded – and may in fact have motivated – more comprehensive record digitization processes. Managers described ongoing governmental efforts to improve the digital management of records which are delivered to the public through the tool, while users called for additional integration as well.
The digital front door project and RecordTrac – we look at how we can make it a better part of our process. The IT department is working to bring in a document management system now…I think if we had a document management system that would help a lot.
Department head, RecordTrac
[We found] our database that holds city [data] on property needs updating, so we're doing updating by community area because you want to keep the program going.
Program manager, LargeLots
Despite the fact that some users experienced the tools as easing their access to services, this was not a universal perception. One routine area of dissatisfaction related to the communication around tool-enabled services. Though the tools create a sense of improvement for some, they also clearly create frustration for users if adequate resources are not devoted to communicating about the service. For example, when managers for DC 311 conducted focus groups with users, they found:
Users didn't always know who the responding agencies were. They also needed more information about the length of response time which we call SLAs - service level agreements - so they'd understand more about the timing of response [from agencies].
Program manager, DC 311
Similar issues in translating "bureaucratese" were mentioned by other users:
Issues started to come up where residents didn't understand the jargon – 'inspection complete' – but didn't let you know whether repair would happen or when. Sometimes people saw their request was "closed" but had no info about why. They didn't know if it was because the repair was made, or they couldn't find it…For someone who works inside of government, it's hard – and it's even harder to find out for others. We work with older adults and when you do that, you try to communicate simply and clearly.
Internal user, DC 311
This issue was echoed by other respondents.
Governments are focused on speaking legally, not clearly, so we've had to focus on that issue. The usability perspective is lacking.
Developer, LargeLots
In the absence of funding additional outreach for people to connect with the new civic technology tool, there was a sense that the tool was going to have to do what it could with the people who could find it or who would be able to use it on their own.
With limited resources you are catering to an audience that's already connected to online sources…The digital divide is a huge problem in Chicago. Exclusively online makes it easier for [digitally-savvy non-residents] to apply as well. This is my city – it still requires organizations to be involved. I do feel without deliberate intervention and strategy it reinforces existing patterns of access.
Technology organizer, LargeLots
Government staff shared the public's frustration with the level of resources available for communicating about and through the tool. Many government respondents felt that cities should provide more communications and outreach resources.
We're not as responsive as I would like to be to user-generated ideas. It takes tracking down the manager who is responsible for that area and working out a response. It's a part time job in itself.
Internal user, SpeakUpAustin
...
I would love to have someone on the staff just doing digital engagement, aggregating responses and making sure that people are responded to and so on. You know, all that stuff which takes time and attention. What's frustrating is that the demand far outstrips the resources that the budgets permit.
Department head, SpeakUpAustin
...
I think it would be a new full-time job to manage the social media and website updating. But we are already inundated - we don't have resources to handle more.
Program manager, OPA Complaint Tracker
...
There's no end to the needs people have, and there's never enough staff.
Department head, RecordTrac
Beyond the need to add further outreach and communication resources, internal users had lists of improvements they wanted for the tool itself.
We had issues in the field. It was hard to get the app to work fast enough when we had lots of things to report. There were multiple steps to report, and geolocation didn't always work. It worked better with the new app, but still slow and meticulous work to go through neighborhood. If it could have been faster, we could have covered more ground.
Internal user, DC 311
...
There's no way to search the closed cases from our site. That's something that at the minimum we want to do moving forward.
Program manager, OPA Complaint Tracker
...
Have there been any frustrations? I wish we had a better analytics-board. We want to know if we could track the time it takes to fulfill requests. The number of requests doesn't really get at the impact. Some of the requests are very small, while others are huge and take a lot of time to fill.
Department head, RecordTrac
...
In 2012 we didn't care as much about geography but now we have a district-based City Council and they really care about geography. Now they want to know about how responses are coming in by district.
Program manager, SpeakUpAustin
While all sites recognized the need for ongoing work on the tools, there were divergent responses to the question of how serious a problem this was. Where respondents assessed it as more serious, this stemmed from the problem of failing to have a regular budgetary line-item or source of revenue for maintaining and improving the tool and related service. Several of the tools, including SpeakUpAustin, and RecordTrac, were developed in an "agile" fashion and initially enjoyed frequent improvements, but this was only possible because the technologists were temporarily offering their services very cheaply or for free.
From launch, it was agile development - we created the idea of a posting forum first, and then allowed surveys too, services in order to allow the longitudinal tracking of individuals, and then we got requests for forums and user groups. It was a collaborative process.
Developer, SpeakUpAustin
When I came on board [three months after the prototype release], the staff were dealing with the need for fixes. I worked on a product development process for one or one-and-a-half years. When I arrived there was a stack of emails with suggestions. We still get a lot of feedback about how to make it work better.
Program manager, RecordTrac
Since the point of initial development, however, three of the tools have had to go without improvements despite user interest because the budgets weren't supporting it. In Austin, community engagement is paid for on a per-engagement basis by individual departments. The SpeakUpAustin tool itself is paid for by subscription at a reasonable cost, but the price does not include software improvements.
We don't have an allocated budget for the Town Halls we design for them. That's why there's a great disparity in what our engagements look like... Each department works according to their interest, need, and ability to pay.
Department head, SpeakUpAustin
Challenging procurement policy – particularly for small, ongoing projects – was cited as main difficulty in making large-scale improvements to RecordTrac.
Open source - there's a lot of challenges to maintain it. Our contracting policies are structured around multimillion-dollar deals, not smaller ongoing maintenance. [We lack] contracting structures that allow for minimal ongoing work.
Program manager, RecordTrac
Meanwhile, in two cases – DC 311 and LargeLots – respondents did not cite resource constraints as a primary problem. Of the examined cases, DC 311 showed the most significant commitment to maintenance and improvement:
What we've learned over years [was that the old] platform wasn't flexible for where we were going. We wanted the ability to tie other agency systems to our DC 311 platform. We wanted the ability to tie Transportation work order system for example and then expand that.
We revamped the online site - that was a big investment.
We invested in a DC-owned mobile app that provided the flexibility that we wanted – to change when we wanted to change, we want to do surveys, make other changes. Now we have a call center, a revamped online presence, and an app that the DC Center owns.
We're working with other agencies as things improve, we're getting requests that are not with our current agency partners and represent a need to expand.
Program manager, DC 311
...
Now you can call go online, tweet, use the app or text. With increased use of Twitter we formed a social media team: me and five other representatives that we chose based on their schedule so we have coverage seven days a week.
Internal user, DC 311
Chicago's LargeLots also has been consistently improved through significant ongoing work, funded chiefly by a community foundation that strongly supports the project.
It's gone through several changes from the first to the second version - from a one-page website it became a Django website with an application in it. Now we're on the fifth website, we have to do some work to get new data in each time. We use each pilot as a chance to revisit feedback from [stakeholders.] There are 500 commits to the [online code repository] over the last one and a half years.
Developer, LargeLots
Discussion
The primary question explored by this research related to how people inside and outside of governments experienced the civic technology tools they implemented. Was the service delivery experience transformed? Did people experience working with these tools as a benefit, or not? The secondary question explored by this research lay in understanding whether civic technology was helping governments achieve the promised benefits of e-government. Can you achieve all the benefits of e-government without going through the traditionally-theorized evolution process? Is it necessary to move through a set of internal stages to get to the point of meaningful online two-way interaction between citizens and government, or can you just skip ahead?
In many ways, the implementation of a civic technology tool that was embraced by the hosting government was experienced positively. For each tool, the direct goal of increasing useful two-way interaction between citizens and government was realized (1a), even if it was not achieved consistently across all potential users (2a). The fact that the civic technology tools thereby did have meaningful effects on the way that people both inside and outside of government experienced government service delivery means that the e-government promise of building "better relationships between government and the public by making interaction with citizens smoother, easier, and more efficient" was at least partially realized in all five sites.
Furthermore, the fact that the process of creating and implementing civic technology tools often gave citizens an additional path to policy influence suggests the indirect achievement of what Janowski (2015), in his review of e-government trajectories, identified as the fourth stage of e-government: government and the public using their new digital relationships to improve policy. Of the tools themselves, only SpeakUp Austin explicitly sought public influence on potential city policies. However, the process of making public-friendly tools ultimately led to many additional forms of public influence on service delivery (1b). These influences ranged from being able to change prohibitively difficult practices, such as those around submitting an EDS in Chicago, to being invited to participate in service-shaping focus groups, as in DC, to entirely changing the nature of what is transparent about potentially-contentious service delivery processes, as in Seattle's new option to anonymously provide police complaints or Oakland's newly public record of response to public records requests.
Notes ():
24: Janowski, T. (2015). Digital government evolution: From transformation to contextualization. Government Information Quarterly, 32(3), 221-236.
At a higher level, several of these civic technology tools also demonstrated ways that technology development and policy development could be merged, integrating technology more firmly within the core policy-making function of government. Not only did the development of tools affect the public's ability to influence policy, but several tools themselves became a core part of a larger policymaking process. Three of these civic technology tools (LargeLots, RecordTrac, and OPA Complaint Tracker) were developed as an aspect of a more comprehensive policy change process in a city. This co-development of policy and technology tool suggests a strong commitment to making technology serve the improvement of government service delivery. Another interesting feature emerging from these examples is that policies developed in concert with technology tools appear to lead to policy with a lower implementation burden, featuring fewer restrictions and requirements. This effect of lowering the weight of policy implementation may have stemmed from the specific interest of tool developers in making things "user-friendly" – that is, to make the policy's technical implementation as simple as possible for both government and public.
Meanwhile, while internal and external users saw immediate benefits to the civic technology tools, respondents also noted that tools also produced new challenges and problems. In exploring the concerns respondents expressed about the tools, a major category was comprised of observations that the city had devoted inadequate resources for proper tool deployment or for the underlying service (2a,b,c). In this category, respondents identified a variety of problems, including inadequacies in outreach and communication, the need for further tool improvements, and the lack of a consistent funding or contracting model to pay for either software or communication improvements.
While civic technology tools might be free or inexpensive to the government to develop initially, they require additional and ongoing investments which governments may not generally be aware of at launch (2d). The absence of these resources can cause problems in different directions. Where the lack of resources meant that only people already knowledgeable about the service and technology could access the tool easily, this exacerbated "digital divide" problems, in which economically advantaged citizens were better able to make use of the tool than economically disadvantaged citizens (2a). On the flip side of this problem, some respondents who were knowledgeable about how to use the service observed an ironic effect of the tools' deployment: that the relatively smooth operation of the tool contrasted sharply with the less user-friendly aspects of government service users experienced immediately after engaging (2b). This problem created by the civic technology tool would also require additional resources to resolve.
User interest in increased digitization and internal integration stemmed from the fact that the civic technology tools were generally well-appreciated, receiving good reviews from both inside and outside of government, and users wanted more of these benefits (2d). However, if resources are not available for tool improvements or outreach, how many more resources would be necessary to achieve those additional digitization goals?
Here we find the largest challenge of skipping ahead of the early stages of e-government evolution. When government begins providing a tool for two-way interaction with citizens but does not have the infrastructure or resources to fully support it, that interaction is more limited than either side desires, has unintended negative side effects, and seems likely to end when the failure to maintain and improve the tool results in it becoming too outdated to use.
To explore the significance of resource commitments for the successful achievement of e-government benefits, it is instructive to "follow the money" and see what happens when governments do commit additional resources. The governments in two cases – DC311 and Chicago's LargeLots – have apparently solved the problem of providing adequate resources for tool-based services, although they did so in two different ways. DC 311 addressed it by adding an online component to a large program with substantial existing outreach: the city's telephone-based 311 service. By adding the improvement of an online civic technology tool to a larger technology purchase, the government was able to bundle the cost of tool improvements with an existing expense. A purchase of this scale was possible because of the way that Washington, D.C., centralizes municipal communications for most agencies within a single department. Outreach and development for LargeLots, meanwhile, is supported through a public-private partnership model. A long-standing collaborative relationship between the City of Chicago and a local non-profit lay behind the initial development of LargeLots, and this ongoing partnership now provides joint funding for the program, website maintenance, and outreach activities.
It is telling that these two cases, in which cities have to some degree resolved the funding conundrum, are also the only two of the five cases to have substantial in-person public outreach programs for their tool. While all respondents in all cities acknowledged that additional outreach would help the tool's uptake and use, making it more inclusive and equitable, this expense was only possible in places with a settled municipal (or public-private) commitment to the tool's continuous future use. Not only accepting, but actually committing resources and infrastructure to sustaining these interactive tools produces tools that deepen and improve over time. The kind of e-government function that provides not only an online tool for citizens and governments to communicate but also commits to the communication and updating that makes it function well over time, fixing and integrating data to make it work fully for people, does actually achieve the better relationship we were promised.
Conclusion
This study has explored how five US cities experienced the implementation of civic technology tools and sought to cast light on the question of whether this is an effective way to achieve the promised benefits of e-government. While it focused solely on local governments in the US, there is no particular reason to believe that the findings are not applicable to any government with an interest in civic technology but an unclear level of commitment to achieving the full transformation to e-government. Certainly, several decades into the process, it feels fair to assess that e-government development will follow a more complicated path than simple and rapid transformation. Taking seriously the public administration critique that government change is usually incremental, we can see that e-government development is likely to be more complex than even a multi-stage trajectory model of e-government evolution would suggest. Exploring the impact of civic technology tools implemented by US municipal governments confirms that technology does not suddenly transform service delivery or change public attitudes. However, it also complicates the notion of how progress toward that state happens. Unlike the simple, unidirectional model of e-government development happening internally before it blossoms into interaction with the public, complex interactions between citizen-facing civic technology and internal digitization may result in cycles of reciprocal effects. Rather than serving as the crowning achievement of successful internal digitization, civic technology may itself propel e-government development through the heightened expectations that develop when the government and public interact more regularly online.
Civic technology tools, in providing a better online interface for communication between the public and service-providing departments, provide new forms of motivation to the departments to improve both the tool and their service. Appreciation for the value of the tool creates internal and external pressure on the department to provide ongoing maintenance and improvements to the tool and its related public service. Even if those improvements are presently out of reach for some departments, appreciation for the digitized service, plus a clearer understanding of the ways that service could be enhanced, creates fertile ground for further incremental change, moving departments down the path towards realizing more comprehensive e-government. Further research could work to identify whether new civic technology projects accelerate government's technical development more quickly as compared with similarly situated governments which do not implement civic technology projects.
If civic technology tools help to speed the adoption of further aspects of e-government, more governments will find they need to figure out supportive funding models. Without funding outreach, digital services will reach only those who are already comfortable with technology, and the additional access to public services will disproportionately benefit only people who already visit government websites. Without funding improvements, enthusiasm for the tools will wither and the added public value provided by the improved digital services will disappear. However, if governments are sufficiently prodded by the interest generated by civic technology tools, that investment will be seen as worthwhile. Civic technology tools, rather than magically materializing at the end of the e-government development road, will have instead provided the push governments need to go down it in the first place.
Boehner, K. & DiSalvo, C. (2016). Data, Design and Civics: An Exploratory Study of Civic Tech. San Jose, CA: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, 2970-2981.
Feeney, M. K., & Welch, E. W. (2012). Electronic participation technologies and perceived outcomes for local government managers. Public Management Review, 14(6), 815-833.
Feeney, M., Welch, E. W., & Haller, M. (2011). Transparency, civic engagement, and technology use in local government agencies: Findings from a national survey. Institute for Policy and Civic Engagement, University of Illinois at Chicago.
Janowski, T. (2015). Digital government evolution: From transformation to contextualization. Government Information Quarterly, 32(3), 221-236.
Layne, K. & Lee, J. (2001) Developing fully functional E-government: A four stage model. Government Information Quarterly, 18(2), 122-136.
Leitner, C. (2003). eGovernment in Europe: the state of affairs. EPIAScope,2003(3), 37-39.
Linders, D. (2012). From e-government to we-government: Defining a typology for citizen coproduction in the age of social media. Government Information Quarterly, 29(4), 446-454.
Macintosh, A. (2004, January). Characterizing e-participation in policy-making. In System Sciences, 2004. Proceedings of the 37th Annual Hawaii International Conference on (pp. 10-pp). IEEE.
Miles, M. B., & Huberman, A. M. (1994). Qualitative data analysis: An expanded sourcebook. Sage.
Miles, M. B., Huberman, A. M., & Saldana, J. (2013). Qualitative data analysis: A methods sourcebook. SAGE Publications, Incorporated.
Norris, D. F. (2010). E-Government 2020: Plus ça change, plus c'est la meme chose. Public Administration Review, 70(s1), s180-s181.
Norris, D. F., & Reddick, C. G. (2013). Local e-government in the United States: Transformation or incremental change?. Public Administration Review, 73(1), 165-175.
O'Reilly, T. (2009, August 10) Gov 2.0: The Promise of Innovation. Forbes. https://www.forbes.com/2009/08/10/government-internet-software-technology-breakthroughs-oreilly.html#736714b73b7b
Peixoto, T. & Fox, J. (2016) When Does ICT-Enabled Citizen Voice Lead to Government Responsiveness? IDS Bulletin, 47(1), 23-40.
Peixoto, T. & Sifry, M. (2017) Civic Tech: New Solutions and Persisting Challenges. In T. Peixoto & M. Sifry, (Eds.), Civic Tech in the Global South (pp. 25-46). Washington, DC: International Bank for Reconstruction & Development / The World Bank.
Rumbul, R. (2015). Who Benefits from Public Technology? Demographic and public attitudes research into the users of civic technologies. London, UK: mySociety. Retrieved August 31, 2018 from https://research.mysociety.org/media/outputs/demographics-report.pdf.
Rumbul, R. (2016). Developing Transparency through Digital Means? Examining Institutional Responses to Civic Technology in Latin America. Journal of e-Democracy, 8(3), 12-31.
Sæbø, Ø., Rose, J., & Flak, L. S. (2008). The shape of eParticipation: Characterizing an emerging research area. Government Information Quarterly, 25(3), 400-428.
Silcock, R. (2001). What is e-government. Parliamentary affairs, 54(1), 88-101.
Welle, K., Williams, J. & Pearce, J. (2016) ICTs Help Citizens Voice Concerns over Water – Or Do They? IDS Bulletin, 47(1), 41-54.
West, D. M. (2004). E-government and the transformation of service delivery and citizen attitudes. Public administration review, 64(1), 15-27.
Wood, C. (2016, August 16). What is Civic Tech? Government Technology. Retrieved August 31, 2018, fromhttp://www.govtech.com/civic/What-is-Civic-Tech.html.