Guide to Building Data-Driven Organizations in the Public Sector

Challenges of Organizational Change

Joseph Lynch Marcela Morales

The Challenges of Big Data: Organizational Change

Topic Overview

With over 2.5 Quintillion bytes of data created every day, the greatest challenge to business is how to use this data to improve businesses while making it profitable.

Chapter Summaries

Desouza, K. C., & Smith, K. L. (2014). Big data for social innovation (Links to an external site.)Links to an external site.. Stanford Social Innovation Review, 2014, 39-43. Big Data for social innovation

The term “big data” is used to describe the growing proliferation of data and our increasing ability to make productive use of it. The business community has also been a heavy user of big data. Each month Netflix collects billions of hours of user data to analyze the titles, genres, time spent viewing, and video color schemes to gauge customer preferences to continually update their recommendation algorithms and programming to give the customer the best possible experience. There, a large chasm exists between the potential of data-driven information and its actual use in helping solve social problems. Social problems are often what are called “wicked” problems. Not only are they messier than their technical counterparts, they are also more dynamic and complex because of the number of stakeholders involved and the numerous feedback loops among inter-related components. Numerous government agencies and nonprofits are involved in tackling these problems, with limited cooperation and data sharing among them. Then there are policy and regulatory challenges that need to be faced, such as building data-sharing agreements, ensuring privacy and confidentiality of data, and creating collaboration protocols among various stakeholders tackling the same type of problem. There are multiple dimensions to big data, which are encapsulated in the handy set of seven “V”s that follow. Volume: considers the amount of data generated and collected. Velocity: refers to the speed at which data are analyzed. Variety: indicates the diversity of the types of data that are collected. Viscosity: measures the resistance to flow of data. Variability: measures the unpredictable rate of flow and types. Veracity: measures the biases, noise, abnormality, and reliability in datasets. Volatility: indicates how long data are valid and should be stored. Barriers creating and using big data include the storage of big data in proprietary systems, the regulation on data capture, storage, and curating for accountability, unreliability of data, and the unintended consequence of big data usage. Recommendations: Building global data banks on critical issues Engaging citizens and citizen science (Citizens can also be enlisted to help create and analyze these datasets) Build a cadre of data curators and analysts (We need to equip students and analysts with the necessary skills to curate data so as to create large datasets.)

Making advanced analytics work for you Barton, D., & Court, D. (2012). Making advanced analytics work for you. Harvard business review, 90(10), 78-83.

  1. Choose the right date by mastering the environment you already have and exploring surprising sources of information. Be specific about the business problem that needs to be solved or opportunities they hope to exploit. Get the right technology and IT infrastructure to help integrate siloes information (huge issue in government). It will be a continuous flow of information so IT infrastructure that reports in “batches” will not be helpful.
  2. Identify the business opportunity and determine how the model can improve performance. Use hypothesis-led modeling to generate faster outcomes and outcomes that are more broadly understood by managers.
  3. Make it simple.

Despite big investments in data, many companies have not made it profitable

Despite big investments in data, many companies have not made it profitable: https://www.theregister.co.uk/2017/06/07/go_small_on_big_data/

Mountains of cash keep pouring into the titans of big data despite the world’s inability to do much of value with their software. Companies like Cloudera and Hortonworks subsequently arose to help mainstream enterprises put this otherwise complex software to work. It’s been a lucrative gig, with each company raising hundreds of millions of dollars and, in turn, generating hundreds of millions of dollars in revenue. What none of them has managed, however, is profit, and that’s cause for concern. In other words, the money keeps pouring into the big data companies even as their customers generally struggle to figure out how to turn those investments into meaningful outcomes. These big data vendors then have to spend mountains of cash to convince would-be customers that this time it’s different, that this time their investment will return “actionable insights” – that illusive dream of data scientists everywhere. Indeed, IDG Research nails it when it finds that “abundant data by itself solves nothing.” Companies need to scale back their ambitions to invest in projects that are more evolutionary than revolutionary in nature, looking to tweak rather than overhaul existing operational practices.

Why Managers hate agile management

Why managers hate agile management: https://www.forbes.com/sites/stevedenning/2015/01/28/more-on-why-managers-hateagile/#186ce9f010ea

In the traditional model, there is a top down model where a vision or product is created and this follows a “relay race” through the various managers, line staff, and sales teams. Each level is assigned a different aspect of the vision or product to achieve an end result. As noted in the article “” Why Do Managers Hate Agile?” (Forbes, 2015), the goal of the traditional model was to “have semi-skilled employees…perform repetitive activities competently and efficiently” and coordinating those efforts so that products could be produced in large quantities.” In the Agile model, speed to service or product is the goal which conflicts with the traditional model by using the concurrent work of many (including private entities) to enhance the product. The traditional manager is used to having control of the outcome of the vision or product and this just does not work in the Agile model which is causing the “tension.” To illustrate the differences between traditional and Agile, the Apple IPhone is a good model. If Apple had designed the IPhone using the traditional model, they would release the IPhone with 40 preset applications that they believed were best using consumer input. Once released, Apple would add applications based on consumer demand which would be vetted by management, created by Apple coders, prioritized for release, tested and placed on the platform. This process would be slow and the variation between Apple IPhone would be non-existent. All IPhone would have the same applications loaded. Consumers could seek out competitors with different variations of applications that met their needs. In the Agile method (which Apple uses), Apple created an IPhone with a number of preset applications, however, they have allowed outside entities to create applications based on the public demand. As of March 2018, there were 2.1 million apps available in the Apple App Store. In July of 2008, there were only 800. (https://www.lifewire.com/how-many-apps-in-app-store-2000252) This Agile approach allows the product to stay relevant to the demands of the consumer rather than the vision of the company. Instead of convincing the consumer to buy their product, Apple is giving the consumer what they want as fast as possible. The Agile method releases or lessens the control that the traditional Manager used to enjoy for the speed and variation that a wider population can create. The speed to market on consumer demand is far beyond what a traditional model can keep up with. The loss of control and power that the traditional Manager has in their product or service is tough to swallow and that is why Mangers hate agile.

Key Take-Aways (for Yellowdig)

https://youtu.be/1VFlZ_GM3q8

Discussion Questions

Can data be used to solve social issues deemed “wicked problems” since the infrastructure of non-profit agencies and government do not have the share data in the same way as business.

How can companies and agencies find a way to use big data? Is there a good roadmap to success?

If data is the such a key to success, why are the largest data companies have a problem making profit? Why does the Agile model of business conflict with traditional methods of management?

Why do you think that big data is so important in public sector yet the availability is so limited?

References