Today, most technology aficionados think of data engineering as the capabilities associated with traditional data preparation and data integration including data cleansing, data normalization and standardization, data quality, data enrichment, metadata management and data governance.  But that definition of data engineering is insufficient to derive and drive new sources of society, business and operational value. The Field of Data Engineering brings together data management (data cleansing, quality, integration, enrichment, governance) and data science (machine learning, deep learning, data lakes, cloud) functions and includes standards, systems design and architectures.

There are two critical economic-based principles that will underpin the field of Data Engineering:

Principle #1:  Curated data never depletes, never wears out and can be used an unlimited number of use cases at a near zero marginal cost.

Principle #2:  Data assets appreciate, not depreciate, in value the more that they are used; that is, the more these data assets are used, the more accurate, more reliable, more efficient and safer they become.

There have been significant exponential technology advancements in the past few years ; data engineering is the most topical of them. Burgeoning data velocity , data trajectory , data insertion , data mediation & wrangling , data lakes & cloud security & infrastructure have revolutionized the data engineering stream. Data engineering has reinvented itself from being passive data aggregation tools from BI/DW arena to critical to business function. As unprecedented advancements are slated to occur in the next few years, there is a need for additional focus on data engineering. The foundations of AI acceleration is underpinned by robust data engineering capabilities.

YourStory & AIQRATE curated and unveiled a seminal report on “Data Engineering 4.0: Evolution , Emergence & Possibilities in the next decade.” A first in the area , the report covers a broad spectrum on key drivers of growth for Data Engineering 4.0 and highlights the incremental impact of data engineering in the time to come due to emergence of 5G , Quantum Computing & Cloud Infrastructure. The report also covers a comprehensive section on applications across industry segments of smart cities , autonomous vehicles , smart factories and the ensuing adoption of data engineering capabilities in these segments. Further , it dwells on the significance of incubating data engineering capabilities for deep tech startups for gaining competitive edge and enumerates salient examples of data driven companies in India that are leveraging data engineering prowess . The report also touches upon the data legislation and privacy aspects by proposing certain regulations and suggesting revised ones to ensure end to end protection of individual rights , security & safety of the ecosystem. Data Engineering 4.0 will be an overall trojan horse in the exponential technology landscape and much of the adoption acceleration that AI needs to drive ; will be dependent on the advancements in data engineering area.

Please fill in the below details to download the complete report.

No conversation around technology today is complete without a reference to artificial intelligence (AI). While the term AI has been around for decades, in recent times, its application across industries, from mining to healthcare, education and finance, among others, is changing the way we live, both at work and at home. 

The increasing ubiquity of AI can be attributed to greater processing power and the declining cost of achieving these tasks at increased speeds. The adoption of AI has been so rapid that technology leaders across industries can no longer ignore it for long-term and sustained financial growth.

At the third edition of YourStory’s Future of Work event in Bengaluru on Saturday, AI was the focus of a panel discussion, which drew attention to ‘How AI is disrupting business models and value chains and unleashing transformative and innovative growth opportunities’. 

The session was moderated by Sameer Dhanrajani, CEO and Co-founder, AIQRATE, and the panelists were Rajan Sethuraman, CEO, LatentView Analytics; Soumendra Mohanty, EVP Analytics and COO, Tredence; Prithvijit Roy, CEO and Co-founder, BRIDGEi2i; and Iqbal Kaur, Co-Founder, Zylotech. The discussion yielded many insights on the top trends around the technology.

1. Experiments at scale are the need of the hour

It is clear that we are at a stage where we need to experiment at scale.

Data Engineering has come out as a prominent area within the AI arena; building robust data pipes has assumed significant importance to fuel machine learning algorithms. At Future of Work 2020 event, YOURSTORY & AIQRATE jointly unveiled a bespoke report on “Data Engineering 4.0: Evolution, Emergence & Possibilities in the Next Decade”.

The report identifies 5G, Data Pipeline+, AI Managed Data Lakes, Edge Cloud and IoT as some of the main drivers for the emergence of Data Engineering 4.0 to realize the much anticipated cross industry collaboration enabled business models involving Smart Cities, Autonomous Vehicles, and Smart Factories.

In continuation to my last article on Building a Robust Data Strategy, let me meaningfully conclude it by highlighting some of the core issues which need to be addressed before data monetization could really be called our as a success and ROI is achieved.  

Privacy Concerns

Company needs to have the implicit and/or explicit statutory or legal right, or the ethical right, to divulge private consumer data – either personalized or depersonalized, individualized or at an aggregated level. Especially in industries where regulatory bodies have a heavy clout over what data is being used to cull out actionable insights or even the data flow within or beyond the walls of the organizations. Numerous articles, reports & surveys have highlighted how crucial is for businesses to operate within the ethical boundaries of data gathering or dissemination. Leave no stone unturned to see what policies/restrictions/guidelines are in place for the industry you operate in, how easy/difficult is to access data, and what are customer or end user reactions. You definitely do not intend to burn bridges with your existing customer base or repel away new prospects. Legal actions can be fatal to business at times. Be doubly sure what you are up for!

Technology Constraints

Do have a thorough understanding of the technological or hardware-related considerations to implement the strategy chosen to monetize the data. At times, organizations don’t have the requisite resources to execute on their strategy, may be because that’s not their core area of operation or it’s happening in silo’es across the organization which the business unit in question is not privy to. A complete landscaping exercise to understand the current state of business, what’s new in the market & what the competition is up to, what’s the future state & a step-by-step roadmap to mature technological prowess. In many cases, businesses hire external consultants or seek handholding by analytics service providers who have the requisite experience in recommending about the gaps & even executing on filling those. A thorough detailed analysis (but not analysis-paralysis) is crucial to the overall success.

Intellectual Property

At times, organizations sitting on huge pile of valuable data choose to make it available in the market (as another viable revenue model to monetize data). How much data to sell and how to determine costs vs. benefits in putting valuable data on the open market should be thought through. Be privy to the pros & cons of each approach & choose your business model accordingly.

Core Competency

Depending on its core competency, organization needs to identify at which level it wants to monetize the data in the data value chain. Data at each & every touchpoint in the value chain may have its own peculiar problems (missing data, incorrect data etc) and not all of it may be relevant. If your differentiator is “speedy delivery” of goods to your customers, focus on picking the right data sets across the value chain which helps streamlining operations, optimize inventory & transit time. Know what you are best at or what you are known for in the market and harness data capabilities to strengthen your business on that front.

Data Accuracy and/or Liability

Potential problems with inaccurate or directly or indirectly regulated data insights or products hitting the market place. Make sure that data assimilation, aggregation & cleansing exercise is robust enough to ensure the analysis/insights being generated out of it have a high probability of giving the right sense of direction to the business. At times, over-ambitious expectations or poor data quality can directly impact the quality of the outcomes. Garbage-in Garbage-out is the mantra & business managers should perfectly understand the gaps in the data & be cautious before making any solid commitments.

Perceived Market Value

For larger market opportunities, it is likely that an organization would want to play at the higher level in the data value chain. Umpteen times that completely derails the whole Analytics ROI & data monetization exercise. Focus should be specifically on business model(s) used to monetize the data than otherwise.

All the aforementioned considerations should set a good pretext to the data monetization exercise and may be the key to unlocking true value from data strategy initiative. In my subsequent edition, I shall bring to light the “Analytics Centre of Excellence” concept & how can organizations setup a full-fledged Analytics unit to deliver insights to departmants/LOB’s/functions across the business and also serve as a backbone to building a data-driven organization of the future. Stay tuned !

Unarguably, data and technology is truly redefining & rehashing the way companies do business. Organizations have always had data, which they have utilized to run their businesses more efficiently but recent developments have transformed the way data is utilized by such organizations.

In today’s disruptive economic environment, all leaders are vying for identifying new revenue streams and identifying existing value streams inside the organization especially data. This is where the concept of crafting a Robust Data Strategy comes in, how do we make most of the Dark Data ? Data is now being looked as an asset and business models are now being build around this vast value pool which is hidden inside the data being stored. Enterprises are now anticipating future needs based on preference insights culled out from past & present data. They are creating new products and services in tune with what their customers exactly seek. They are lending an ear to all suggestions/recommendations/feedback shared and also responding to queries/concerns in real time. They are doing it all with data and analytics.

While many companies are becoming aware of the opportunities embedded in their enterprise data, only a few have developed active strategies to monetize it successfully. Data Strategy requires companies to not only understand their data, but also to uncover gaps and evaluate suitable business model(s) for appropriately monetizing the enterprise data. To evaluate their respective monetization opportunities in a more informed and results-driven manner, companies need to assess the value of enterprise data, determine how best to maximize its potential and figure out how to get the data to the market efficiently.

Four Stages to Analytics Sophistication

Based on the current state of data affairs, any organization can be categorized as a beginner, developing, matured or leader. In the initial stages of transformation, organization typically
lacks synergies due to silo’ed efforts, is less agile and more prone to errors, with perennial data quality concerns. As they mature to be leaders in the Analytics space, data sits at the heart of business, with increasingly automated, instant, accurate and seamless data driven decision-making.

• Beginner: Basic infrastructure and tools, proliferation of dashboards and reports
• Developing: Building tools and processes for historical as well as deep diving analysis to gain some insights for future actions
• Matured: Organization adoption of advanced analytical capabilities to predict future outcomes
• Business Transformation or Leader: Centralized analytics focus with capabilities to anticipate future and act in a data driven manner

Time’s ripe to ride on the Data & Analytics wave

Enterprises capture a lot of data, most of which is often overlooked. With reducing costs of capturing and storing data, increasing data analysis capabilities and superior analytical technologies available, enterprises have started to recognize data as one of their most valuable assets. In the few years, enterprises who lead the way in reorienting their approach, initiating enterprise wide data-led transformations and effectively monetizing their data are expected to be in the forefront. Typical market forces driving widespread adoption of Analytics are:

Technological Advancement

• Technology advancement has facilitated real time data analysis and personalized communication
• Big data technologies, cloud computing, machine intelligence and other advancements etc. have made analysis simpler & efficient

Rise of Consumerism

• Influx of more demanding consumers will force a wave of change
• Consumer engagement and experience management are key levers to success

Data Explosion

• Daily volume of data being captured increasing rapidly
• Cost of storing data decreasing massively
• Recognition of amount of under-utilized data that can be used to derive additional value

Increasing importance of Analytics & BI

• Business Intelligence and Analytics becoming an integral part of organization’s decision making

Economic Pressures

• Pressure on profit margins are forcing increased focus on efficiency and cost reduction
• Increasing competitive pressure

Is your Data truly worth it?

How much business value can be created via data on which organizations are sitting on depends primarily on the following factors & to an extent determines the success of any Analytics initiative.

Predict Behaviour (Patterns)

Enterprise data should be detailed enough to build a successful data monetization strategy. E.g. Customer data should be detailed enough to be able to predict customer behavior, patterns etc.

Size of the Ecosystem

Businesses with high volume, large breadth of data have the ability to generate highest value from the data. Companies with national or global scale can easily establish market view, which makes it more meaningful and valuable

Accessibility and Actionable

Data becomes valuable only if its rich, actionable and accessible. Structured, & readily scalable data makes the process of monetization simpler and efficient, providing higher potential for data monetization

Customer Identification (Granularity)

Data becomes valuable only if it is granular enough to be able to identify the end user/ customer. Ability to identify/ profile customers helps in expanding the range of products and servives that can be offered

Uniqueness

Uniqueness of the enterprise data is extremely valuable. It makes the products/services offered by the enterprise exclusive to the enterprise, sustainable differentiation which most organization yearn for

Stages to Data Maturity

Based on maturity of organization’s data, it can take a call what kind of a player it wants to be in the market – a “data seller” or a “full services provider”.

Raw Data

• Selling raw unprocessed data to outside stakeholders
• Companies with rich pool of high quality raw data can onsell such data with little investment required

E.g. – Pharma related data or even NASDAQ’s “Data on Demand” service to its ecosystem of partners in the capital markets

Processed Data

• Companies collect and integrate data from multiple sources
• Data is processed, stored and leveraged in summary form
• Secure capture and transport of data
• Proper storage and management of data using a data platform

E.g. Card Advisory companies provide processed data to merchants and/ or use it for improving its operational efficiency

Business Intelligence/ Predictive Insights

• Tools and technologies such as data mining, predictive modeling and analytics convert data into insights
• Insights are made available to the stakeholders (both internal and external) to drive business decisions

E.g. Wal-Mart segments its customers into three primary groups based on purchasing patterns to spur growth

Products & Solutions Implementation

• Data-driven interactions with end users
• APIs and ability for companies to access platform and data to build comprehensive products and solutions
• Companies use the intelligence to improve product and solutions offering portfolio

E.g. Tesco bank uses Clubcard customer data to identify customer needs and creates new personalized offers

Key Elements to Designing a Robust Data Strategy

Unravel Customer Needs

• Continually understand the customer needs to unearth customer requirements and preferences
• Understanding the delivery and integration models that clients require in order to benefit from enhancements
• Create a business model which fits into the core competency and create offerings which fir into client platforms and applications
• Invest in continuous learning and management of customers’ unmet needs ranging from enhancements to new products/ solutions

Decrypting the Enterprise Data

• Understand the enterprise data captured across all business lines and develop an enterprise wide nomenclature for the same
• Identify and map data and analytics services across business units to understand what types of capabilities can be leveraged to build new products and services using the appropriate business model

Gauging the Market Potential

• Calculate the market potential for the various opportunities identified
• Estimate the revenue potential, internal rate of return, investment required, cost reduction, efficiency etc. for the process
• Understand the key competition, factor in macro and micro factor which can affect the marketplace demand
• Seek out opportunities to enhance the core business or develop new products and services.

Deciphering the Value Chain

• Develop insights into partners and competitors across the value chain including upstream suppliers, data partners etc.
• Identify the new opportunities that can be available across the value chain
• Create a comprehensive view of the data ecosystem

Enhance the existing infrastructure

• Develop a sophisticated yet flexible architecture, suitable technology and applications which can help unlock the value that the opportunities might presents
• Put in place a data infrastructure that can provide the necessary foundation to enable the organization to unlock the value of data assets

The crux of the matter is that with the huge amount of data available with the enterprise’s in today’s competitive and converging business environment, they should start looking for market opportunities leveraging the data available with them. Most of the enterprises still do not consider data as an asset which they can monetize if they choose the correct business strategy and build the required capabilities. Enterprises can not only make better use of their internal data to enhance the current product and services portfolio, it can also provide new insights into the value chain and could transform the enterprise, unleashing a whole new set of products and services for the customers.

By utilizing internal data with external data, powerful generation of high margin solutions can be developed which can transform an entire organization which possesses enormous revenue potential. Done properly, data ecosystems can fund the transformation, create value for customers, and build long lasting relationships with other partners firms, 3^rd party vendors & suppliers. But to ensure the true value of data is being monetized by the enterprise, it is essential that it follows a streamlined process to identify the most suitable business model(s) taking into account all constraints which the process might need addressing.

Imagine a Lamborghini or Ferrari or any car of your choice for that matter, with a fantastic engine parked in the garage, you’d love to get your hands on the wheel, wouldn’t you? After all, it makes no sense to invest so heavily in such a mean machine and yet leave it hidden under a wrap, right? Think of the Data sitting inside the organizations as the great, potential “engine”; an invaluable asset which remains elusive most of the times, well, pretty much in storage, as most CXO’s agree that they’re not doing nearly enough to maximize the use of effective analytics to unleash the potential of dark data they are sitting on. The future belongs to those organizations that effectively employ analytics to understand their markets, customers and operations. Forward-thinking organizations recognize that data is becoming the new source of competitive advantage, and organizations are re-thinking value creation and investing in new analytics infrastructure. In fact, data analytics is routinely cited by CXO’s as among their top one or two priorities year after year. Companies are already making use of data to advance a variety of business goals and to help consumers. Few of the leading organizations who are pioneering in this space of harnessing data for business value (like Facebook, Google, LinkedIn, and Amazon have) shown the world what is possible when data is used to its truest potential in cutting-edge ways, and the idea that enterprises must recommit themselves to become data-driven is now a widely held notion. While many companies have excelled in the use of data analytics and predictive modeling, data-driven decision-making is no slam-dunk. Several companies are struggling to make data-driven decision-making part of their DNA.

Across industries, “Big Data” and Analytics are helping businesses to become smarter, more productive, and better at making predictions. Organizations today are collecting increasing amounts of disparate data. In fact, they are collecting more data than they can manage or analyze; which means most of the data being collected is underutilized. Yet, organizations understand and know that data and data analysis can provide an important strategic competitive advantage. Businesses today are under extreme pressure and face significant challenges to reduce overall costs, improve outcomes, adapt to new technologies, comply with strict regulatory restrictions and face the ever increasing power of the consumer. Organizations agree that building analytics competency can and will drive improved delivery outcomes, quality and cost leveraging the “power” of data. Best in class organizations are adopting analytics to drive decision making, improve outcomes, increase member loyalty/ retention, reduce unnecessary costs, and increase accountability.

Organizations that know where they stand on the analytics maturity continuum are better prepared to turn challenges into opportunities. By performing their current state assessment and building an enterprise value roadmap for analytics adoption, organizations can define the “best way forward” to completely engage a data-driven culture. Tapping this potential for your organization begins with shaping a plan. You have to set a strategy; draw a detailed road map for investing in assets such as technology, tools, and data sets; and tackle the intrinsic challenges of securing leadership buy-in, reinventing processes, and changing organizational behavior. Analytics is not just about generating insights, but getting those insights to the right people. To sustain the long-term success of data-driven innovation, it is necessary to continually revise one’s analytical approach in order to generate insights that lead to new innovation and competitive advantage.

The first stepping stone in the direction or crafting a robust data strategy starts with doing a comprehensive Analytics Maturity Assessment exercise. Inherent question which crosses our minds is, “Why Analytics Maturity Assessment”?

Need for Analytics Maturity Assessment

• The most critical aspect to any organization is to leverage true benefit of data, decipher where they are today, where they’ve been in the past, the progression curve and a direction where they intend to go in the future based on data/information available at their disposal
• By leveraging maturity assessment framework, organizations can measure the current maturity of the data (how good is it to perform analysis) and the overall analytics program in an objective way across various dimensions that are key to deriving accelerated value from data
• Uncover how their data efforts stand in comparison to those of their peers in order to ensure best-in-class insight and support, and ensure we are in tune with the contemporary market trends
• The assessment shall also render guidance to companies at the cusp of starting their data journey, by helping them understand industry best practices used by companies across geos, of different sizes & even from industries that are more mature in their deployments
• After performing the benchmark study, organizations will be able to quantify the maturity of their deployment in an objective way, understand the progress, and identify what it takes to graduate to the next level of maturity

Key Challenges impeding Analytics Proliferation

Organizations want to leverage data analytics but face challenges while trying to formulate a strategy around it due to:

• Lack of Vision
  • Business leadership doesn’t have a defined corporate strategy to drive data driven culture
  • No vision on how to embed analytics into the decision making process
• Disparate Data Sources
  • Data stored in silos across departments
  • Many different types of data sources
  • Large amount of data generated
• Talent Crunch
  • Lack of people with Domain knowledge as well as business analytics expertise
  • Lack of people with knowledge of varied data types and tools to integrate, process and develop insights
• Resource Availability
  • Lack of resources to quickly turn around on-demand analytics
  • Low bandwidth with IT resources to provide near real time information

Maturity Assessment Guiding Principles

In order to ensure your Analytics Maturity Assessment exercise in worth the time & effort, a few guiding principles would come in handy.

• Data aggregation across multiple data sources: Analytics needs to gather the information from multiple sources across business/ functional areas
• Blending existing and new data: Analytics should be capable to use the existing data that is available inside the organization and utilize with the new data outside the organization (social, market research, surveys, competition etc)
• Business user friendly: Analytics should be understandable to all relevant stakeholders intended to be consuming the insights
• Predictive Analytics: Analytics must provide the anticipative/predictive model and should also support “what-if” analysis for different scenarios
• Scalability and Flexibility: Analytics should be able to get customized but at the same time should be scalable and extensible for future needs
• Real-time Analytics Tools/Services: Analytics should be using tools to quickly process the data and translate that into actionable insight

Key Levers impacting Analytics Usage & Adoption

For any organization’s expectations and aspirations, and the current state of analytics takeoff primarily banks on the following five key levers. Key stakeholders who shall be impacted (including CXO’s & Senior Managers) need to be included as part of the assessment workshop where appropriate brainstorming needs to happen on existing challenges being face by business, current maturity of Analytics usage across departments/functions, thorough deep-dive into use cases where analytics consumers share their experiences of how they see analytics as a key ingredient to value creation in LOB’s or departments under their ambit.

• People
  • Identifying key stakeholders
  • Carving out roles/responsibilities
  • Talent needs
  • Change management
  • Training requirements
  • Analytics skills & competencies

• Data
  • Data sources & management
  • Data integration & accessibility
  • Data Infrastructure
  • Aligning data sources to use cases
  • Tools/tech/platforms requirement
  • Data maturity

• Vision
  • Analytics vision and goals
  • Assessment of key BU’s vision
  • Analysing business drivers & needs
  • Executive Sponsorship
  • Business readiness
  • Top down buy-in for analytics uptake
  • Accountability and ownership

• Process
  • Streamline existing analytics, reporting and operational processes
  • Identification of modelling approaches required
  • Benchmarking to best practices
  • Mapping information needs
  • Monitoring and refresh

• Ongoing Improvement
  • Governance, Risk and compliance
  • Improvement Process & Analysis Methodology
  • Data and systems governance
  • Documenting and reporting distribution needs
  • Identification of future investment areas

In the next edition to this data strategy blog, I shall be touching upon the key stages an organization goes through as it matures along the analytics adoption curve, core design principles to execute a successful data monetization strategy, and key data and analytics transformation dimensions to choose the ideal business model(s). Stay tuned !