The internet of things and big data: Unlocking the power

The 'internet of things' (IoT) and 'big data' are two of the most-talked-about technology topics in recent years, which is why they occupy places at or near the peak of analyst firm Gartner's most recent Hype Cycle for Emerging Technologies:


The ‘internet of things’ (IoT) and ‘big data’ are two of the most-talked-about technology topics in recent years, which is why they occupy places at or near the peak of analyst firm Gartner’s most recent Hype Cycle for Emerging Technologies:

If you have somehow missed the hype, the IoT is a fast-growing constellation of internet-connected sensors attached to a wide variety of ‘things’. Sensors can take a multitude of possible measurements, internet connections can be wired or wireless, while ‘things’ can literally be any object (living or inanimate) to which you can attach or embed a sensor. If you carry a smartphone, for example, you become a multi-sensor IoT ‘thing’, and many of your day-to-day activities can be tracked, analysed and acted upon.

Big data, meanwhile, is characterised by ‘four Vs‘: volume, variety, velocity and veracity. That is, big data comes in large amounts (volume), is a mixture of structured and unstructured information (variety), arrives at (often real-time) speed (velocity) and can be of uncertain provenance (veracity). Such information is unsuitable for processing using traditional SQL-queried relational database management systems (RDBMSs), which is why a constellation of alternative tools — notably Apache’s open-source Hadoop distributed data processing system, plus various NoSQLdatabases and a range of business intelligence platforms — has evolved to service this market.

The IoT and big data are clearly intimately connected: billions of internet-connected ‘things’ will, by definition, generate massive amounts of data. However, that in itself won’t usher in another industrial revolution, transform day-to-day digital living, or deliver a planet-saving early warning system. As EMC and IDC point out in their latest Digital Universe report, organisations need to hone in on high-value, ‘target-rich’ data that is (1) easy to access; (2) available in real time; (3) has a large footprint (affecting major parts of the organisation or its customer base); and/or (4) can effect meaningful change, given the appropriate analysis and follow-up action.

As we shall see, there’s a great deal less of this actionable data than you might think if you simply looked at the size of the ‘digital universe’ and the number of internet-connected ‘things’.


A huge number of ‘things’ could join the IoT, whose recent rise to prominence is the result of several trends conspiring to cause a tipping point: low-cost, low-power sensor technology; widespread wireless connectivity; huge amounts of available and affordable (largely cloud-based) storage and compute power; and plenty of internet addresses to go round, courtesy of the IPv6 protocol (2^128 addresses, versus 2^32 for IPv4).

Estimates and projections of the current and future number of internet-connected objects vary, depending on the definitions used and the optimism of whoever is doing the estimating and projecting. The best-known figures come from Cisco, which puts the current (February 2015) number at around 14.8 billion and the expected number in 2020 at around 50 billion (and that’s just 2.77 percent of an estimated 1.8 trillion potentially connectable ‘things’):


EMC and IDC are somewhat more conservative, putting the 2020 IoT population at 32 billion, while Gartner comes in with 26 billion.

HOW much ‘big’ data?

data was getting critically ‘big’ even before IoT devices entered the picture. EMC and IDC had been tracking the scale of the ‘digital Universe’, or DU, due to the fact that 2007 (the DU is all of the digital data created, replicated and consumed in a single year). In 2012 EMC and IDC predicted that the DU could double every two years to reach 40 zettabytes (ZB) with the aid of 2020 — a number in view that revised upwards to 44ZB (that is 44 trillion gigabytes). Astronomical numbers possibly need astronomical illustrations, which can be why EMC/IDC pictured 44ZB as 6.6 stacks of 128GB iPad Air tablets reaching from Earth to the moon. The DU estimate for 2013 was 4.4ZB (or one stack of iPads achieving -thirds of the way to the moon).

The 2013 4.4ZB DU estimate breaks down into 2.9ZB generated by means of customers and 1.5ZB generated by way of businesses. however, only 0.6ZB (15%) of the purchaser portion is not touched by means of enterprises in some way, leaving enterprises responsible for the big majority of the world’s data (about 3.8ZB in 2013). As referred to above, EMC and IDC forecast that the IoT will develop to 32 billion related devices via 2020, with a view to contribute 10 percent of the DU (up from 2% in 2013). In phrases of geography, EMC and IDC expect that the balance will swing from mature markets, which accounted for 60 percentage of the DU in 2013, to emerging markets, with a purpose to generate a similar percentage in 2020, with the inflection point happening round 2016/17.

fortuitously for firms, the ‘target-rich’ data which can doubtlessly supply actionable business insights is estimated by EMC and IDC to be a greater attainable 1.5 percentage of the 2014 general (it really is 0.066ZB, or 66 million terabytes).

Cisco’s big-scale data tracking a task, the worldwide Cloud Index (GCI), specializes in information center and cloud-primarily based IP traffic, estimating the 2013 amount at 3.1ZB/year (1.5ZB in ‘conventional’ facts centers, 1.6ZB in cloud data centers). by 2018, this is expected to have risen to 8.6ZB, with most of the people increase happening in cloud data centers (2.1ZB traditional, 6.5ZB cloud). in line with Cisco, 75 percentage of the 2018 site visitors (6.4ZB) will continue to be inside data centres (storage, manufacturing, development, and authentication visitors), with 17 percentage (1.5ZB) flowing between data centres to users, and 8 percent (0.7ZB) flowing between data centres (replication and inter-database links).

ALSO READ  Big data is about more than lots of data. Here's 4 Way to use it to make smarter decisions

what’s startling, but, is Cisco’s estimate of the amount of data generated through net of the whole lot (IoE) devices — which encompasses people-to-people (P2P), machine-to-human beings (M2P), and device-to-device (M2M) connections — with the aid of 2018: 403ZB, that is 47 times the predicted overall data centre traffic and 267 times the expected amount flowing among data centres and users (see Cisco’s Forecast and methodology white paper for extra information). No wonder service companies and mobile operators are taking the IoT extremely seriously.


The IoT and big data are simply growing apace, and are set to convert many regions of business and everyday lifestyles. however which precise sectors are probable to experience the IoT/big data disruption first? In its 2015 net of things predictions, IDC notes that: “nowadays, over 50% of IoT activity is targeted in production, transportation, clever city, and purchaser applications, however within 5 years all industries can have rolled out IoT projects”.

In its 2014 virtual Universe document, EMC and IDC see the IoT developing new business possibilities in 5 primary regions, summarised in this slide:

To supply on these possibilities, in keeping with EMC’s bill Schmarzo, a new era of IoT packages is needed to address particular commercial enterprise wishes which includes: predictive upkeep; loss prevention; asset usage; inventory tracking; disaster planning and recovery; downtime minimization; energy utilization optimization; device overall performance effectiveness; network overall performance management; potential usage; capability planning; call for forecasting; pricing optimization; yield management; and load balancing optimization.

If these and other nuts and bolts of the IoT/big data revolution can be put in location, there’s a remarkable deal of economic fee to play for. as an example, Cisco posted a couple of research in 2013 that evaluated the anticipated fee at stake from the internet of the entirety (IoE) and got here up with $14.4 trillion for the private area and $4.6 trillion for the general public sector.

inside the private sector, Cisco expects the value to be driven in five most important regions: asset utilisation ($2.5 trillion); worker productivity ($2.5 trillion); supply chain and logistics ($2.7 trillion); client experience ($3.7 trillion); and innovation, together with lowering time to market ($3.0 trillion). within the public region the primary proposed drivers are: worker productivity ($1.8 trillion); connected militarized protection ($1.5 trillion); price reductions ($740 billion); citizen experience ($412 billion); and increased revenue ($125 billion).

As regular on the subject of the IoE/IoT, Cisco’s economic value predictions err at the positive aspect; other analysts are commonly extra conservative — despite the fact that the numbers bandied about are still massive.


before the IoT/big data nexus can supply on its promise, there are some of the obstacles to conquer. the main ones are summarized below.


For the IoT to work, there must be a framework within which gadgets and applications can trade data securely over wired or wi-fi networks. One player in this region is OneM2M, an umbrella orgainsation consisting of seven requirements bodies, five worldwide ICT fora and over 200 organizations (mostly from the telecoms and IT industries). In February this year, OneM2M issued launch 1, a set of 10 specifications overlaying requirements, structure, API specifications, safety solutions and mapping to not unusual industry protocols (which includes CoAP, MQTT and HTTP). “release 1 utilises nicely-verified protocols to allow applications throughout industry segments to speak with every other as never before — no longer simplest moving M2M forward however definitely enabling the internet of things,” said Dr Omar Elloumi, Head of M2M and clever Grid standards at Alcatel-Lucent and OneM2M Technical Plenary Chair, in a declaration.

OneM2M has additionally published a useful white paper that characterises the history to its task therefore: “The emerging need for interoperability throughout extraordinary industries and applications has necessitated a pass far from an industry-unique approach to one which entails a common platform bringing together connected motors, healthcare, clever meters, emergency services, neighborhood authority offerings and the many other stakeholders in the atmosphere”.

not exceedingly, given the scope and potential cost of the IoT marketplace, there are masses of different standards bodies vying to get their ideas followed. those include: the AllSeen Alliance, Google’s The bodily internet, the industrial internet Consortium, the Open Interconnect Consortium and Thread.

safety & privacy
in line with IDC, “inside years, ninety% of all IT networks will have an IoT-based security breach, although many could be taken into consideration ‘inconveniences’…chief information protection officers (CISOs) may be pressured to adopt new IoT guidelines”. progress on information standards (see above) will help here, however there’s no question that protect and privacy is a big fear with the IoT and big data — in particular when it comes to areas like healthcare or essential national infrastructure.


The IoT became certainly prominent in the safety predictions for 2015 issued with the aid of analysts and other pundits at the start of the year. here’s a variety:

Your refrigerator isn’t an IT security hazard. business sensors are (Websense)
assaults at the internet of things will recognition on clever domestic automation (Symantec)
internet of factors attacks circulate from proof-of-idea to mainstream risks (Sophos)
the distance between ICS/SCADA and real international protection most effective grows bigger (Sophos)
Technological diversity will keep IoE/IoT devices from mass attacks however the equal might not be real for the information they system (fashion Micro)
A wearables fitness records breach will spur FTC action (Forrester)
it is not all doom and gloom, although. commentators foresaw safety tightening around critical infrastructure in 2015, for example:

vital infrastructure will see protection improvements (Neohapsis)
extra attention to securing our important infrastructure (Damballa)
big data featured within the 2015 safety predictions too, but no longer to such an volume: upward thrust of Salami attacks will leave a awful taste at the big data banquet (Varonis systems); machine studying will be a game-changer within the fight towards cyber-crime (Symantec); and big data will become a buzzword for the awful men too (Neohapsis).

As OneM2M factors out, safety inside the IoT is hard because the more than one stakeholders could have different needs: “For a telecoms operator, protection is about making sure availability; for a purchaser enterprise, it’s approximately protective their facts; and for an M2M and IoT provider it’s about making sure uptime…The large range of M2M and IoT device sorts, their specific abilities and the variety of deployment situations makes security a unique project for the M2M and IoT industry”.

network & data centre infrastructure
The full-size quantities of data as a way to be generated by IoT devices will put sizeable pressure on community and records centre infrastructure. IoT data flows could be on the whole from sensors to packages, and could variety among continuous and bursty depending on the type of software:

traffic patterns for special M2M/IoT programs. image: OneM2M/Alcatel-Lucent Bell Labs
As Gartner points out, the importance of IoT-associated network connections and facts volumes is probably to favour a distributed method to data centre control, with a couple of ‘mini-facts centres’ appearing preliminary processing and applicable information forwarded over WAN links to a valuable website for similarly analysis. this will show serious problems round storage for (necessarily selective) information backup, community bandwidth and statistics centre capacity planning, where statistics center Infrastructure management, or DCIM, equipment will become highly needed. Cisco, meanwhile, has coined the time period ‘fog computing’ to explain data processing on the network part, to get around location-based and/or network latency problems — something so as to be a feature of the IoT.

Analytics equipment
The volume, speed and range (not incude variable veracity) of IoT-generated data makes it no easy job to choose or build an analytics answer which could generate beneficial business insight. The most recent (q4 2014) analyst report to mainly deal with this trouble is from ABI studies, which also a section on vendors that operate in unique elements of the analytics pipeline (data integration, data storage, center analytics and data presentation), in addition to ‘full-stack’ providers like IBM, Microsoft, Oracle, SAP and software AG.

The intersection of the IoT and big data is a multi-disciplinary subject, and specialized abilties can be required if organizations are to extract maximum cost from it. two kinds of human beings may be in demand: business analysts who can frame suitable questions to be asked of the available data and present the effects to choice makers; and facts scientists who can orchestrate the (rapidly evolving) cast of analytical equipment and curate the veracity of the data getting into the analysis pipeline. In rare cases, the business analyst and the statistics scientist can be one and the same (valuable) person.

We could be on the edge of an IoT/big data revolution, however if Gartner’s Hype Cycle is an inexpensive model, then we still have the backlash (a.k.a. the Trough of Disillusionment) and the rehabilitation (a.k.a. the Slope of Enlightenment) to deal with. limitations to mainstream adoption that want to be addressed consist of data standards, safety and privacy, network and data centre infrastructure, the provision of suitable analytical equipment, and access to enterprise analysis and data science capabilities to be able to permit actionable insights to be extracted from these equipment.

looking into the additional out, it is possible to assume a world where augmented-reality platforms like Microsoft’s lately unveiled HoloLens have matured, and people automatically work and play within an facts-rich amalgam of the bodily and the IoT-augmented surroundings.


the UK authorities’S WALPORT report AND AN ANALYST’S response

December 2014 noticed the publication of a report entitled The internet of things: making the maximum of the second one digital Revolution, by the United Kingdom government’s leader scientific adviser Sir Mark Walport. It become commissioned by David Cameron following a March 2014 speech on the CeBIT trade fair, wherein the top Minister declared his ambition to “make the UK the most digital nation inside the G8”.
the key section of the record is the government summary, which incorporates ten clear and particular suggestions:

government desires to foster and sell a clean aspiration and vision for the internet of things. The aspiration ought to be that the UK could be a global chief within the improvement and implementation of the internet of factors. The goal is that the internet of things will permit goods to be produced more imaginatively, offerings to be supplied more efficaciously and scarce assets to be used greater sparingly.
authorities has a leadership role to play in delivering the vision and ought to set high ambitions. government should remove obstacles and provide catalysis. There are 8 regions for movement: Commissioning; Spectrum and networks; requirements; abilties and research; data; regulation and legislation; trust; Co-ordination.
government have to be an professional and strategic client for the internet of things. It have to use informed buying power to define great practice and to commission technology that uses open standards, is interoperable and secure. It need to encourage all entrants to market; from begin-united states of america to set up players. It need to support scalable demonstrator tasks to offer the environment and infrastructure for developers to strive out and enforce new programs.
(a) government have to work with specialists to increase a roadmap for an internet of things infrastructure. (b) authorities need to collaborate with industry, the regulator and academia to maximize connectivity and continuity, for both static and mobile devices, whether utilized by business or consumers.
With the participation of industry and our studies communities, authorities need to help the development of standards that facilitate interoperability, openness to new market entrants and safety against cybercrime and terrorism. government and others can use expert commissioning to encourage participants in demonstrator programmes to develop standards that facilitate interoperable and secure systems. government should take a proactive role in using harmonisation of standards internationally.
growing skilled human beings begins at school. the mathematics curriculum in secondary school should circulate far from an emphasis on calculation per se toward using calculation to solve issues. authorities, the education sector and companies must prioritise efforts to broaden a skilled team of workers and a supply of capable facts scientists for enterprise, the 1/3 sector and the Civil service.
Open application programming interfaces must be created for all public our bodies and controlled industries to enable innovative use of actual-time public information, prioritising efforts inside the energy and transport sectors.
authorities need to develop a flexible and proportionate version for law in domains laid low with the internet of things, to react speedy and correctly to technological alternate, and stability the consideration of potential advantages and harms. The facts Commissioner will play a key position inside the location of personal statistics. Regulators have to be held accountable for all choices, whether or not these accelerate or postpone applications of the internet of factors that fall within the scope of law.
The Centre for protection of national Infrastructure (CPNI) and Communications and Electronics security organization (CESG) should work with industry and worldwide companions to agree excellent practice safety and privacy principles primarily based on “safety through default”.
The virtual financial system Council must create a web of factors advisory board, bringing collectively the personal and public sectors. The board would have a remit to: co-ordinate authorities and private quarter investment and support of the relevant technologies; foster public-private collaboration wherein this will maximise the performance and effectiveness of implementation of the net of things; work with government to advise policymakers while regulation or legislation may be wished; keep oversight and awareness of potential dangers and vulnerabilities related to the implementation of the internet of factors; and sell public communicate. To be powerful this board need to be supported via an appropriately funded secretariat.
Responding to the Walport record, Jeremy green, principal analyst at IoT/big data specialist Machina research, commended it as a “a surprisingly strong file with an great analysis of the position authorities can play in co-ordinating and promoting a healthy IoT sector”. green did criticise the file for “a haphazard and partial set of assisting case research” and stated “scope for a more joined-up approach to maintaining a database of deployments”. but, general Machina research feels that “this report deserves to be read broadly and taken forward at a departmental stage”