eCommerce News New Zealand logo
The latest digital commerce news for Kiwi businesses
Story image

Dell EMC: We built this city on a data lake

Thu 15 Dec 2016
FYI, this story is more than a year old

Smart cities are rapidly moving from concept to reality across Australia and New Zealand, with a number of cities hiring chief digital officers, the recent launch of a local smart cities council, and a region-wide race to be the first city to go “smart”.

With such a futuristic concept, it’s easy to get caught up in all the hype and excitement. But as with any new technology in a mostly conceptual phase, it’s important for us to take a moment to stop and think about what implementation will look like.

The United Nations predicts that by 2050 around 70 per cent of the world’s population will be concentrated in urban areas, a whopping 16 percentage point increase from 2014. Couple this with Cisco’s expectation that each person will have an average of 25 connected devices in 2050, and the amount of data generated by smart cities of the future is almost incomprehensible.

A city built around trillions of 0s and 1s rather than physical infrastructure is a foreign concept, so naturally several questions spring to mind. Where will all that data live? How will we be able to quickly convert data into valuable insights? Will it all be worth the investment?

The answers to those questions? Well, it depends on the technology behind the city. For a city to be truly smart, its systems need to be able to access and process vast amounts of data on the fly. With roughly 80 per cent of today’s data growth ‘unstructured’, traditional approaches to data storage and analytics tools simply won’t work. What is needed is a modern data center that keeps pace with rapid data growth, simplifies management, lowers costs and enables easy access to analytics.

This is where data lakes come in. A modernised version of the data warehouse, a data lake is scale-out storage for data consolidation. It spans from edge locations to the core data center and right up to the cloud, and stores, manages and protects unstructured data. So, what would a smart city built on a data lake look like? And why are data lakes looking like the way for us to get smart with our cities?  

Hosting diverse data

In today’s data-driven world, we’re dealing with data of multiple sources and types. There’s structured data, such as corporate databases, with its well-defined structure that means estimating future data storage needs is a fairly straightforward process.

Then we have unstructured data, which includes documents, videos, images, internet data and telemetry from myriad IoT devices. Such data can range from a few bytes to terabytes, meaning predicting when or how much of this data will need to be stored can be almost impossible.

Think about all the different types of data generated across a single city. Take Sydney, with its population of around five million people. You’ve got data being collected from millions of individuals across infrastructure, transport, communications, and urban planning. Or consider just the CCTV surveillance system in Brisbane, which generates data across 200 cameras, 200 access points, 12 monitors and 130 channels of analytics. This data will take multiple forms and won’t be of equal importance.

Data lakes can address this challenge by automatically moving data to its most appropriate storage tier based on parameters set around frequency of data access, age of data, type of application supported and file size.

Keeping things secure

That smart cities will be a target for hackers is not a difficult conclusion to draw. With CCTV cameras, power grids, water supply and several other vital city services at risk of malicious intent, security must and will be a paramount consideration in the implementation of smart city initiatives. With such sensitive data at stake, our smart cities will demand robust and secure IT built around more sophisticated storage technologies such as data lakes.

In the case of storage silos, it is more difficult to maintain control of security as the varying silos force duplication and inconsistent application of governance and security policies. By consolidating data and eliminating silos, data lakes afford better oversight and control of data protection and security policies.

Faster time to insights

Smart cities need to do more than just collect mass amounts of data, to deliver real value for our society they must be smart with it. There will need to be a huge focus on generating insight as quickly as possible to allow infrastructure and services to react.

Consider the potential for emergency services in a smart city. In the case of a road incident, emergency services will be alerted and receive the necessary information, traffic will be intelligently redirected to avoid the affected area, communications networks will be flexible in directing capacity to where it is needed, and information will be collated to inform citizens about the incident.

Data lakes have in-place analytics and afford faster time to insight as there is no need to chew time by making multiple copies or moving large data sets.

The scale, complexity and demands that will be placed on urban infrastructure and services in the future are vast. Smart initiatives will drastically alter the distribution of cities, with technologies such as driverless cars, automation and augmented reality reducing the need for citizens to physically be in a city centre, and allowing the freedom to live and work anywhere. When this unfolds, people will need technology to connect them on a level we can’t yet imagine.

Data lakes can underpin the remote workforce and smart city of the future. But in the case of new technologies, we need to ensure our heads aren’t stuck in the clouds and clear up the base level questions necessary to successful implementation and return on investment.  

Article by Matt Zwolenski, chief technology officer, Dell EMC ANZ.

Related stories
Top stories
Story image
Tablets & laptops
Chromebook and tablet shipments see another rapid decline for the year
According to research from Canalys PC Analysis, Chromebook and tablet shipments have fallen for the fourth quarter in a row for Q2 of 2022.
Story image
Google NZ numbers show strong comprehensive profit increase
The latest financial report from Google New Zealand Limited has revealed an increase in total comprehensive profit of over $NZD 7 million.
Story image
Financial results
Facebook NZ financial report reveals notable revenue increase
Revenue from contracts with customers increased by $NZD 1,089,292 compared to 2020's figures.
Story image
OfficeMax NZ sees significant growth through Seismic partnership
OfficeMax New Zealand has announced it has seen a significant increase in customer and sales confidence as a result of Seismic’s digital enablement software.
Story image
BNZ launches first tap-on-phone point of sale app in NZ
Bank of New Zealand has launched BNZ Pay, an innovative mobile app for retailers that transforms an Android device into a contactless payment terminal. 
Story image
Freshworks integrates with Google's Business Messages
"The integration with Freshworks makes it fast and easy for businesses to have conversations with their customers within the Google apps."
Story image
Tranxactor Group to build customer loyalty with Oracle
Tranxactor has chosen Oracle Cloud Infrastructure (OCI) with Enterprise Database Service to allow it to provide brands with immersive customer engagement and loyalty programs.
Story image
Social Media
ActiveCampaign reveals consumers seek trusthworthy content
Consumers will engage with new brands across all channels, including online, in-store and social media, as long as the content is relevant and trustworthy.
Story image
Akamai research finds PayPal security measures utilised in new phishing scam
New research from Akamai has found that a new threat actor is parasitising benign WordPress sites to execute an extensive PayPal phishing scam.
Story image
Cisco NZ revenue declines by over $18.5 million - report
Cisco NZ has released its latest financial report, showing the company's total revenue has declined by more than $18.5 million year-over-year.
Story image
Marketplacer and Intelligent Reach to help retailers sell online
Intelligent Reach can now support Marketplacer marketplaces that want to sell their products through other places, such as Google and Facebook, eBay and Amazon.
Story image
Cost of fraud up 10 to 16% in APAC from pre-pandemic levels
LexisNexis Risk Solutions released the latest APAC edition of the True Cost of Fraud Study, which surveyed 387 risk and fraud executives.
Story image
Microsoft and Auckland Transport announce new cloud agreement
Auckland Transport (AT) and Microsoft have announced a new cloud agreement aimed at promoting innovation, reducing costs and improving sustainability in transport services.
Story image
eCommerce fraud increasing pressure on businesses margins
It is vital for businesses to maximise the value of every dollar by turning away as many fraudulent actors as possible without blocking good customers."
Story image
Hands-on review: Jabra Engage 55 wireless headset
We get our hands on a German design professional headset that many knowledge workers could benefit from.
Story image
IBM NZ sees significant revenue increase in latest report
IBM NZ has posted revenue of $172,449,000 for the financial year, according to its latest report, a year-over-year increase of over $47.5 million compared to $124,904,000 in 2020.
Story image
How manufacturers can respond to rapid change with technology
Disruption, innovation, and continual refinement of shop floor processes are driving factors in today’s complex market landscape. 
Story image
KICKS CREW selects Forter to help scale global eCommerce operations
KICKS CREW has selected Forter to help scale its global digital commerce operations.
Story image
Facial recognition
Māori data specialists not consulted on facial recognition technology - data sovereignty expert
Māori data specialists are accusing the government of ignoring them while going ahead and expanding the reach of facial recognition technology.
Story image
Customer experience
Why scalable networks are key to delivering consumer-centric outcomes
To be consumer-centric, an organisation must have IT infrastructure that adopts the strategy of the 'infinite enterprise'. Extreme Networks explains.
Story image
Digital key for smart investment in public infrastructure for NZ cities
Major public infrastructure projects can better manage risks of cost overruns and delays if they deploy data and digital tools at the earliest planning stages.
Story image
Artificial Intelligence
Cyara rolls out comprehensive, automated chatbot feature
Cyara has unveiled new chatbot testing features with the latest release of Cyara Botium, creating a solution for comprehensive, automated chatbot testing and assurance.
Story image
Airwallex launches an online payments app on Shopify
Airwallex has launched an online payments app on Shopify, allowing merchants to integrate a gateway plugin on their online store to accept payments from their global customers.
Story image
Digital Transformation
Understanding modern retail through the Gen Z filter
Retailers face the challenge of managing their own digital transformation while also scrambling to serve the needs of a younger and increasingly online-savvy audience.
Story image
Customer experience
BillingPlatform introduces new enhancements to revenue management services
Some of the new developments include hosted payment pages, and new and updated connectors to Salesforce, NetSuite, OneSource, Avalara and other enterprise systems.
Story image
Artificial Intelligence
How to overcome the digital experience dilemma - NICE
While self-service solutions can provide major benefits, live voice and chat interactions are still an important part of customer service that continue to need support and investment.
Story image
Digital Journey
NICE unveils new CXone capabilities with latest release
NICE has announced the Summer 2022 release of CXone, which adds new capabilities that enhance journey orchestration and complete performance.
Story image
New Zealand
Research finds Kiwis prefer real backdrops in video calls
New research from Natural Paint Co. has found that 74% of Kiwis prefer seeing a natural background behind people during video meetings.
Story image
SAS is a leader in anti-money laundering - Forrester
The latest Forrester report revealed that SAS received the highest score in the anti-money laundering category of 15 vendors.
Story image
Havas Media Group
Havas solution meets critical time as brands shift to online sales
Havas Media Group says its Havas Market solution provides businesses with a strategic full-service eCommerce offering at a time when sales are increasingly moving online.
Story image
Sitecore caters to modern marketing teams with CMS cloud launch
"Sitecore's move towards a composable SaaS offering for creating and delivering digital experiences is in line with what marketing teams are looking for.”
Story image
Infobip’s SMS and WhatsApp services are now available through Microsoft
Infobip has integrated its WhatsApp and SMS channels through Microsoft Dynamics 365 Sales and Microsoft Dynamics 365 Marketing.
Story image
Commerce Commission
The NZ TCF endorses move by ComCom to promote TDR dispute scheme
The New Zealand Telecommunications Forum has welcomed the move by the Commerce Commission to further promote customers' access to the Telecommunications Dispute Resolution Scheme (TDR).
Story image
Remote Working
Mantel Group continues NZ expansion with Auckland office
"Our desire is to offer real understanding to our New Zealand clients, and help provide solutions that better their business.” 
Story image
Enterprise Resource Planning / ERP
Why the right ERP (and partner) is crucial to an innovative and successful business
Enterprise Resource Planning (ERP) is a foundational step to ensuring a robust business model; here's why choosing the right one could be vital to ensuring long-term success and innovative results.
Story image
Artificial Intelligence
Oracle unveils AI-powered application to automate sales
Oracle has unveiled the next generation of Fusion Sales, an application that automates sales and identifies the opportunities worth pursuing.
Story image
UNIQLO operator chooses Adyen for multiple market payments
The operator of UNIQLO, Fast Retailing, has selected Adyen to power payments for its in-store and online checkout in multiple markets.
Story image
FedEx and eBay team up to boost APAC eCommerce options
FedEx Express' new alliance with eBay enables eBay sellers in APAC to sign up for a FedEx account and access the full spectrum of FedEx e-commerce delivery service options at competitive rates.
Story image
Forter and ShopBack enhance partnership to further prevent fraud
Forter and ShopBack have enhanced their partnership with the addition of better eCommerce security solutions for customers.
Story image
InternetNZ appoints new chief executive. Will take over in October
InternetNZ has announced the appointment of its new chief executive, with Vivien Maidaborn taking over the role from interim chief Andrew Cushen in October.
Story image
Robotic Process Automation / RPA
Western Union implements intelligent automation programme from SS&C Blue Prism and ISG
Western Union has successfully implemented a strategic intelligent automation programme from SS&C Blue Prism and its certified partner, ISG.
Story image
Comedy legend Jimeoin fronts Epson advertising campaign in NZ and Australia
According to Epson the company’s EcoTank models now account for 74% of all printers sold in the category in New Zealand, alone.