Welcome!

Mobile IoT Authors: Pat Romanski, LeanTaaS Blog, Liz McMillan, Elizabeth White, Shelly Palmer

Related Topics: @DXWorldExpo, Microservices Expo, @CloudExpo

@DXWorldExpo: Blog Post

Big Data Analysis Aids Advertising By @Dana_Gardner | @BigDataExpo [#BigData]

Rapid matching of consumer inferences to ads serves up a big data success story

Big Data Analysis Aids Advertising Capabilities

The next BriefingsDirect big data innovation success story uncovers how New York-based adMarketplace, a search syndication advertising network, uses big data to improve its search advertising capabilities.

In part two of our series on adMarketplace, we'll explore how they instantly capture and analyze massive data to allow for efficient real-time bidding for traffic sources for online advertising. And we'll hear how the data-analysis infrastructure also delivers rapid cost-per-click insights to advertisers.

To learn how, BriefingsDirect sat down with Raj Yakkali, Director of Data Infrastructure at adMarketplace, at the recent HP Big Data 2014 Conference in Boston. The discussion is moderated by me, Dana Gardner, Principal Analyst at Interarbor Solutions.

Here are some excerpts:

Gardner: Tell us about adMarketplace. What do you do, and why is big data such a big part of that?

Yakkali: adMarketplace is a leading advertising platform for search intent. We provide the advertisers with the consumer space where they can project their ads. The benefit of the adMarketplace comes into play where we provide a data platform that can match those ads with the right user intent.

Yakkali

When user searches for a certain keyword, they're directly telling us what they want to see, and we match it perfectly well with our ads. The relationship that we have with our advertisers is that we match them well and make it accessible in exactly what the user is thinking. We do some predictive analytics on top of what the user is saying. We add that dimension to our user search and provide ads aptly.

Gardner: I'm all for getting better ads based on lot of things I already get. Do you have more than just keywords in terms of how you can draw inference, and what sort of scale of data are we talking about when it comes to all that inference information about an intent on behalf of the consumer?

15 dimensions

Yakkali: Keyword search is one side or one dimension of the user search. There are also category campaigns that the advertisers are running. At the same time, there's a geospatial analysis to it as well. There are 15 dimensions that we go through to provide an ad that is perfectly fit for the advertiser and for the consumer to see and take advantage of to meet their needs. With some of the ads, we are trying to serve the user’s requirements and needs.

http://bit.ly/1sWpHmCGardner: With all these variables, this sounds like you're going to be gathering an awful lot of information. You also need to reply back with your results very fast or you lose the opportunity for that consumer to get the ad and then even click through and make a decision. Tell me about scale and speed.

Yakkali: You're right on with that question. In this business, latency is your enemy. If you look into the certain metrics, there are almost a half a billion requests that we're receiving every day and we have to match all of those ads with a sub-second performance. We have internal proprietary datasets, which we take care of before matching these ads. And there are two platforms that we've built internally.

One is called Bid Smart. That performs the analysis between the user intent and the traffic sources that the user search is coming from. At the same time, the price of that ad goes to the publisher. There are the pricing strategies, the traffic sources, and the user intent of the search. All of these things are put together. That predictive analytics system gathers all this information and emits the right ad towards the consumer.

With the partnership with Vertica, we’re able to take the dataset, derive analytics about it, and provide our marketers with all that information.

On top of it, if you look into the amount of data, those half a billion requests that are coming into our system, it generates around two terabytes per hour. At certain times, we can't store all of it for analytics. There is a lot of data that's not inside the database. Now, with the partnership with Vertica, we’re able to take the dataset, derive analytics about it, and provide our marketers with all that information. Bid Smart is the one that does the pricing and matching.

The other thing is Advertiser 3D, which provides that detailed analytics into all these dimensions on the metrics. That provides a very good insight. Now, when it comes to the competition or the opportunity to deliver the right ad at the right time, that's where data work flows make a difference.

We utilize Vertica to directly stream all this click data into it, rather than going into certain other locations and then doing it in a batch format. We directly live-stream that data into Vertica, so that it is readily available for analytics. Our Bid Smart System makes use of that dataset. That's where we get the opportunity to deliver much better ads, with price tags, and the right user intent matched.

Gardner: It sounds very complex. There's an awful lot going on for just serving up an ad. I suppose people don’t appreciate that, but the economics here are very compelling, the more refined and appropriate an ad can be, the more likely the consumer is to buy, but there are a lot of resources that don't get wasted in the meantime. Do you have any sense of what the payoff is, either in business, financial, or technical terms for when you can really accomplish this goal of targeted advertising?

Conversion rate

Yakkali: So our conversion rate is a major key performance indicator (KPI) when it comes to understanding how well we are doing. When we see higher conversion rates, that gives us the sense that we've done the best job and user is happy with what they are searching and what they are getting.

At the at the same time, the publishers, as well as the advertisers, are happy, because the user is coming to us again and again to get that similar, beautiful experience. The advertisers are able to sell more products that meet the needs of the user. And the users are able to get the product that really caters to their needs. We're in the middle of all these things, trying provide the facilitation to the advertisers, as well as the users and the publishers' space.

Gardner: I daresay this is the very future of advertising. Now for you to accomplish these goals and create those positive KPIs, are you housing Vertica in your own data center, do you use cloud, hybrid cloud? Given that you have different platforms, different datasets, how do you manage this technically?

Yakkali: On that end, we started with testing cloud two or three years ago, but again, it turned out that because of so many unknowns and troubleshooting, we had to go with our data centers. Now, we host all our systems in our own data centers and we manage it.

We have our own hardware to deal with. Our system is a 24/7, and we have to be able to deliver the sub-second latency performance. Having your own infrastructure, you have the controlled environment where you can tweak and tune your system to get the best performance out of it.

Considering that it is a 24/7, there are fewer excuses that you can get away with in not delivering it. For that, we do innovation in terms the data flows and the process of how we ingest the data, how we process the data, how we emit the data, and how we clean up the data when we don’t really need it.

All these things have to come together, and it really helps us having that control on all of our infrastructure and all elements in the data pipeline, starting from the user intent and user search, until we provide the data and the results.

Gardner: How long have you been using Vertica in this regard and how did you go about making that decision?

Yakkali: We've been using Vertica for four to five years and our data pipeline was not on Vertica to start with, but as Vertica came into the picture and we saw the great beauty and the powerful features that it brings to capitalize our ability.

That really helped us. With Vertica in place, we have been migrating our mechanics slowly to use it for the real-time analysis and real-time bidding and all those beautiful features that make us do what we can do better. So it’s been a great partnership with Vertica and we see many more features coming in with the new version. Our Bid Smart mechanism is also improving, and with that, algorithmic capabilities are increasing. So it’s progressing.

Feedback loop

Gardner: Tell us a little bit about where your business is heading. In addition to speed, complexity, and scale, where do you see the ability to create this feedback loop? It’s very rapid feedback loop between a lot of incoming data and an action like streaming up an ad. It seems like this could be applied to either other marketing or advertising chores or perhaps even have an ancillary business-development direction. You’ve got this platform and these data centers. Is there something else that you're gearing up for?

Yakkali: At this point, we're in the business of connecting the advertisers, the publishers, and the users. But that is an untapped business to what it can accomplish. The market has started its pathway towards the level of reaching that epitome. If we take a step back and try to understand it, initially, when search started, there was no Google or anything. It was more about curated search.

So the publishers put out all this content together and then projected it out to the user. They didn't know what user wanted. At the same time, when the user looked at this content, they didn't know whether they want it or whether it catered to their needs.

Then, Google came along and user search started. What that directly told was "I want this piece of information. I want to use this piece of information. And I want to see this ad that is relevant to my needs." That’s a very powerful thing. When you hear that part, you're able to analyze that piece and match it properly with the advertisers. But then again, it started to fragment.

At this point, we're in the business of connecting the advertisers, the publishers, and the users.

Now, it’s not only Google. There is Yahoo, Bing, there is mobile, and there are certain apps. There are many apps in the mobile space and each one has its own search. So not all the searches are going to Google, Yahoo, or Bing. Search is already fragmented.

We tap all those pieces. The market that is beyond Google. Yahoo-Bing is stronger and it is growing. So there is a lot of market that needs to be tapped into. We come into the place connecting the advertisers to tap that untapped marketplace.

We've been improving our internal Bid Smart algorithm that came out in the last year. Then, we also launched Advertiser 3D last year as well. Those two products have been providing tremendous growth in our revenue, and the retention rates have been stellar.

The top 60 percent of Google’s top spenders are working with us to complement their business. At the same time, we're also able to provide 50 percent increase in year-over-year revenues. It's additional revenue for them, and even our revenues are increasing based on that fact.

Gardner: It seems like you have an awful lot of runway ahead of you in terms of where search could be applied, and analytics can be drawn from that to augment these services and explode that market.

Is Vertica being used just for the intercept between the incoming data and the outgoing ad, or you are also analyzing what goes on within these marketplace so that you better appreciate, whether you can offer reports, audit trails, and that sort of thing? Is this an inclusive platform, or do you use different analytics platforms for different aspects of what you are doing?

End to end

Yakkali: We do almost everything. It is an end-to-end platform. As part of the business we look into the operational metrics of the whole thing, starting from the user search until the ad is delivered. Then, from that end, there is always that analytics piece that comes onto play, which provides insights to the marketers.

Our market base is filled with the very data-savvy marketers, and they look into each and every data dimension to understand their return on investment (ROI). We give them transparency through our Advertise 3D System and utilizing that, they're able to navigate through the space and aptly tune their campaigns to get the best out of it and to deliver the best to the customer.

Gardner: Any thoughts about other organizations that are also facing significant challenges around speed, scale, also perhaps with a big runway, in terms of knowing that more and more business could be coming their way therefore more data? What would you advise them in terms of the data architecture or the planning in order to accomplish the goals?

Yakkali: When we look at the industries and the market, the ad industry still is untapped. The healthcare industry is just getting into the business of doing much more with analytics. It’s all about the speed and the latency and the insights as well. One, at the operational level and the other, at the insight level to do more innovation on top of it.

Our market base is filled with the very data-savvy marketers, and they look into each and every data dimension to understand their return on investment (ROI).

The ability to listen to the customer depends on how fast you can capture all that feedback, and you tighten that loop of feedback so that you're able to do something with it and make a better product out of it.

So it’s all about taking a look at the datasets very closely as to what they mean, what the user is asking us, what do they want to see, and how you are listening to the customer. Those two aspects really make the difference.

You want to listen to the customer, what they really want. Are you providing it and are you able to guess what they want for tomorrow for that predictive, and going into prescriptive analytics, phase later on. You're telling them what they need to do even before they tell you.

That's the stage that the market is going towards. We're not even scratching the surface of prescriptive analytics. The wave has not yet started towards that route. We're still at the predictive analytics phase, and there is still a lot more to go within that space. Get the foundation stronger, drive towards prescriptive analytics, and listening to your customer, are the three aspects that would make any industry. Those three would be the key foundational pieces for making innovation.

You may also be interested in:

More Stories By Dana Gardner

At Interarbor Solutions, we create the analysis and in-depth podcasts on enterprise software and cloud trends that help fuel the social media revolution. As a veteran IT analyst, Dana Gardner moderates discussions and interviews get to the meat of the hottest technology topics. We define and forecast the business productivity effects of enterprise infrastructure, SOA and cloud advances. Our social media vehicles become conversational platforms, powerfully distributed via the BriefingsDirect Network of online media partners like ZDNet and IT-Director.com. As founder and principal analyst at Interarbor Solutions, Dana Gardner created BriefingsDirect to give online readers and listeners in-depth and direct access to the brightest thought leaders on IT. Our twice-monthly BriefingsDirect Analyst Insights Edition podcasts examine the latest IT news with a panel of analysts and guests. Our sponsored discussions provide a unique, deep-dive focus on specific industry problems and the latest solutions. This podcast equivalent of an analyst briefing session -- made available as a podcast/transcript/blog to any interested viewer and search engine seeker -- breaks the mold on closed knowledge. These informational podcasts jump-start conversational evangelism, drive traffic to lead generation campaigns, and produce strong SEO returns. Interarbor Solutions provides fresh and creative thinking on IT, SOA, cloud and social media strategies based on the power of thoughtful content, made freely and easily available to proactive seekers of insights and information. As a result, marketers and branding professionals can communicate inexpensively with self-qualifiying readers/listeners in discreet market segments. BriefingsDirect podcasts hosted by Dana Gardner: Full turnkey planning, moderatiing, producing, hosting, and distribution via blogs and IT media partners of essential IT knowledge and understanding.

@ThingsExpo Stories
To get the most out of their data, successful companies are not focusing on queries and data lakes, they are actively integrating analytics into their operations with a data-first application development approach. Real-time adjustments to improve revenues, reduce costs, or mitigate risk rely on applications that minimize latency on a variety of data sources. In his session at @BigDataExpo, Jack Norris, Senior Vice President, Data and Applications at MapR Technologies, reviewed best practices to ...
A strange thing is happening along the way to the Internet of Things, namely far too many devices to work with and manage. It has become clear that we'll need much higher efficiency user experiences that can allow us to more easily and scalably work with the thousands of devices that will soon be in each of our lives. Enter the conversational interface revolution, combining bots we can literally talk with, gesture to, and even direct with our thoughts, with embedded artificial intelligence, whic...
Cloud Expo | DXWorld Expo have announced the conference tracks for Cloud Expo 2018. Cloud Expo will be held June 5-7, 2018, at the Javits Center in New York City, and November 6-8, 2018, at the Santa Clara Convention Center, Santa Clara, CA. Digital Transformation (DX) is a major focus with the introduction of DX Expo within the program. Successful transformation requires a laser focus on being data-driven and on using all the tools available that enable transformation if they plan to survive ov...
Smart cities have the potential to change our lives at so many levels for citizens: less pollution, reduced parking obstacles, better health, education and more energy savings. Real-time data streaming and the Internet of Things (IoT) possess the power to turn this vision into a reality. However, most organizations today are building their data infrastructure to focus solely on addressing immediate business needs vs. a platform capable of quickly adapting emerging technologies to address future ...
With tough new regulations coming to Europe on data privacy in May 2018, Calligo will explain why in reality the effect is global and transforms how you consider critical data. EU GDPR fundamentally rewrites the rules for cloud, Big Data and IoT. In his session at 21st Cloud Expo, Adam Ryan, Vice President and General Manager EMEA at Calligo, examined the regulations and provided insight on how it affects technology, challenges the established rules and will usher in new levels of diligence arou...
In his session at 21st Cloud Expo, Raju Shreewastava, founder of Big Data Trunk, provided a fun and simple way to introduce Machine Leaning to anyone and everyone. He solved a machine learning problem and demonstrated an easy way to be able to do machine learning without even coding. Raju Shreewastava is the founder of Big Data Trunk (www.BigDataTrunk.com), a Big Data Training and consulting firm with offices in the United States. He previously led the data warehouse/business intelligence and B...
"Digital transformation - what we knew about it in the past has been redefined. Automation is going to play such a huge role in that because the culture, the technology, and the business operations are being shifted now," stated Brian Boeggeman, VP of Alliances & Partnerships at Ayehu, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
SYS-CON Events announced today that Synametrics Technologies will exhibit at SYS-CON's 22nd International Cloud Expo®, which will take place on June 5-7, 2018, at the Javits Center in New York, NY. Synametrics Technologies is a privately held company based in Plainsboro, New Jersey that has been providing solutions for the developer community since 1997. Based on the success of its initial product offerings such as WinSQL, Xeams, SynaMan and Syncrify, Synametrics continues to create and hone inn...
"Evatronix provides design services to companies that need to integrate the IoT technology in their products but they don't necessarily have the expertise, knowledge and design team to do so," explained Adam Morawiec, VP of Business Development at Evatronix, in this SYS-CON.tv interview at @ThingsExpo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
The 22nd International Cloud Expo | 1st DXWorld Expo has announced that its Call for Papers is open. Cloud Expo | DXWorld Expo, to be held June 5-7, 2018, at the Javits Center in New York, NY, brings together Cloud Computing, Digital Transformation, Big Data, Internet of Things, DevOps, Machine Learning and WebRTC to one location. With cloud computing driving a higher percentage of enterprise IT budgets every year, it becomes increasingly important to plant your flag in this fast-expanding busin...
In his Opening Keynote at 21st Cloud Expo, John Considine, General Manager of IBM Cloud Infrastructure, led attendees through the exciting evolution of the cloud. He looked at this major disruption from the perspective of technology, business models, and what this means for enterprises of all sizes. John Considine is General Manager of Cloud Infrastructure Services at IBM. In that role he is responsible for leading IBM’s public cloud infrastructure including strategy, development, and offering m...
Nordstrom is transforming the way that they do business and the cloud is the key to enabling speed and hyper personalized customer experiences. In his session at 21st Cloud Expo, Ken Schow, VP of Engineering at Nordstrom, discussed some of the key learnings and common pitfalls of large enterprises moving to the cloud. This includes strategies around choosing a cloud provider(s), architecture, and lessons learned. In addition, he covered some of the best practices for structured team migration an...
No hype cycles or predictions of a gazillion things here. IoT is here. You get it. You know your business and have great ideas for a business transformation strategy. What comes next? Time to make it happen. In his session at @ThingsExpo, Jay Mason, an Associate Partner of Analytics, IoT & Cybersecurity at M&S Consulting, presented a step-by-step plan to develop your technology implementation strategy. He also discussed the evaluation of communication standards and IoT messaging protocols, data...
Recently, REAN Cloud built a digital concierge for a North Carolina hospital that had observed that most patient call button questions were repetitive. In addition, the paper-based process used to measure patient health metrics was laborious, not in real-time and sometimes error-prone. In their session at 21st Cloud Expo, Sean Finnerty, Executive Director, Practice Lead, Health Care & Life Science at REAN Cloud, and Dr. S.P.T. Krishnan, Principal Architect at REAN Cloud, discussed how they built...
SYS-CON Events announced today that Evatronix will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Evatronix SA offers comprehensive solutions in the design and implementation of electronic systems, in CAD / CAM deployment, and also is a designer and manufacturer of advanced 3D scanners for professional applications.
22nd International Cloud Expo, taking place June 5-7, 2018, at the Javits Center in New York City, NY, and co-located with the 1st DXWorld Expo will feature technical sessions from a rock star conference faculty and the leading industry players in the world. Cloud computing is now being embraced by a majority of enterprises of all sizes. Yesterday's debate about public vs. private has transformed into the reality of hybrid cloud: a recent survey shows that 74% of enterprises have a hybrid cloud ...
22nd International Cloud Expo, taking place June 5-7, 2018, at the Javits Center in New York City, NY, and co-located with the 1st DXWorld Expo will feature technical sessions from a rock star conference faculty and the leading industry players in the world. Cloud computing is now being embraced by a majority of enterprises of all sizes. Yesterday's debate about public vs. private has transformed into the reality of hybrid cloud: a recent survey shows that 74% of enterprises have a hybrid cloud ...
DevOps at Cloud Expo – being held June 5-7, 2018, at the Javits Center in New York, NY – announces that its Call for Papers is open. Born out of proven success in agile development, cloud computing, and process automation, DevOps is a macro trend you cannot afford to miss. From showcase success stories from early adopters and web-scale businesses, DevOps is expanding to organizations of all sizes, including the world's largest enterprises – and delivering real results. Among the proven benefits,...
@DevOpsSummit at Cloud Expo, taking place June 5-7, 2018, at the Javits Center in New York City, NY, is co-located with 22nd Cloud Expo | 1st DXWorld Expo and will feature technical sessions from a rock star conference faculty and the leading industry players in the world. The widespread success of cloud computing is driving the DevOps revolution in enterprise IT. Now as never before, development teams must communicate and collaborate in a dynamic, 24/7/365 environment. There is no time to wait...
SYS-CON Events announced today that T-Mobile exhibited at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. As America's Un-carrier, T-Mobile US, Inc., is redefining the way consumers and businesses buy wireless services through leading product and service innovation. The Company's advanced nationwide 4G LTE network delivers outstanding wireless experiences to 67.4 million customers who are unwilling to compromise on qua...