Real World Big Data in Azure

MP4 | AVC 336kbps | English | 1024x768 | 15fps | 5h 21mins | AAC stereo 93kbps | 758 MB

Genre: Video Training

How do you make sense of Big Data? When you’re receiving 100 million events per hour and you need to save them all permanently, but also process key metrics to show real-time dashboards, what technologies and platforms can you use? This course answers those questions using Microsoft Azure, .NET, and Hadoop technologies: Event Hubs, Cloud Services, Web Apps, Blob Storage, SQL Azure, and HDInsight. We build a real solution that can process ten billion events every month, store them for permanent access, and distill key streams of data into powerful real-time visualizations.

Understanding Big Data in Azure

Introducing Big Data in Azure

Is This Course for Me?

Building the Phase 1 Solution

Evolving the Phase 2 Solution

Introducing the Telemetry API

Demo: EventsController

Key Features of the Telemetry API

Zipping the Event Batch

Demo: GZipToJsonHandler

Testing the API

Demo: IEventSender

Module Summary

Ingesting Data into Event Hubs

Module Introduction

Introducing Event Hubs

Configuring Event Hubs

Creating Event Hubs

Automating Event Hub Creation

Sending Events

Demo: EventHubEventSender

Batching Events

Demo: EventBatchIterator

Batching and Testing

Demo: Acceptance Tests

Understanding Throughput Units

Demo: Deploy, Monitor & Scale

Next Steps for Deployment & Monitoring

Isolating the Event Hub

Module Summary

Storing Event Data for Batch Queries

Module Introduction

Aim of Deep Storage

Concurrency and Partitions

Demo: IEventProcessor

Competing Consumers

Buffering Output

Demo: DeepStorageEventProcessor

Level 1: In-Process Memory

Demo: MemoryEventStore

Level 2: Local Disk

Demo: DiskEventStore

Level 3: Blob Storage

Demo: BlobStorageEventStore (Part 1)

Demo: BlobStorageEventStore (Part 2)

Timed Flushing

Demo: Deep Storage Worker Role

Scaling Read with Throughput Units

Module Summary

Querying Batch Data in Deep Storage

Module Introduction

About Pig

Demo: AzCopy and Pig

Importing Libraries

Demo: Pig and JSON Data

HDInsight and Pig

Demo: Running Pig Scripts in HDInsight

Streaming Data to Commands

Demo: Streaming from Pig to .NET

Visualizing Output from Pig

Demo: Visualization with D3.js

Module Summary

Normalizing Event Data for Real-Time Queries

Module Introduction

Modeling Event Metrics

Demo: Building the Data Model

Refactoring the Event Receiver

Demo: Core.EventProcessor

Dependency Injection and the EventProcessorHost

Buffering in the Event Handler

Demo: ModelBuffer of T

Handling Event Metrics

Demo: EventMetricsHandler

Running On-Premise

Demo: RealTime.WorkerRole

Running in Azure

Demo: Azure Deployment

Module Summary

Building Real-time Dashboards

Module Introduction

Demo: Installing Dashing.net

About Widgets

Demo: The Number Widget

About Scheduled Jobs

Demo: Event Count Job

How Dashing Pushes Updates

Adding New Widgets

Demo: The Pie Widget

Running Dashing on Azure

Demo: Deploying Azure Web App

Demo: Securing the Web App

Module Summary

Using Storm for the Plumbing

Module Introduction

Spouts, Bolts, and Topologies

Parallelism in Storm

Demo: The Event Hub Spout

Demo: Connecting to the Event Hub Spout

Topology for Event Metrics

Demo: Event Metrics App

Storm and HSInsight

Demo: Running in Azure

Module Summary

Using HBase for Storage

Module Introduction

About HBase

Modeling Device Errors in HBase

Demo: HBase on HDInsight

HBase Row Key Design

Demo: Device Errors in Storm

Demo: Device Errors in HBase

The HBase .NET SDK

Demo: Dashing and HBase

Module Summary

Course Summary