From 0 to 1bn transactions a month on a shoestring budget

Target Audience:

People looking for real-world implementations of scalable data solutions in the cloud

Abstract:

Follow along with my six-month solo journey going from no infrastructure to an Azure distributed data platform that could process more than a billion transactions a month on less than £500 p/m.

You won’t just see the super duper end product, you’ll see the struggles to get there. The tech that didn’t work. The tech that didn’t scale. The oopsy moments that cost a lot. The lessons learnt.

You’ll get to hear about my experiences with Azure data techs like Azure SQL, SQL DW, Azure Storage, Redis, HDInsight, DocumentDB, Azure Functions, Event Hubs, Stream Analytics, Azure Data Factory and more! You’ll see how each fitted (well, mainly didn’t fit) my requirements and what sorts of scales they worked on for my use case.

Why I Want to Present This Session:

Because making Azure work can be hard and few people share the problems and painful lessons learnt.

Additional Resources:

Previous Post
Data + Docker = Discombobulating?
Next Post
Azure SQL Database – Managing your database on the cloud

4 Comments. Leave new

“I worked in a startup and our volumes started small, really small (only 20k or so!)”

I think you’re missing a word – over 20k what?

Reply

Oh man, Steph you are burying the lede here. This looks like a phenomenal presentation disguised as “guys, check out what I did with my raspberry pi!” type story.

First I would start with the title. I’m one of those horrible human beings that votes off the title and only looks at the abstract if I am torn. Your current title is kind of generic and doesn’t have a great hook. I could open this up and find out that you drank the Microsoft koolaid and think that DocumentDB makes your company “global-scale”. From the title, I have no way of knowing what counts as a “global Saas data platform”.

Suggested title change: How 1 data scientist scaled a data platform to millions of transactions per day, using Azure. Now you have my attention! Here we have 3 interesting hooks: 1) you don’t need a whole team, 2) significant scaling, 3) we are using Azure.

Next, I would suggest tightening your audience. Curiosity is generally a poor audience goal, because it’s so broad and doesn’t tie in to a personal need. What you have here isn’t some nifty story about how you got your arduino to tweet the moisture of your plants using Data Lake. What you have here is a case study. That is immensely valuable. There is so much hype around these technologies. You have a chance to cut through the marketing bs. That’s your core value-add here. Embrace it.

Suggested audience: People looking for real-world implementations of scalable data solutions in the cloud.

Next, your abstract is kind of meandering. It takes 3 small paragraphs before we really know what it’s about. And even then you only hint at the pain that you could be solving for us. Somewhere in the first 3 sentences you should be answering the question “Why do I care?”

The reason I care is because scaling is HARD. Scaling at startups speeds is TERRIFYINGLY HARD. The marketing around big data is all bs and you, yes YOU, are going to cut through that bs like a samurai sword through butter! You are going to show us all the little roadblocks that we didn’t think about.

This isn’t a “guys, check out what I made” story. This is a battle-hardened story of how you scale to meet the needs of immense problems. You’ve got so much potential here.

Reply

Here’s my original variant, which just got picked for DevReach

Coping with x2,000 data volumes growth with a team of one
I worked in a startup and our volumes started small, really small. I was the data scientist but I needed to actually build a data platform first. By the end of 6 months, daily volumes were 2,000 times the size.

This talk takes you through my 6 months of learning how to build a high-scale data platform whilst requirements, scale, and technology were all changing. We’ll look at the various bits of Azure I used, how they started off or became wrong for my situation, how I designed the system to cope with the constant swapping and changing of technologies, and then we’ll look at the system I ended up with that could cope with the scale and at then some.

Reply
Wes Crockett
May 9, 2017 1:22 pm

The overall idea is great and I would REALLY love to lean how to build a solution using Azure on the cheap. That being said, the abstract doesn’t give me any indication about what you will be presenting on other than building an undersized Azure instance.

Reply

Leave a Reply to Steph Locke Cancel reply

Your email address will not be published. Required fields are marked *

Fill out this field
Fill out this field
Please enter a valid email address.

Menu