Author: rubberchickn

It really is. Will try to blog some stuff on this at some point..

In this post, I’ll be talking about two of the more common parametric distributions, the Binomial and Poisson distributions. We kind of covered the Normal distribution last post so I’ll skip that for now unless I come across some world-changing info about it.

So, what are Parametric Distributions? Usually, probabilities depend on some unknown constant. For example, the probability of flipping a Heads depends on if the coin is fair or two-headed. The unknown constant in this example is called the parameter. Depending on the type of problem, you can have a different parameter, and the distributions I will mention today are the most common (probably).

The Binomial Distribution

In order for an experiment to be considered a Binomial, it has to meet a few requirements:

1. The outcome must be either a success or failure
2. Each trial must be identical
3. The probability of success is the same for each trial
4. Each trial must be independent

The Binomial Distribution is really prevalent in the real world. The most common example I’ve seen is a series of coin flips, aka “What’s the probability of getting 3 heads in 5 flips of a fair coin?”

To answer this question, we need to know three things. The probability of success, the probability of failure, and the number of ways I can get 3 heads in 5 tries. The probability of success is just 1/2 because the coin is fair, and because of this, the probability of failure is also 1/2. Now, I can write out all the different ways to get 3 heads from 5 flips, but I won’t – and I’ll use a shortcut if you remember Combinations. The number of ways to get 3 heads out of 5 flips is 5C3 = 5!/(5-3)!3! = 10. You can verify this if you want.

The Binomial Distribution is really helpful because theres an equation that gives the probability of k successes within N trials with a probability of success (parameter) of θ. Here it is:

P(X=k|θ) = (N choose k) * θ^k * (1-θ)^(N-k)

Which just means the probability of k successes in N tries with an underlying chance of success of θ = (the number of ways you get get k groupings from N) * (the probability of k successes) * (the probability of N-k failures). The last part is because in order to have 3 successes out of 5, you need to also have 2 failures.

One thing I want to mention here is there’s a special type of Binomial Distribution called the Bernoulli Distribution which is when N=1. More to come on that later but it’s worth a shout out.

The Poisson Distribution

As the Binomial Distribution deals with counting successes over a certain amount of trials, the Poisson Distribution helps to understand successes over a period of time or space. The parameter we use in the Poisson is referred to as λ, which represents the rate. This is good for estimating things like how many customers show up to a counter over an hour.

There’s a way to show it, but the Poisson equation is derived directly from the Binomial distribution. I’m not going to try to explain it. Instead, here’s the equation:

P(X=k|λ) = λ^k * e^-λ/k!

From what I’ve gathered, this equation is used to estimate λ. You can choose a bunch of different values for λ and plot the resulting probability to find out what values of λ are the most likely for the given k. If given a λ, it should be relatively straightforward to calculate a probability, but I’m not sure how someone in the real world would get to a λ. It seems kind of circular to me but that’s probably just because I still haven’t nailed down the applications of this distribution.

Regardless, the idea of trying to estimate λ given k kind of touches on Likelihood estimates which I don’t fully understand yet but it seems like a Poisson problem can get much more complicated. I guess for now, it’s just good to have a little foundation on what this is.

That’s it for now.

Chapter 1

Literally chapter 1. So we all should have a general understanding or concept of Probabilities.  It’s effectively a function that yields some value between 0 and 1. There are two types of probabilities: Discrete and Continuous. Let’s talk about these first.

Types of Probabilities

Discrete: When dealing with any data that is finite or countably infinite, that is considered discrete. This means anything that needs to be counted. Think about number of customers that sign up, or number of cars that pass by my house.

Continuous: Variables that are infinite are considered continuous. This mostly has to do with measurement. An example would be the amount of snow that falls. Did 3 inches fall, or 3.01? Or 3.001? Or 3.000001? etc.

Probability Density Function

When dealing with discrete variables, we can calculate probabilities by taking (# of successes)/(# of total outcomes). For example, what’s the probability of Heads on a fair coin? There are 2 outcomes, (Heads or Tails), and therefore, 1/2 is the probability.

However when dealing with continuous variables, it’s different. Take the snow example above. What’s the probability it will snow exactly 3 inches? Not 2.99 inches, or 3.01 inches, exactly 3? It’s pretty much 0 because it might snow close to 3 inches, but its extremely unlikely it will snow exactly 3.

This is where density functions come in. It’s basically a curve that describes some probability. You can’t read it like a normal plot though, because for any given value of x, the probability is 0. To get a probability from this, you measure an integral or area under the curve. In the snow example, we can measure the probability 2.99 to 3.01 inches which will be > 0. I’ll leave it there because I don’t feel like going into calculus. Just remember that area under the curve = probability with a pdf.

Summary

Discrete counts. Continuous measures. PDF’s aren’t a dumb file type that you can’t edit. Area under the curve. Pretty much what I took away from these sections. These are foundations for the next piece, which will go into Parametric Distributions. Specifically, the Binomial, Poisson, Normal, and Exponential Distributions. More to come.

I’d be lying if I said I didn’t need to at least touch up on a couple of really basic concepts, just to validate my assumptions. I believe the stats class I took in college pretty much focused on the normal distribution. This is just a review of basic concepts which should be pretty straight forward or familiar.

Let’s start on the few definitions that are needed to move forward: mean, median, variance, and standard deviation.

Mean

The average. Add up all the values you have and divide by the number of data points. I think everyone knows this already.

Median

The point at which 50% of the data falls below and 50% falls above.  In a set of (1,2,3,4,5), the median is 3. In a set of (1,1,4,6,8), the median is 4. And so on. This should be equal to the mean in a perfectly normal distribution. This is more valuable when dealing with distributions that aren’t normal.

Variance (σ^2)

Variance is a measure of how spread out your data is. If you have data all over the place, the variance will be high. Conversely, if you have data that’s really close together, variance is low. It’s calculated by taking the average squared distances to the mean.

In other words, take each data point, subtract the mean, and square the value. Now add all of these squared values together and divide by the number of data points and you’ll be left with your variance. This is important because of what we can derive from it which is…

Standard Deviation (σ)

It’s the square root of variance. It’s helpful because variance uses units^2 which is kind of abstract so standard deviation measures how spread out your data is in human readable units.

The Normal Distribution

AKA the bell curve.  This distribution shows up when we’re measuring things like the height of a population or the weight of a box of cereal. The most important part of this distribution is that it’s mean is in the center, and the empirical rule: nearly all of the data (99.7%) will fall within three standard deviations of the mean.

The other important piece to know about the normal distribution is that you are able to determine how much data falls within a certain standard deviation from the mean. For example, 68% of data falls within one standard deviation. 95% within two. 99.7 within three.

Z-Score

There’s this other thing called a z-score which effectively measures how many standard deviations a data point is away from the mean. So it could be 0.3 standard deviations, 1.8, etc. This is helpful because there’s this thing called a z-table that can tell you how much data falls below that z-score.

After talking to people in the industry, it seems like z-scores or z-tables aren’t really used, but I remember there was a big focus on this stuff in school so I thought it was worth a mention.

That’s the quick review of some basic concepts. I don’t think any of this should be new or unfamiliar (if it is, I don’t know what to tell you, really…). More to come.

Happy St. Patty’s Day!

The End of an Era

So the day I’ve been dreading is finally here. 4 years as an analyst and I’ve been able to skirt by with my (extremely) limited knowledge of statistics. Every now and then over the course of my professional career, a stats problem or question would come my way and I’d be able to say enough words like ‘distribution’ or ‘bias’ to squeak out of trouble before I start babbling like an idiot.

Over the past 4 years, I’ve been able to work on my skills as an analyst. Specifically, business acumen and programming. I can make a business case and support it with persuading data, dashboard like a boss, and build a slick automation that would have left 2014 me in awe. I knew how to do exactly none of these when I first joined the workforce and often impress myself with how far I’ve come.

However, I’m finding myself lacking compared to some of my peers. I’ve noticed that many of the high performers in the industry not only have business understanding AND programming, but they are well versed in statistical concepts.

That being said, I’m obviously at a disadvantage here. Which means I’ve either got to stay where I am and hope for the best, or do something to better myself and grow. So here’s the verdict : I’m going to learn stats.

By learning stats I don’t mean things like what’s a mean, median or standard deviation either. I’m talking full on probability using density functions, Bayesian Stats, Time Series and Markov Chains. I hear these terms so much and nod blankly at the person talking about them. No more. I’m gonna do it, and I’m going to do it in two months.

Why two months? Summer’s right around the corner and once that warm weather hits I’ll have zero motivation to stay inside and bury my face in equations and strange symbols. I want to golf and grill and hike and camp and cruise with the windows down.

Alas is what brings me to the point of this blog. I’ve tried learning stats before. It’s not easy. The concepts are difficult and retention requires practice. I’m going to take what I learn each {time period TBD} and summarize it here so future me can go back and remember it without relearning. If you’re interested in Data Science or Analytics, maybe you’ll find it interesting… eh, maybe not.

Either way, it’s gonna happen.