Modeling with Data: the blog

Why I don't call software 'technology'

Ben Klemens — 9 June 2017 00:00:00 +0000

Over the course of the last hundred posts, I published a C textbook, so I talked a lot about computing technique. I'm hoping to go back to more statistics and pure(r) math in the next hundred.

Git subtrees

Ben Klemens — 25 May 2017 00:00:00 +0000

That thing where you mean to write six paragraphs on something and you have to break it up to a two-part series, it got so long.

Git submodules

Ben Klemens — 23 May 2017 00:00:00 +0000

The math I've been doing has not been especially blog-worthy, so here's more about git.

Platforms for reliable research

Ben Klemens — 23 April 2017 00:00:00 +0000

I like the idea of using a Twitter thread for comments.

D3: a travelogue

Ben Klemens — 29 August 2016 00:00:00 +0000

I broke my promise from last time and included a visualization.

Dataviz packages: a travelogue

Ben Klemens — 28 August 2016 00:00:00 +0000

So named because I really do feel like a tourist in these parts, and because this post is about the general structures I happened to observe, not specific how-tos.

Murphy bed projects

Ben Klemens — 8 August 2016 00:00:00 +0000

This is a post about my workflow, which seems to be different from the norm. It comes with a bonus script to facilitate life with git, and provides a contrast to the last post about revision control as narrative—this post is about revision control for just plain archiving.

Version control as narrative device

Ben Klemens — 1 August 2016 00:00:00 +0000

Here is a post on why people use Git even though it is inscrutable. While I was away, I posted a series on tax policy on my other blog and, fwiw, an essay on listening to classical music. I hope to get back to posts with math in them very soon.

Apophenia v1.0

Ben Klemens — 6 December 2015 00:00:00 +0000

A package announcement; kind of a big deal for me (and I hope for others). The first draft of this post was originally dated 1 January 2014.

HTTPS: the end of an era

Ben Klemens — 7 May 2015 00:00:00 +0000

I posted a nontechnical personal essay on medium.com, which you may be interested in reading. By `nontechnical' I mean that it only uses Bayes's rule once.

Editing survey data (or, how to deal with pregnant men)

Ben Klemens — 7 June 2015 00:00:00 +0000

In which I discuss the eminently unglamorous problem of cleaning data. It would be nice if we could assume it away, but done wrong, it can ruin your results.

Banning the hypothesis test

Ben Klemens — 12 March 2015 00:00:00 +0000

I try to avoid commenting on things everybody else has already commented on, but my take seems to be a little different from most.

Overlapping bus routes

Ben Klemens — 3 March 2015 00:00:00 +0000

In which we crunch some General Transit Feed Specification data, and do good-enough GIS with a stock database engine.

m4 without the misery

Ben Klemens — 23 January 2015 00:00:00 +0000

Hopefully the last I'll ever explicitly write about m4, but it's great for all kinds of applications. The XML for a single entry in this RSS feed is produced via an m4 macro.

A version control tutorial with Git

Ben Klemens — 3 January 2015 00:00:00 +0000

Just another 13-page blog entry.

Why I capitalize distribution names

Ben Klemens — 5 November 2014 00:00:00 +0000

It's a philosophy of science post, obviously.

The formalization of the conversation in a social network

Ben Klemens — 4 November 2014 00:00:00 +0000

Next time your not-academic friends talk about journals as a Gold Standard, you can give them this essay.

The difference between Statistics and Data Science

Ben Klemens — 30 October 2014 00:00:00 +0000

A plea for civility which, like all pleas for civility, will probably be ignored.

Bayes v Kolmogorov

Ben Klemens — 23 October 2014 00:00:00 +0000

Or, how I think about Bayesian updating. I wrote this up because I get the sense that this isn't how most people think about it.

Microsimulation games, table top games

Ben Klemens — 14 July 2014 00:00:00 +0000

Every Intro to Game Theory class starts with an admonishment that Game Theory is not about Monopoly or Chess. Let's suspend that for a minute and ask what modelers can learn from the designers of Monopoly and Chess.

Intercoder agreement: the R script and its tests

Ben Klemens — 12 July 2014 00:00:00 +0000

Sort of part IV of the series from a few months ago, sort of something I'd been meaning to post in case somebody finds it to be useful. Next time will be a lot more fun.

Testing statistical software III: the contract

Ben Klemens — 9 May 2014 00:00:00 +0000

It's not just a set of assertions--it's a way of life.

Testing statistical software II: there's a theorem somewhere

Ben Klemens — 6 May 2014 00:00:00 +0000

This is the part that the coding textbooks don't cover because they're not aimed at people writing numeric routines, and that the math textbooks don't cover because they avoid coding technique.

Testing statistical software I: units

Ben Klemens — 3 May 2014 00:00:00 +0000

A sort of warm-up for next time, in which I remind the reader of the benefits of assertions.

Regular expression parsing in C

Ben Klemens — 9 March 2014 00:00:00 +0000

This post may primarily be useful for the utility functions at the end. If you want a regular expression tutorial (without C), see Appendix B of _Modeling with Data_.

The one line of C code that will make your program ten times faster

Ben Klemens — 19 February 2014 00:00:00 +0000

It's a link-bait title promising quick solutions, but it links to a 25-page essay covering everything you need to know about OpenMP, pthreads, and C atomic variables.

A short asprintf implementation

Ben Klemens — 17 February 2014 00:00:00 +0000

It's going to be a few C-oriented posts for a few entries.

A Julia wrapper for Apophenia

Ben Klemens — 22 December 2013 00:00:00 +0000

Or, the payoff we get from a host language that understands C structs.

On Julia

Ben Klemens — 9 December 2013 00:00:00 +0000

To summarize why I'm willing to try yet another language: it just worked.

Another two-line Logit analysis

Ben Klemens — 7 October 2013 00:00:00 +0000

At its core, this post is another demo of a simple thing to do with the Apophenia library. There are reasonable odds that we'll have a version of this via Julia next time.

Interrater agreement via entropy

Ben Klemens — 26 September 2013 00:00:00 +0000

Looking through the textbooks, I get the sense that information theory was poised to revolutionize statistics in the 1960s and `70s. I don't know why it didn't catch on.

Interrater agreement

Ben Klemens — 22 September 2013 00:00:00 +0000

A brief introduction to interrater agreement measures, with a focus on the problem of defining the rate of agreement at random.

Making a data structure comfortable

Ben Klemens — 9 September 2013 00:00:00 +0000

I could have stretched this out for a few more posts, but I've been wanting to get to interrater agreement, which has been on my to-do list for months.

Dealing with arrays of text

Ben Klemens — 30 August 2013 00:00:00 +0000

Handling arrays of text is not challenging if you have the right tools. If you need a copy of asprintf, btw, there's one in the Apophenia code base.

Vector or matrix

Ben Klemens — 26 August 2013 00:00:00 +0000

It's not the aggressively minimalist solution, but there's more to life than minimalism. Next time, text.

Mapping functions

Ben Klemens — 21 August 2013 00:00:00 +0000

You can get C to be rather LISPy with some use of macros and functions like these.

Interacting with C code

Ben Klemens — 17 August 2013 00:00:00 +0000

In which I use the first person frequently to describe how I spend most of my time as a wage-earning statistician. I've tried lots of alternative setups, and this is the one that I wind up using most often.

De-crosstabbing

Ben Klemens — 16 August 2013 00:00:00 +0000

A short interstitial post with a simple example, in emulation of a short post from the World Bank's open data group.

The data set: views of a C structure

Ben Klemens — 13 August 2013 00:00:00 +0000

Or, how matrix structures are commonly structured.

Another C macro trick: parenthesized arguments

Ben Klemens — 9 August 2013 00:00:00 +0000

Will get back to data-crunching next time. I get enthused when I learn about another way to improve UI, and had to write it up.

A Logit and some elasticities

Ben Klemens — 5 August 2013 00:00:00 +0000

I always start by promising that this time, I will write something short and simple, and I always somehow get carried away.

Do what I mean

Ben Klemens — 4 August 2013 00:00:00 +0000

DWIM features often engender a lot of affection from novice users, though professional programmers tend to despise them. I can spend all day picking at Matlab, a mechanical language that does little automagically, and nobody really cares. But when I write about these sorts of problems in R, I have good odds of getting hate mail from R fans.

A test for independence

Ben Klemens — 1 August 2013 00:00:00 +0000

These sample scripts are correctly proportioned: I think for the typical real-world analysis, most of the work is in data prep, followed by one or two lines pulling the trigger on the statistical routine. This is language-independent, because data starts off sloppy and in not quite the right format regardless of language.

Let us not forget the basics

Ben Klemens — 28 July 2013 00:00:00 +0000

I am so very enthused by the future of statistical modeling that I forget that there are about seven billion people out there who have no idea what I'm talking about.

Uncertain ABM settings

Ben Klemens — 24 July 2013 00:00:00 +0000

I'm having a good time writing up a new model and analysis every three days. This is the most code-intensive post, because it's the most complete agent-based model. Next episode will be entirely code-free.

Bayesians and ABMs

Ben Klemens — 21 July 2013 00:00:00 +0000

In which I argue that if you call yourself a Bayesian, you'll love microsimulation models.

An ABM in the box

Ben Klemens — 18 July 2013 00:00:00 +0000

This is really the denouement of the series.

RNG-based likelihoods

Ben Klemens — 15 July 2013 00:00:00 +0000

This is the last piece of machinery we need for using microsimulations as distributions. It is frankly the least pleasant part.

The model asserts the likelihood

Ben Klemens — 12 July 2013 00:00:00 +0000

My intent here is to cover all of the objections people have to doing hypothesis tests with models they didn't learn in grad school.

The setup

Ben Klemens — 10 July 2013 00:00:00 +0000

A navel-gazing entry, as a break from more technical material.

Hierarchies and chains

Ben Klemens — 9 July 2013 00:00:00 +0000

This is the denoument of the narrative about transformations on simple models

A sample transformation: truncation

Ben Klemens — 6 July 2013 00:00:00 +0000

I dislike pseudocode, so here's a fully functioning example of standardized model forms and their transformations. Next time, I'll be back to less coding-detail-oriented considerations.

Transforming models

Ben Klemens — 2 July 2013 00:00:00 +0000

I think this paradigm goes a long way toward better modeling. If I were to write a stats textbook today, it'd be centered around this.

What goes into a model?

Ben Klemens — 29 June 2013 00:00:00 +0000

Posted this yesterday; revamped the blogging system to produce much more attractive output today. If your browser has trouble with the math or code, don't forget that there's always a PDF version linked from the head of every entry.

Uniform model forms and microsimulations

Ben Klemens — 26 June 2013 00:00:00 +0000

My outline for this series has fifteen parts, so this will take roughly a month and a half.

Older posts

Ben Klemens — 25 June 2013 00:00:00 +0000

If you are reading through the post archive, see also the older RSS entries here