Modeling with Data: the blog

Emulating others and fat-tailed distributions

Ben Klemens — 4 April 2013 00:00:00 +0000

In my ideal world, every academic paper would have an accompanying blog entry discussing the context and motivation.

My coworkers

Ben Klemens — 22 May 2013 00:00:00 +0000

It's like Follow Friday but on a blog. These are all super-informative writers.

Bayesians, Frequentists, and why I don't care

Ben Klemens — 19 January 2013 00:00:00 +0000

This is entry #143 and the first time I've seriously mentioned the dichotomy. I'm mostly doing it because I feel like you can't have a stats blog without having at least one entry on the debate.

SQLite in-memory databases; database merging

Ben Klemens — 16 January 2013 00:00:00 +0000

It's certainly a facility to be used sparingly, but I'm really liking this deprecation-via-blog idea.

Raking with structural zeros

Ben Klemens — 16 January 2013 00:00:00 +0000

Two minor technical posts today. This is the last in the series on raking, because I feel like you can't talk about raking without mentioning structural zeros.

Completing missing data via raking

Ben Klemens — 8 December 2012 00:00:00 +0000

Applying a simple technique to what would otherwise be a difficult problem.

Syntheis via raking

Ben Klemens — 25 November 2012 00:00:00 +0000

Also, I introduce many-dimensional tables.

Raking

Ben Klemens — 19 November 2012 00:00:00 +0000

Finally, a blog entry on statistical methods. I just take the computing technique required to make this work for a 2 billion cell table as given.

The book version

Ben Klemens — 17 November 2012 00:00:00 +0000

Most of my non-work brainpower has gone toward turning the last 87 blog entries into a book. Here are some notes on that, in case you're interested. Next time: actual statistical content.

End tip mode

Ben Klemens — 22 March 2012 00:00:00 +0000

In which I confess my motives as a computational epistemologist. You can do a lot of modeling without knowing any computing, expanding our range of understanding in modeling is difficult without a ton of computing.

Tip 85: How to learn a new programming language

Ben Klemens — 20 March 2012 00:00:00 +0000

I started writing this piece on 1 April 2011 (and finished it shortly thereafter). It's a fine way to conclude this tip series.

Tip 84: Use m4 to automate OOP boilerplate

Ben Klemens — 18 March 2012 00:00:00 +0000

The moral here is that m4 doesn't have to be rocket science, and is a good thing to think about when you need boilerplate.

Tip 83: Use m4 in the middle of your documents

Ben Klemens — 16 March 2012 00:00:00 +0000

Over on the other screen I'm using Asciidoc as my XML-generator. But there's a lot of pass-through XML I have to write for anything out of the ordinary.

Tip 82: insert NA, NaN, and other markers into your data set

Ben Klemens — 14 March 2012 00:00:00 +0000

I do know one person who enthuses about semaphores. He distinguishes between `valid missing' and `missing, needs impute'. That's the only example I've found so far.

Tip 81: Deprecate floats

Ben Klemens — 12 March 2012 00:00:00 +0000

It's just a little odd that where everybody else says `real' I say `double'.

Tip 80: Send in parameters fast

Ben Klemens — 10 March 2012 00:00:00 +0000

I think this is how they did it before they invented getopt.

Tip 79: Try a multiplexer

Ben Klemens — 8 March 2012 00:00:00 +0000

All I do in this entry is enthuse. The only tip is: go try one.

Tip 78: Use mmap for gigantic data sets

Ben Klemens — 6 March 2012 00:00:00 +0000

I was inspired to write this by a seminar where a professor espoused the joys of his R package which handles gigantic data sets, and how unfettered research can be with his new innovation. It was an hour-long sell, and afterward, I finally got him to explain how he makes it work: "I used mmap."

Tip 77: Attach databases for greater speed

Ben Klemens — 4 March 2012 00:00:00 +0000

SQLite has spoiled me on how easy databases can be.

Tip 76: Bootstrap!

Ben Klemens — 2 March 2012 00:00:00 +0000

I used to feel bad about using the bootstrap so much--it just seemed too easy. But it's a great computational method, because it's so easy.

Tip 75: Compile C for Windows

Ben Klemens — 29 February 2012 00:00:00 +0000

Srsly, write me if you know of anything that isn't from Microsoft and isn't POSIX-compliant at some level.

Tip 74: Use Unicode libraries

Ben Klemens — 27 February 2012 00:00:00 +0000

This has to be a relatively incomplete entry, because there's an infinite range of things you could be doing with text. But I'll bet you that the plain string-handling routines on UTF-8 will cover a lot of your needed ground.

Tip 73: Store Unicode in UTF-8

Ben Klemens — 25 February 2012 00:00:00 +0000

There are a lot of articles that try to keep some sort of neutral point of view about encodings, but the more I read, the less I saw any point in the other options. Ask your search engine for `UTF-16 considered harmful'.

Tip 72: Export one-off variables

Ben Klemens — 23 February 2012 00:00:00 +0000

I can't believe I'm up to Tip 72.

Tip 71: Don't confuse pointer declarations with pointer uses

Ben Klemens — 21 February 2012 00:00:00 +0000

I've been getting long-winded lately, so here's a short one. Note to textbook authors: is pointing this out really all that difficult?

Tip 70: Parse text with strtok

Ben Klemens — 19 February 2012 00:00:00 +0000

Simple, useful, quirky, and often overlooked in favor of much heavier solutions to the problem. Regular expressions are too much machinery, and anything with `parser' in the name is one step too high in the hierarchy.

Tip 69: Use a makefile for everything

Ben Klemens — 17 February 2012 00:00:00 +0000

Think of a makefile as a memo pad for procedures.

Tip 68: Write literate code

Ben Klemens — 15 February 2012 00:00:00 +0000

It really does feel different. Probably not for everybody, but some of you will love it.

Tip 67: Document your code with Doxygen

Ben Klemens — 13 February 2012 00:00:00 +0000

At ~114 lines, this is the longest example I've ever written (though it's under 40 lines if you cut the documentation). It's interesting to see the contrast between cURL's easy interface and libxml2's less friendly, more traditional interface.

Tip 66: Protect threaded resources with mutexes

Ben Klemens — 11 February 2012 00:00:00 +0000

At this point, we've covered the 10% of the syntax that does 90% of the threading work. However, many a dissertation has been written about that last 10%; proceed cautiously.

Tip 65: Easy threading with Pthreads

Ben Klemens — 10 February 2012 00:00:00 +0000

It's not the syntax that makes threading hard.

Tip 64: Consistency-check external variables

Ben Klemens — 8 February 2012 00:00:00 +0000

I wrote this two days ago, after way too much time treating the wrong bug.

Tip 63: Seamlessly extend C structures

Ben Klemens — 5 February 2012 00:00:00 +0000

Although it has its own annoyances, this solves so many annoyances of the standard inheritance-by-inclusion that it is very worth it.

Tip 62: Get to know configure.ac

Ben Klemens — 3 February 2012 00:00:00 +0000

Autoconf: an amazing feat of dealing-with-reality engineering.

Tip 61: Get to know Makefile.am

Ben Klemens — 1 February 2012 00:00:00 +0000

I've written packages for scripting languages whose requirements were much more onerous than this.

Tip 60: Package your code with Autotools

Ben Klemens — 30 January 2012 00:00:00 +0000

It took me a long time and a lot of pain to work all this out. The documentation is consistently fragmented across very interconnected components.

Tip 59: Use a package manager

Ben Klemens — 28 January 2012 00:00:00 +0000

This is a part of the world that is especially in flux. Nonetheless, all of these are awesome, as are the people who do the work to maintain all of these systems.

Tip 58: Destroy your inputs

Ben Klemens — 26 January 2012 00:00:00 +0000

I ended this one with a question mark, because I would like your input. Any other amusing tips I've missed so far? Feel free to email or comment.

Tip 57: base your code on pointers to objects

Ben Klemens — 23 January 2012 00:00:00 +0000

In which I answer a question I am frequently asking myself. This is about making the transition from a small and cuddly project to a larger one where you really have to start worrying about maintainability.

Tip 56: Enums---don't bother

Ben Klemens — 21 January 2012 00:00:00 +0000

Seriously, please stop yelling.

Tip 55: Mark input pointers with const

Ben Klemens — 19 January 2012 00:00:00 +0000

The Game Theorists would call this a first mover problem.

Tip 54: Put functions in your structs

Ben Klemens — 17 January 2012 00:00:00 +0000

Although such mutability is the norm in C++ and Java, I always thought it should be used more sparingly.

Tip 53: Count references

Ben Klemens — 15 January 2012 00:00:00 +0000

A variant on last time, and a still fuller example program. It's hard to demonstrate the synthesis of a technique in tweet-length tips, so I felt this sort of thing necessary. I like this model, and think of it as more than just a toy.

Tip 52: Mark the owner of your data

Ben Klemens — 13 January 2012 00:00:00 +0000

The point is pretty simple, and I don't know how often you have to deal with views of a large data set, so I used this as a chance to give a full, worked example of an object and its associated methods implemented in C. I may elaborate on or modify this a little later, such as moving the methods into the struct itself.

Tip 51: Construct and destruct your structs

Ben Klemens — 11 January 2012 00:00:00 +0000

I was thinking of pausing after that last tip, but let us press on. The next few items borrow the most relevant parts of the object-oriented world.

Tip 50: optional and named arguments

Ben Klemens — 9 January 2011 00:00:00 +0000

This is the 100th post on the blog. How time flies. I'm already repeating myself---I'd given two or three pages of exposition on how I implement these named arguments in Apophenia. Used sparingly, repetition is not the worst thing that a blog can do, I suppose.

Tip 49: return multiple items from a function

Ben Klemens — 7 January 2011 00:00:00 +0000

I refer to these as throwaways, but they don't have to be. You may easily find that the structure you're returning is actually a natural way to express things, and has expressive utility beyond just being a container.

Tip 48: Limit typedef scope

Ben Klemens — 5 January 2011 00:00:00 +0000

I open with a lot about style today because this stuff is often misused, and so as a result a lot of people reject the entire concept of any of this--but that's also pernicious. The next few episodes are gonna be awesome if you're willing to have a definitions section at the top of a file.

Tip 47: Static versus global variables

Ben Klemens — 3 January 2011 00:00:00 +0000

The awkward thing about how C requires that variables be exported via a separate header file turns out to be useful, giving us a way to have variables that aren't specific to one function but aren't really global. It's subjective, but I think this level of scope is underused.

Tip 46: Typedefs save the day

Ben Klemens — 1 January 2011 00:00:00 +0000

This is a variant of the rule that we should write our code for human legibility. A good typedef, like a good structure, describes how we think of the problem.

Tip 45: Structures get copied

Ben Klemens — 30 December 2011 00:00:00 +0000

Some textbooks seem to forget to mention this. K&R mention it in passing ("The only legal operations on a structure are copying it or assigning to it as a unit, taking its address with &, and accessing its members."). So I felt it worth worth pointing out.

Tip 44: Brevity is the soul of incomprehensibility

Ben Klemens — 28 December 2011 00:00:00 +0000

I see so many polemics online that take as a truism that getting code to be more compact is always an improvement. Srsly?

Tip 43: wrap substructures in parent structures

Ben Klemens — 26 December 2011 00:00:00 +0000

This is as much a tip for struct authors as struct users: you could make using your struct really easy.

Tip 42: Don't bother with union

Ben Klemens — 24 December 2011 00:00:00 +0000

So yes, there are reasons to use this keyword, but they're still pretty obscure, and if you never knew it existed you'd still get by.

Tip 41: Initialize arrays and structs to zero

Ben Klemens — 22 December 2011 00:00:00 +0000

Our first payoff from the rules about initializating compound types.

Tip 40: Designated initializers

Ben Klemens — 20 December 2011 00:00:00 +0000

I don't think I'm exaggerating when I say that designated initializers changed my life. I would've given up on C a long time ago if I hadn't known about them, while having them on hand has made the language fun.

Tip 39: Know the constraints of C structs

Ben Klemens — 18 December 2011 00:00:00 +0000

This one is exposition, but may save you a lot of banging your head against things when you switch languages.

Tip 38: Use Valgrind to check for errors

Ben Klemens — 16 December 2011 00:00:00 +0000

I don't use Valgrind often, but there are lots of situations where it is essential.

Tip 37: Rename things with pointers

Ben Klemens — 14 December 2011 00:00:00 +0000

This is my question #1 when I'm learning a new language.

Tip 36: Try a new shell

Ben Klemens — 12 December 2011 00:00:00 +0000

The point of this post is to express the variety of things users of the default shell are missing. So if you're a fan of a different shell than the one I enthused about, please tell us about your own favorite features in the comments.

Tip 35: Use the shell to test for files

Ben Klemens — 10 December 2011 00:00:00 +0000

There's a lot more to these conditionals, but there are so very many caveats...

Tip 34: Use the shell's for loops to operate on a set of files

Ben Klemens — 8 December 2011 00:00:00 +0000

I use this on the command line all the time. However, there's more than one way to do it (alternatives often involve the find and xargs commands).

Tip 33: Replace shell commands with their outputs

Ben Klemens — 6 December 2011 00:00:00 +0000

Perl users, now you know where it comes from.

Tip 32: Get to know your shell

Ben Klemens — 4 December 2011 00:00:00 +0000

Shell scripting is especially byzantine. It's a macro language, so the usual surprises about text substitution come up. It's an especially good example of a syntax which evolved as hundreds of hackers threw in a trick that seemed like it would be useful, and then one day the POSIX standard codified all these hacks. Now, post-standard, shell authors are especially comfortable not conforming to the standard anyway and adding new features. So it's best to pick up the most useful and reliable tricks, but do any work requiring more than a few straightforward lines to C, Python, Perl, et al.

Tip 31: Use the database for configuration info

Ben Klemens — 2 December 2011 00:00:00 +0000

7 lines of macros + 2 lines of C code = a complete configuration system.

Tip 30: Use Apophenia to read in data and configuration info

Ben Klemens — 30 November 2011 00:00:00 +0000

As this progresses, the tips will be more focused on scientific and statistical computing.

Tip 29: Preprocessor tricks!

Ben Klemens — 28 November 2011 00:00:00 +0000

For the Apophenia fans, this is how Apop_row, Apop_data_col, &c. produce disposable views.

Tip 28: Vectorize a function

Ben Klemens — 26 November 2011 00:00:00 +0000

This is another variant on the trick from the last tip or two, but is the variant you're most likely to use every day.

Tip 27: Foreach in C

Ben Klemens — 24 November 2011 00:00:00 +0000

This is not Fight Club: tell your naysaying, C-skeptical friends about the series. There's still a long list of amusing tricks in the queue.

Tip 26: Safely terminated lists

Ben Klemens — 22 November 2011 00:00:00 +0000

List processing in C.

Variadic macros

Ben Klemens — 20 November 2011 00:00:00 +0000

It's the little things that make our lives easier.

Compound literals

Ben Klemens — 18 November 2011 00:00:00 +0000

In which we have turned the corner, and begin introducing a few elements of C that you've probably never heard of. I did check a few common, recent C textbooks, and my claim that most of them don't mention this stuff (sadly) checks out.

Tip 23: The diference between char* and char[]

Ben Klemens — 16 November 2011 00:00:00 +0000

The sizeof operator is really an anomaly in the language.

Tip 22: all the casting you'll need

Ben Klemens — 14 November 2011 00:00:00 +0000

Soon (but not next time), I will stop writing about what the textbooks teach you but you don't need to waste time with, and get to the things that are worth knowing but that the textbooks didn't include.

Tip 21: become a better typist

Ben Klemens — 12 November 2011 00:00:00 +0000

This is sort of like having a good makefile, but much more fundamental. So take some time today to check your posture and check your typing technique.

Tip 20: get to know static, automatic, manual memory

Ben Klemens — 10 November 2011 00:00:00 +0000

The static type is especially of historical interest at this point. Except for the trick from Tip 19, merging static in with auto would only lift restrictions and eliminate annoyances.

Tip 19: define persistent state variables

Ben Klemens — 8 November 2011 00:00:00 +0000

I use Staticdef all the time, and not just because it's a B short of being the first six letters of the alphabet.

Tip 18: Declare arrays when you know their size

Ben Klemens — 6 November 2011 00:00:00 +0000

Here's Born in the 70s, by the Fruit Bats.

Tip 17: Define a string type

Ben Klemens — 4 November 2011 00:00:00 +0000

At this point, we've gone a long way toward fixing C strings, and we used C to do it.

Tip 16: All the pointer arithmetic you need to know

Ben Klemens — 2 November 2011 00:00:00 +0000

Have you noticed how many of my tips for using C are about using less of it?

Tip 15: get GDB to print your structures

Ben Klemens — 31 October 2011 00:00:00 +0000

Everybody deals with different data structures, so we all have to costomize our tools a little.

Tip 14: easier interrogations with GDB variables

Ben Klemens — 29 October 2011 00:00:00 +0000

This tip and the next are why I'm more comfortable using the GDB command line than the GUI front-ends.

Tip 13: Use a debugger

Ben Klemens — 27 October 2011 00:00:00 +0000

To some people, this tip is too obvious to bother with; to others it is a revelation.

Tip 12: Use asprintf to extend strings

Ben Klemens — 25 October 2011 00:00:00 +0000

Extending a string seems to be a most common operation to me, but as I started writing this I started to think that maybe it's just my own habit. Nonetheless, this third tip of three is really all you need for string production.

Tip 11: String literals

Ben Klemens — 24 October 2011 00:00:00 +0000

Next tip will come out tomorrow.

Tip 10: Use asprintf to make string handling less painful

Ben Klemens — 21 October 2011 00:00:00 +0000

There's work being done on a new C standard. I expect this function to be part of that standard, whenever it should come out. [And if it isn't, I will be very sad.]

Tip 9: Compile C programs via here script

Ben Klemens — 19 October 2011 00:00:00 +0000

This really gets fun when you're integrating into a shell script. If you're used to handling shell variables, try assigning the output of a.out to a variable [ primes=`./a.out` ] and see what kind of fun you can have with it.

Tip 8: Use here scripts

Ben Klemens — 17 October 2011 00:00:00 +0000

I feel as though pretty much every script I write these days has a here document somewhere, slipping out to another language for a quick something.

Tip 7: Include header files from the command line

Ben Klemens — 15 October 2011 00:00:00 +0000

Sometimes we pick a point on the portability-versus-fun spectrum over on the fun side.

Dennis Ritchie

Ben Klemens — 13 October 2011 00:00:00 +0000

On setting the standard.

Tip 6: Aggregate your includes

Ben Klemens — 11 October 2011 00:00:00 +0000

One less thing to think about.

Tip 5: Initialize wherever the first use may be

Ben Klemens — 9 October 2011 00:00:00 +0000

There once was a time when declarations had to be at the top of block, not mixed in as needed. Cyndi Lauper was really popular then.

Tip 4: don't bother returning zero from `main`

Ben Klemens — 7 October 2011 00:00:00 +0000 Our first tip about C syntax itself.

Tip 3: use libraries (even if your sysadmin doesn't want you to) Ben Klemens — 5 October 2011 00:00:00 +0000 This is real lore: experienced POSIX geeks just know to set up a local root, without having read it in any formal manuals. Tip 2: use libraries Ben Klemens — 3 October 2011 00:00:00 +0000 We're still stating the unstated basics. Tip 1: makefiles Ben Klemens — 1 October 2011 00:00:00 +0000 Before we get to amusing features of C syntax, the first several tips will be about compilation and environment. Tip-a-day mode Ben Klemens — 29 September 2011 00:00:00 +0000 Happy new year everybody. I'll be giving short C tips every other day for the next few weeks. Leaving SAS: the script Ben Klemens — 15 September 2011 00:00:00 +0000 Somebody had to write this. Scope in C is e-z Ben Klemens — 7 September 2011 00:00:00 +0000 This is graceful design at work. Why you should teach your stats students C Ben Klemens — 5 September 2011 00:00:00 +0000 Happy Labor Day everybody. Here's a nice, traditional language advocacy essay for you. Labels, gotos, switches, and breaks Ben Klemens — 14 August 2011 00:00:00 +0000 Me, I use break and continue from time to time, but find the switch-case setup to be too elegant for its own good. The great packaging problem--the hard part Ben Klemens — 22 May 2011 00:00:00 +0000 I could turn this blog into nothing but a list of social problems that people mistake for technical problems and be busy for a year. The great packaging problem--the easy part Ben Klemens — 16 May 2011 00:00:00 +0000 Today's episode demonstrates diverse solutions that don't really cause serious problems, and just coexist. The Balkanization comes in the next episode. Typesetting code listings Ben Klemens — 18 April 2011 00:00:00 +0000 Summary: I put far too much thought into typing up this book. Supreme Court rules against overreliance on p-values Ben Klemens — 9 April 2011 00:00:00 +0000 You know it's bad when even the lawyers think a statistical technique is too bright-line. Powerful software! Ben Klemens — 4 April 2011 00:00:00 +0000 Next time: the Supreme Court's ruling about the meaning of statistical significance. But first, I complain about how people pitch code. Structures versus dictionaries Ben Klemens — 31 March 2011 00:00:00 +0000 Continuing the thought from last time to cover how we handle heterogeneous data. Structures Ben Klemens — 24 March 2011 00:00:00 +0000 Hi again. I've mostly been distracted by work on another book on modern economics, but I'm back with a short series on commonalities across programming languages. Multiple imputation Ben Klemens — 1 October 2010 00:00:00 +0000 At this point, I basically think of any nontrivial statistical model as a combination of simpler models. Object-oriented programming in C Ben Klemens — 22 April 2010 00:00:00 +0000 Another essay about doing more with fewer keywords. Testing the model using the model Ben Klemens — 11 January 2010 00:00:00 +0000 I start by picking at a cartoonist, but this goes well beyond that in importance. Keeping paper current Ben Klemens — 9 January 2010 00:00:00 +0000 In which I put an absolutely huge amount of thought into things most folks don't really care about. Then, I exert great effort fighting against the inevitable. The statistics style report Ben Klemens — 19 December 2009 00:00:00 +0000 I like to think that I always lean toward parsimony, but I imagine others looking at my work wouldn't think so. Git status interactive Ben Klemens — 12 December 2009 00:00:00 +0000 This is a useful script for people using git; if that's not you, don't bother clicking through. I know I've been writing a lot about computing technique lately, so next time: statistics! The schism, or why C and C++ are different Ben Klemens — 21 November 2009 00:00:00 +0000 Take this as a solicitation for suggestions on how we can get Apophenia's variable-length lists of named arguments (which are entirely C standards-compliant) to work from C++. Yet another Git tutorial Ben Klemens — 8 November 2009 00:00:00 +0000 I wrote this for my coworkers, upon whom I've just dropped this whole revision control thing, because the existing tutorials didn't seem at all appropriate for them. Easy re-typing with designated initializers Ben Klemens — 1 November 2009 00:00:00 +0000 A straight-up lesson on coding technique. Some of this should have been in the texbook, but I was somehow reluctant to rely too much on designated initializers and compound literals; in retrospect, I'm not sure why. Computing history and its scars Ben Klemens — 19 October 2009 00:00:00 +0000 Although it may seem like I'm complaining about whatever glitches, in an alternate universe we'd just have different scars. The path-dependency and potentially infinite life of code means that software is inherently prone to scarring. Causal ethics Ben Klemens — 11 October 2009 00:00:00 +0000 Yes, this is a blog about statistics, and yes, this is a post about the development of ethics. Programming your blog Ben Klemens — 21 July 2009 00:00:00 +0000 When I'm having trouble with a machine of any sort, I try to think of it as a cultural problem. How were the designers thinking differently from how I'm thinking? Probability versus likelihood Ben Klemens — 2 July 2009 00:00:00 +0000 The question of whether data is plural or a singular can be reasoned through; the question here is well on its way to unanswerable. Data is typically not a plural Ben Klemens — 10 June 2009 00:00:00 +0000 This is a lead-in to next episode about another vocabulary problem that's much more difficult. Better variadic functions in C Ben Klemens — 3 June 2009 00:00:00 +0000 A new edition of the C standard will only come in the distant future (perhaps never). In the mean time, here's my best solution to what I see as the language's most glaring problem. I'm open to suggestions on streamlining it further. Alternatives to Word Ben Klemens — 30 May 2009 00:00:00 +0000 In two episodes, I'll tell you what I use for this here web site. Word and standards Ben Klemens — 22 May 2009 00:00:00 +0000 I once toyed with writing a book about computing standards, as a follow-up to my exciting book on software patents. The ergonomics of the down arrow Ben Klemens — 19 May 2009 00:00:00 +0000 This one may seem odd, because it's not a subject many people (including designers) spend much time talking about, but it can still make a big difference, and points to the value of at least having a choice. Intuition versus ease of use Ben Klemens — 15 May 2009 00:00:00 +0000 Ease of initial use is a distinct concept from ease of long-term use. Sometimes they are in happy agreement, and sometimes the first blocks the second. Views on a paradigm Ben Klemens — 11 May 2009 00:00:00 +0000 Part two of six on efficient document prep techniques that are precluded by Word's design. Don't forget: if you don't have time in front of the computer to read this web page, you can print the PDF. Why Word is a terrible program Ben Klemens — 7 May 2009 00:00:00 +0000 This is an essay from late 2005, in six parts. I'm reprinting it here because I think it's a good vehicle to get across a number of points about design of information and software. A general model object Ben Klemens — 4 May 2009 00:00:00 +0000 Last time, I talked about the important problem of balkanization in modeling; this episode is about doing something about it in code. Bringing theoretical models to data Ben Klemens — 1 May 2009 00:00:00 +0000 It's amazing how very differently people can read a phrase as simple as "Modeling with data". Operator overloading Ben Klemens — 29 April 2009 00:00:00 +0000 Sometimes human language and mathematical language fall into conflict. Making integers two-dimensional Ben Klemens — 22 April 2009 00:00:00 +0000 Or, a means of visualizing integer arithmetic. Using a program as a library Ben Klemens — 13 April 2009 00:00:00 +0000 I'd been neglecting computing in favor of statistics, so here's a useful tip for managing your code. Your genetic data---ethics Ben Klemens — 8 April 2009 00:00:00 +0000 I'll get back to this in a few episodes, but for now, the point here that the imbalance between descriptive and inferential power has ethical as well as purely research implications. Your genetic data Ben Klemens — 6 April 2009 00:00:00 +0000 Given a hundred cases and controls and their genetic markers, I can find you markers to perfectly categorize them in under two minutes. Of course, those markers provide no predictive power outside those hundred subjects. Freakophenia Ben Klemens — 3 April 2009 00:00:00 +0000 The summary of today's entry: if the research is exciting enough to be written up in the newspaper, it's probably not to be trusted. Breaking the pipeline Ben Klemens — 30 March 2009 00:00:00 +0000 Much of this one is about the requirements for the OLS coefficients to provide the line of best fit (i.e., the Gauss-Markov theorem). If you want to play along, try to recall them before you click through. E.g., must the expected value of the errors be zero? Are errors assumed to be Normally distributed? Crowdsourcing data mining Ben Klemens — 24 March 2009 00:00:00 +0000 I think that this gets to the question of what exactly is a hypothesis test. That's a question that we don't seriously have an answer to, though I may offer some comments at a later date. Dataviz Ben Klemens — 18 March 2009 00:00:00 +0000 This is cut from a presentation I gave last November (so if you have time to kill, have the right plug-ins, and want to hear my voice, here's a link). Too many tests Ben Klemens — 16 March 2009 00:00:00 +0000 Today for lunch, I had rice with wheat gluten, tomatoes, and cashews. The rice is somewhat sweet and very purple, but I can't tell you what kind it is exactly, because the package is entirely in Korean. Parallels II Ben Klemens — 13 March 2009 00:00:00 +0000 This is a continuation of last episode, discussing how technology affects the math we do. Next time, I'll switch from computing to conflicts in stats. Moore's law won't save you Ben Klemens — 8 March 2009 00:00:00 +0000 A lot of us are stuck in an optimistic early-90s mentality toward progress in computing power. Today I am a blog Ben Klemens — 6 March 2009 00:00:00 +0000 I never really know what to say in these introductions.