The Weekly Blip: August 13th 2023

How to best measure LLM performance + Getting copied & kicked by Shopify

It can be hard to measure the performance of language models with datasets - because often those same datasets are used in their training.

There are some off-beat tests that do better:

1. Sally (a girl) has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have?

2. Find the 15th last letter in the sequence:

   HCDHCDDCCHDHHDHDCHHCDHDCHHCDHHCHDCHHDC…