the key stuff they do different

fanatic discipline

this one resonates with me personally because i see it play out in my own life. it's about ruthless consistency rather than sporadic bursts of effort. my running journey illustrates this perfectly: strava's fitness score shows how i've been stuck at 36 for years, cycling through intense periods followed by inactivity. the lesson is clear - consistent, moderate effort would have yielded far better results than my pattern of sporadic intensity.

empirical creativity

innovation matters, but the best companies ground their creative decisions in data. take the story of the everest climbing team who faced a critical decision about whether to continue their summit attempt. instead of pushing forward on determination alone, they analyzed the weather data and made the difficult but correct choice to turn back - likely saving their lives in the process.

productive paranoia

according to the book, bill gates exemplified this quality during microsoft's peak years. even as the company dominated the software industry, he remained intensely focused on potential threats. his persistent concern about the internet's impact led microsoft to pivot at a crucial moment, preventing the company from becoming irrelevant as technology evolved.

these companies also share an interesting approach to innovation called "firing bullets before cannonballs." rather than making massive bets on untested ideas, they start with small experiments to validate their thinking. only after proving a concept do they commit significant resources. this is also a result of being paranoid but in a good way.

level 5 ambition

perhaps the most counterintuitive quality: the leaders of these exceptional companies rarely seemed driven by personal wealth or recognition. instead, they possessed a deeper sense of purpose that transcended individual success - a refreshing contrast to today's personality-driven business culture.

beyond business

what makes these principles particularly valuable is their applicability beyond the corporate world. my running experience isn't just a personal anecdote - it's a perfect illustration of how consistency outperforms sporadic intensity in any endeavor. whether you're building a company, developing a skill, or pursuing a personal goal, the formula remains surprisingly similar: maintain discipline, use data to guide decisions, stay vigilantly aware of risks, and connect your efforts to a larger purpose.

the real insight isn't that these companies do anything revolutionary. instead, they excel by executing fundamental principles with exceptional consistency and purpose. it's a reminder that sustainable success often comes from doing the basics extraordinarily well rather than chasing the next breakthrough innovation.

Problem

The task is to implement a key/value server with the following specifications:

1. The server runs on a single machine and maintains an in-memory map of key/value pairs.

2. It should support three types of operations:

   • Put(key, value): Installs or replaces the value for a given key

   • Append(key, arg): Appends arg to the existing value of the key and returns the old value

   • Get(key): Retrieves the current value for the key

3. We should also build a client that can be used to calls the server to execute these operations.

4. Each operation should be executed exactly once, even in the presence of network failures.

5. Edge cases:

   • Get on a non-existent key should return an empty string.

   • Append to a non-existent key should treat the initial value as an empty string.

Solution

Building a first version of this without caring too much about client retries and exactly once execution is pretty simple. We keep a map in memory and make changes to the map as needed.

type KVServer struct {
	mu            sync.Mutex
	Entries       map[string]string
}

func (kv *KVServer) Get(args *GetArgs, reply *GetReply) {
	kv.mu.Lock()
	defer kv.mu.Unlock()

	if val, exists := kv.Entries[args.Key]; exists {
		reply.Value = val
		return
	}
	reply.Value = ""
}

func (kv *KVServer) Put(args *PutAppendArgs, reply *PutAppendReply) {
	kv.mu.Lock()
	defer kv.mu.Unlock()

	kv.Entries[args.Key] = args.Value
	reply.Value = args.Value
}

func (kv *KVServer) Append(args *PutAppendArgs, reply *PutAppendReply) {
	kv.mu.Lock()
	defer kv.mu.Unlock()

	oldValue, exists := kv.Entries[args.Key]
	if exists {
		kv.Entries[args.Key] = oldValue + args.Value
		reply.Value = oldValue
	} else {
		kv.Entries[args.Key] = args.Value
		reply.Value = ""
	}
}

func MakeKVServer() *KVServer {
	return &KVServer{
		Entries: make(map[string]string),
	}
}

The client then makes RPC calls to the server. The lab provides helpers that handle running the server and provide methods for making RPC calls, so this task is also relatively easy.

func MakeClerk(server *labrpc.ClientEnd) *Clerk {
	ck := new(Clerk)
	ck.server = server
	return ck
}

// fetch the current value for a key.
// returns "" if the key does not exist.
// keeps trying forever in the face of all other errors.
//
// you can send an RPC with code like this:
// ok := ck.server.Call("KVServer.Get", &args, &reply)
//
// the types of args and reply (including whether they are pointers)
// must match the declared types of the RPC handler function's
// arguments. and reply must be passed as a pointer.
func (ck *Clerk) Get(key string) string {
	args := GetArgs{Key: key}

	reply := GetReply{}
	ok := ck.server.Call("KVServer.Get", &args, &reply)
	if ok {
		return reply.Value
	}
        return ""

}

// shared by Put and Append.
//
// you can send an RPC with code like this:
// ok := ck.server.Call("KVServer."+op, &args, &reply)
//
// the types of args and reply (including whether they are pointers)
// must match the declared types of the RPC handler function's
// arguments. and reply must be passed as a pointer.
func (ck *Clerk) PutAppend(key string, value string, op string) string {
	rpcName := "KVServer." + op

	args := PutAppendArgs{Key: key, Value: value}

	reply := PutAppendReply{}
	ok := ck.server.Call(rpcName, &args, &reply)
	if ok {
		return reply.Value
	}
        return ""

}

func (ck *Clerk) Put(key string, value string) {
	ck.PutAppend(key, value, "Put")
}

// Append value to key's value and return that value
func (ck *Clerk) Append(key string, value string) string {
	return ck.PutAppend(key, value, "Append")
}

The only thing that remains is to add some retries and make the server not execute the same request twice. To do this, we start sending a Client ID and a Request ID to the server with each request. For the purposes of the lab, whenever we initialize a new client, we give it a random number as the client ID. For each client, request IDs start with zero and go up by one on each request made.

Here’s what the Get and PutAppend methods looks like after we add retries (with sleeps of 100ms between each retry).

func (ck *Clerk) Get(key string) string {
	args := GetArgs{Key: key, ClientId: ck.clientId, RequestId: ck.requestId}
	ck.requestId++

	for {
		reply := GetReply{}
		ok := ck.server.Call("KVServer.Get", &args, &reply)
		if ok {
			return reply.Value
		}

		time.Sleep(100 * time.Millisecond)
	}
}

func (ck *Clerk) PutAppend(key string, value string, op string) string {
	rpcName := "KVServer." + op

	args := PutAppendArgs{Key: key, Value: value, ClientId: ck.clientId, RequestId: ck.requestId}
	ck.requestId++

	for {
		reply := PutAppendReply{}
		ok := ck.server.Call(rpcName, &args, &reply)
		if ok {
			return reply.Value
		}

		time.Sleep(100 * time.Millisecond)
	}

}

After this, all that remains is making sure the server tracks these request and client IDs and does not execute something multiple times while replying to same request (if the client multiple requests). To track this, we add two maps to the server:

type KVServer struct {
	mu            sync.Mutex
	Entries       map[string]string
	lastRequestId map[int64]int64  // key: clientId, value: lastRequestId
	appendReplies map[int64]string // key: clientId, value: last reply for Append
}

The lastRequestId map tracks the last request that the server successfully replied to for each client. Then if we get a request that has an ID lesser than the lastRequestId, we know that we’ve already executed that request and return data instead of executing the request again.

func (kv *KVServer) requestAlreadyDone(clientId int64, requestId int64) bool {
	lastRequestId, exists := kv.lastRequestId[clientId]
	if !exists {
		return false
	}
	return requestId <= lastRequestId
}

func (kv *KVServer) Put(args *PutAppendArgs, reply *PutAppendReply) {
	kv.mu.Lock()
	defer kv.mu.Unlock()

	if kv.requestAlreadyDone(args.ClientId, args.RequestId) {
		reply.Value = kv.Entries[args.Key]
		return
	}

	kv.Entries[args.Key] = args.Value
	reply.Value = args.Value
	kv.lastRequestId[args.ClientId] = args.RequestId
}

We ned the appendReplies map for one specific reason. The Append operation returns the value of the key BEFORE the new value is appended to it. We need to use appendReplies to track this, because this old data will not be available in the Entries map after we’ve executed the request.

func (kv *KVServer) Append(args *PutAppendArgs, reply *PutAppendReply) {
	kv.mu.Lock()
	defer kv.mu.Unlock()

	if kv.requestAlreadyDone(args.ClientId, args.RequestId) {
		reply.Value = kv.appendReplies[args.ClientId]
		return
	}

	oldValue, exists := kv.Entries[args.Key]
	if exists {
		kv.Entries[args.Key] = oldValue + args.Value
		reply.Value = oldValue
	} else {
		kv.Entries[args.Key] = args.Value
		reply.Value = ""
	}

	kv.lastRequestId[args.ClientId] = args.RequestId
	kv.appendReplies[args.ClientId] = reply.Value
}

That’s it! I ran the provided tests using `go test` to make sure the code does what it’s supposed to.

Fin

If you read until here, thank you. This was not the most complicated assignment, but I figured I shouldn’t skip any assignment. Next up is Lab 3, which involves implementing Raft, which should require much more depth. See you then!

As always, if you liked this post and would like to see more, please let me know by leaving a comment. You can also support me by supporting my personal project, blunderfest.

Subscribe now

Share

Share newsletter.param.codes

In this post, I’ll go through the first lab, where the assignment was to implement MapReduce in Go.

What is MapReduce?

MapReduce is a very popular way to run parallel jobs for processing large datasets easily. The original paper by Jeff Dean and Sanjay Ghemawat is worth a read, but I’ll quickly go through the details and then discuss my implementation.

Let’s understand the programming model using the example from the paper: word counts. Our task is to count the occurrences of each unique word across a large collection of text files. To do this, we define a map function and a reduce function that we’ll pass to the MapReduce framework.

Map function

type KeyValue struct {
	Key   string
	Value string
}


func Map(filename string, contents string) []KeyValue {
	// function to detect word separators.
	ff := func(r rune) bool { return !unicode.IsLetter(r) }

	// split contents into an array of words.
	words := strings.FieldsFunc(contents, ff)

	kva := []mr.KeyValue{}
	for _, w := range words {
		kv := mr.KeyValue{w, "1"}
		kva = append(kva, kv)
	}
	return kva
}

The Map function is called once for each file of input. The first argument is the name of the input file and the second is the file’s contents. The return value is a slice of key-value pairs, with the word as the key and “1” as the value.

Reduce function

func Reduce(key string, values []string) string {
	// return the number of occurrences of this word.
	return strconv.Itoa(len(values))
}

The Reduce function is called once for each key generated by the map tasks, meaning that it’s called once for each unique word. The second parameter to the Reduce function is a slice of ALL the values that the map function generated for the key. So to count the number of times the word occured in all our text files, we can just return the length of the slice passed as the second parameter.

How MapReduce uses these functions

We provide these functions to the MapReduce framework. Then, the framework has two types of processes: a co-ordinator and a worker. Tasks are completed like this:

1. The co-ordinator breaks the input into M splits.

2. It then passes each split to a worker that calls map on the split. The intermediate key-value pairs returned by the map function call are written to disk in R splits (R is the number of reduce workers). This was the Map phase.

3. Once all the map tasks are done, the co-ordinator starts assigning reduce tasks. When a worker is assigned a reduce task by the co-ordinator, it reads all the intermediate data required for the task, sorts it by the intermediate keys so that all occurrences of the same key are grouped together.

4. After sorting, the reduce worker iterates over the sorted intermediate data and for each unique key, calls the reduce function with the key and all the values that the key has. The output is then written to disk. This was the Reduce phase.

5. When all map and reduce tasks are done, the workers are stopped and the user program can be told that the task has been completed.

A diagram can explain the flow a little better.

Implementation

(Disclaimer: this isn’t the cleanest code and there is lots of room for improvement, but for a first cut I’ve tried to keep it understandable.)

The code for all this can be found in `coordinator.go` and `worker.go`. For now, since it’s only for educational purposes, we assume that both the coordinator and the workers run on the same machine.

Coordinator

First we define a Task and a Coordinator.

type TaskStatus int

const (
	NotStarted TaskStatus = iota
	Assigned
	Completed
)

type Task struct {
	taskStatus TaskStatus
	assignedAt time.Time
}

type Coordinator struct {
	mu          sync.Mutex
	Files       []string
	MapTasks    []Task
	ReduceTasks []Task
	done        chan struct{}
}

A task tracks its current status and when it was assigned to a worker. The assignedAt field is used by the coordinator to see if some worker has not done the task in a while and re-assign to some other worker if needed.

The coordinator has the following fields:

1. mu: Mutex to ensure thread-safe access to shared data like the MapTasks and ReduceTasks slices

2. Files: The input files we’ll process.

3. MapTasks: A slice to track all the Map tasks we need to do

4. ReduceTasks: A slice to track all the Reduce tasks

5. done: Channel to signal when all tasks are completed

The coordinator has a server() method to start a server that listens for RPC calls from the workers.

func (c *Coordinator) server() {
	rpc.Register(c)
	rpc.HandleHTTP()
	sockname := coordinatorSock()
	os.Remove(sockname)
	l, e := net.Listen("unix", sockname)
	if e != nil {
		log.Fatal("listen error:", e)
	}
	go http.Serve(l, nil)
}

We create a MakeCoordinator method that takes a list of input files and the number of reduce tasks we want to run, starts a coordinator server and returns the coordinator.

func MakeCoordinator(files []string, nReduce int) *Coordinator {
	c := Coordinator{
		Files:       files,
		MapTasks:    make([]Task, len(files)),
		ReduceTasks: make([]Task, nReduce),
		done:        make(chan struct{}),
	}
	fmt.Printf("Coordinator: MakeCoordinator\n")
	fmt.Printf("Coordinator: files %v\n", files)
	fmt.Printf("Coordinator: map tasks %v\n", c.MapTasks)
	fmt.Printf("Coordinator: reduce tasks %v\n", c.ReduceTasks)

	c.startPeriodicChecks()
	c.server()
	return &c
}

(The startPeriodicChecks call checks for tasks that have timed out. We’ll go into those details later, let’s first explore the happy path.)

GetTask

The workers can request the coordinator for a new task using a RPC call to `GetTask`. The interface for this RPC is simple. You ask for a task (no params needed) and get a reply with an input file name, the operation you need to perform (map or reduce) and some metadata like operation number etc.

type GetTaskArgs struct {}

type GetTaskReply struct {
	InputFileName   string
	Operation       string
	OperationNumber int
	NMap            int
	NReduce         int
	WaitForTask     bool
}

The implementation of the GetTask method is also pretty straightforward.

• Look for map tasks, if you find any, assign and return.

• If we didn’t find a map task to assign but all map tasks have not been completed by workers yet, tell the worker to wait for a while. Important note: we cannot assign a reduce task in this case, because we need all map tasks to have finished before we can move on to the reduce phase.

• If all map tasks have been completed, try to find a reduce task. If found, assign and return.

• Again, all reduce tasks are assigned, but not all of them have finished, tell the worker to wait. This is important because a worker who’s assigned a task might fail and the task would have to be reassigned to another worker.

• If all reduce tasks have finished, all the work is done!

Another important note: we need to use the mutex to acquire a lock before making changes to our MapTasks and ReduceTasks slices, otherwise different workers asking for tasks at the same time might cause race conditions.

var ErrAllTasksCompleted = fmt.Errorf("all tasks completed")

func (c *Coordinator) GetTask(args *GetTaskArgs, reply *GetTaskReply) error {
	c.mu.Lock()
	defer c.mu.Unlock()

	if i, task := c.findAvailableTask(c.MapTasks); task != nil {
		c.assignTask("map", i, task, reply)
		return nil
	}

	if !c.allTasksCompleted(c.MapTasks) {
		reply.WaitForTask = true
		fmt.Println("No map tasks available, worker should wait")
		return nil
	}

	fmt.Println("All map tasks completed, looking for reduce tasks")

	if i, task := c.findAvailableTask(c.ReduceTasks); task != nil {
		c.assignTask("reduce", i, task, reply)
		return nil
	}

	if c.allTasksCompleted(c.ReduceTasks) {
		return ErrAllTasksCompleted
	}

	reply.WaitForTask = true
	fmt.Println("No reduce tasks available, worker should wait")
	return nil
}

func (c *Coordinator) findAvailableTask(tasks []Task) (int, *Task) {
	for i, task := range tasks {
		if task.taskStatus == NotStarted {
			return i, &tasks[i]
		}
	}
	return 0, nil
}

func (c *Coordinator) assignTask(operation string, index int, task *Task, reply *GetTaskReply) {
	reply.Operation = operation
	reply.OperationNumber = index
	reply.NMap = len(c.MapTasks)
	reply.NReduce = len(c.ReduceTasks)
	reply.WaitForTask = false
	task.taskStatus = Assigned
	task.assignedAt = time.Now()

	if operation == "map" {
		reply.InputFileName = c.Files[index]
	}
}

MarkTaskCompleted

The coordinator exposes a MarkTaskCompleted RPC, which the workers call as soon as they finish a task they were assigned.

type MarkTaskCompletedArgs struct {
	Operation       string
	OperationNumber int
}

type MarkTaskCompletedReply struct{}

func (c *Coordinator) MarkTaskCompleted(args *MarkTaskCompletedArgs, reply *MarkTaskCompletedReply) error {
	c.mu.Lock()
	defer c.mu.Unlock()
	if args.Operation == "map" {
		c.MapTasks[args.OperationNumber].taskStatus = Completed
		return nil
	} else if args.Operation == "reduce" {
		c.ReduceTasks[args.OperationNumber].taskStatus = Completed
		return nil
	}
	return fmt.Errorf("invalid operation")
}

Worker

Now that we have an understanding of the coordinator, let’s go through the worker. A worker spins up and starts requesting tasks. Once it gets assigned a task, it performs the task and tells the coordinator that the task was done. Then it asks for another task. It does this until it hears from the coordinator that there are no more tasks to do.

func Worker(mapf func(string, string) []KeyValue, reducef func(string, []string) string) {
	for {
		task, taskExists := GetTask()
		if !taskExists {
			break
		}
		if task.WaitForTask {
			// sleep  before trying to get a new task
			time.Sleep(500 * time.Millisecond)
			continue
		}
		if task.Operation == "map" {
			handleMapTask(task, mapf)
		} else if task.Operation == "reduce" {
			handleReduceTask(task, reducef)
		} else {
			log.Fatalf("unknown operation: %v", task.Operation)
			panic(fmt.Errorf("unknown operation: %v", task.Operation))
		}

		MarkTaskCompleted(task.Operation, task.OperationNumber)
	}
}

A few things to note:

1. The map and reduce functions are functions that the user provides. This is the point of the framework.

2. If the worker is told to wait for a while, it sleeps and then tries again.

Both the GetTask and MarkTaskCompleted functions make RPC calls to the coordinator.

// send an RPC request to the coordinator, wait for the response.
// usually returns true.
// returns false if something goes wrong.
func call(rpcname string, args interface{}, reply interface{}) bool {
	sockname := coordinatorSock()
	c, err := rpc.DialHTTP("unix", sockname)
	if err != nil {
		log.Fatal("dialing:", err)
	}
	defer c.Close()

	err = c.Call(rpcname, args, reply)
	if err == nil {
		return true
	}

	fmt.Println(err)
	return false
}

func MarkTaskCompleted(operation string, operationNumber int) {
	args := MarkTaskCompletedArgs{
		Operation:       operation,
		OperationNumber: operationNumber,
	}
	reply := MarkTaskCompletedReply{}
	ok := call("Coordinator.MarkTaskCompleted", &args, &reply)
	if !ok {
		fmt.Printf("call failed!\n")
	}
}

func GetTask() (*GetTaskReply, bool) {
	args := GetTaskArgs{}
	reply := GetTaskReply{}
	ok := call("Coordinator.GetTask", &args, &reply)
	if ok {
		return &reply, true
	} else {
		fmt.Printf("call failed!\n")
		return nil, false
	}
}

The other two important parts of the worker are handleMapTask and handleReduceTask. As the names signify, these handle map and reduce tasks respectively.

Map operation handler

The handleMapTask takes the input file, calls the map function on it. Here the key is the filename and the value is the entire content of the file. This gives us back a slice of intermediate keys and values. For each intermediate key, we write the key-value pair to a particular file based on what reduce task should handle that key.

func handleMapTask(task *GetTaskReply, mapf func(string, string) []KeyValue) {
	fmt.Printf("Map task received...\n")
	fmt.Printf("filename: %v\n", task.InputFileName)
	fileName := task.InputFileName
	contents, err := os.ReadFile(fileName)
	if err != nil {
		log.Fatalf("cannot read %v", fileName)
		panic(err)
	}

	kva := mapf(fileName, string(contents))

	filecontentsmap := make(map[string]string)
	for _, kv := range kva {
                reduceTaskNumberForKey := ihash(kv.Key) % task.NReduce
		outputFileName := fmt.Sprintf("mr-%d-%d", task.OperationNumber, reduceTaskNumberForKey)
		output := filecontentsmap[outputFileName]
		filecontentsmap[outputFileName] = fmt.Sprintf("%s%s %s\n", output, kv.Key, kv.Value)
	}

	// ... write to output files ...

	fmt.Printf("Map task completed: %v\n", task.InputFileName)
}

The important parts of this code are:

kva := mapf(fileName, string(contents))

This calls the map function for the input file.

filecontentsmap := make(map[string]string)
for _, kv := range kva {
	outputFileName := fmt.Sprintf("mr-%d-%d", task.OperationNumber, ihash(kv.Key)%task.NReduce)
	output := filecontentsmap[outputFileName]
	filecontentsmap[outputFileName] = fmt.Sprintf("%s%s %s\n", output, kv.Key, kv.Value)
}

This code creates a map that contains the content for each intermediate output file. These intermediate output files are inputs for the reduce tasks.

This is the part that assigns each key to a reduce task based on the total number of reduce tasks we have.

reduceTaskNumberForKey := ihash(kv.Key) % task.NReduce

Reduce operation handler

The Reduce task handler reads ALL The input files created for it to create a list of intermediate keys.

intermediate := []KeyValue{}

for i := 0; i < task.NMap; i++ {
        filename := fmt.Sprintf("mr-%d-%d", i, task.OperationNumber)
	kva := parseKeyValuePairsFromFile(filename)
	fmt.Printf("reduce task %v: got intermediate keys from %v\n", task.OperationNumber, filename)
	intermediate = append(intermediate, kva...)
}

This gives us all the key value pairs that the reduce task works on. This list is then sorted, based on the key.

sort.Sort(ByKey(intermediate))

Now, for each key, a list of all values is created and reduce is called with the key and these values.

for i := 0; i < len(intermediate); {
        j := i + 1
	for j < len(intermediate) && intermediate[j].Key == intermediate[i].Key {
		j++
	}

	values := []string{}
	for k := i; k < j; k++ {
		values = append(values, intermediate[k].Value)
	}

	output := reducef(intermediate[i].Key, values)
        // ... write output and move on to next key
}

Now we have all the parts to get a basic word count running.

Worker failure

The coordinator also has a thread running that checks for tasks that have been assigned for a long time but not finished. After a timeout, it assumes that the worker that was assigned the task has failed and marks the task as ready for re-assignment.

We use a ticker to do this.

func (c *Coordinator) startPeriodicChecks() {
	ticker := time.NewTicker(10 * time.Second)
	go func() {
		for {
			select {
			case <-ticker.C:
				c.checkTimeoutsAndReassignTasks()
			case <-c.done:
				ticker.Stop()
				return
			}
		}
	}()
}

The checkTimeoutAndReassignTasks is simple to implement as we track when the task was assigned for each task.

func (c *Coordinator) checkTimeoutsAndReassignTasks() {
	c.mu.Lock()
	defer c.mu.Unlock()

	for i, task := range c.MapTasks {
		if task.taskStatus == Assigned && time.Since(task.assignedAt) > 10*time.Second {
			c.MapTasks[i] = Task{}
		}
	}

	for i, task := range c.ReduceTasks {
		if task.taskStatus == Assigned && time.Since(task.assignedAt) > 10*time.Second {
			c.ReduceTasks[i] = Task{}
		}
	}
}

That’s it! We made it through implementing MapReduce in Go!

Epilogue

There’s some things to improve in the code. A good exercise for the future for me is to get this code to run on multiple machines with some shared storage, as that’s how this would run in a real scenario.

If you liked this post and would like to see more, please let me know by leaving a comment. You can also support me by supporting my personal project, blunderfest.

I finished reading a great biography of Robert Oppenheimer, the father of the atomic bomb. Oppenheimer was the director of the Los Alamos laboratory that designed the first atomic bombs dropped on Hiroshima and Nagasaki, ending the second world war.

The book contains a large number of amazing tidbits, especially because Oppenheimer’s personality is so interesting. It goes through his disturbingly weird behaviour as a college student, his success as the director of Los Alamos and his embarrassing loss of security clearance when the US Government suspected him of being a communist spy.

I’ve been sharing a few of the most interesting highlights on Twitter, but I thought I’ll share a few of them here too.

Initial craziness

Early in his career, Oppenheimer tried to poison his Ph.D. advisor out of jealousy.

He also tried to strangle his best friend out of jealousy, and had done multiple things that should put anyone in prison for a long time.

For some people, he was not very fun to work.

The bomb

There was debate about whether any publication of nuclear chain reactions should be stopped for safety.

Information about the atomic bomb was declassified was that a trucker reverse engineered the bomb in 2004.

The book has a lot more interesting stuff, these examples only scratch the surface. If you read it, let me know what you think!

Share

Subscribe now

Last week, I mentioned that I was working through and implementing the paper “Generative Agents: Interactive Simulacra of Human Behavior.” I’ve made some progress that I thought I would log here.

Background

The paper implements “generative agents”, realistic simulations of humans living in a town called Smallville. These agents act like autonomous entities with the ability to perform human-like activities, such as brushing their teeth, going to work, or grabbing lunch.

The paper describes the architecture of these agents as containing three major components:

1. Memory

2. Reflection

3. Planning

Memory

Each agent has a memory stream, which is a list of things that the agent has experienced. These memories are stored as a list, and a retrieval function is implemented which takes a particular situation and returns a subset of the agent’s memories that LLM should use to decide the agent’s reaction to that situation.

The retrieval method uses three factors to decide if a particular memory should be sent to the language model.

1. Recency: How old the memory is.

2. Relevance: How relevant a particular memory is to the situation. The paper uses OpenAI embeddings to calculate this.

3. Importance: For each memory, the paper asks the LLM to rate it between 1-10 in terms of the importance of the memory. 1 is purely mundane (e.g., brushing teeth, making bed) and 10 is extremely poignant (e.g., a break up, college acceptance).

Then we add up these three factors to get a score for each memory.

The code for this is in memory.py.

Reflection

The agents also have the ability to reflect on their experiences. These reflections are also saved in the memory stream, and they help the agents make more realistic choices in the long term as they get time to “think” about their experiences.

The paper makes the agents reflect on their recent experiences if the sum of the importances of the memories gets higher than a particular threshold.

The code for this is in agent.py.

Planning

The paper uses the LLM to get the agents to make plans for their future actions. These plans can be as detailed as needed. I’ve currently only added a high-level broad strokes planning component but I intend to improve this as things go on.

So, how does it work?

Not that good so far. I tried simulating a single agent, “Robinson Crusoe”, a castaway on a remote island. There are some issues where the agent keeps doing the same thing over and over for me.

Next steps

I’m going to continue improving the prompts and see if there are some obvious bugs I need to fix. After that, I want to start a small graphical interface so that the agents can move around and interact with their environment. This is done using Phaser in the paper.

I’m fascinated by the paper “Generative Agents: Interactive Simulacra of Human Behavior.” It’s been on my radar to read in depth for a while, but I only got a chance to do it this week.

Let’s go through what the paper does, on a high level. In the paper, the authors create a small digital town populated by twenty-five 'agents’, each having their own digital life. They're not just static figures, but autonomous entities with the ability to perform human-like activities, such as brushing their teeth, going to work, or grabbing lunch.

Each agent possesses a 'memory' - a record of their experiences, decisions, and interactions. This memory is not just a passive data bank; it informs the agent's actions and choices. They also have a mechanism for 'reflection,' which means they can use their past experiences to influence their future actions.

So, what's the result? The agents can plan and decide what they want to do next based on their 'lived' experiences. They can react to each other, learn from what happens around them, and even plan events like a Valentine's Day party or an election campaign. It's like watching a digital community live out their lives, all within the bounds of a computer program.

This gave me very strong “The Truman Show” vibes. Watching realistic simulations of people living their lives would be straight out of a science fiction movie or book. Sadly, the authors do not really provide code that I’m able to run, so my current side-project is implementing this myself.

Every time I see something like this, it makes me think that consumer media is going to change forever very soon. If these simulations get more and more convincing, there is nothing stopping me from creating a character in the Game of Thrones universe who has an action-packed life and watching them 24/7. There is no need to wait ten years for George R.R. Martin to write 1000 pages.

Of course, there’s a huge caveat here that no AI is as good as George at crafting a good story right now. Still, it’s a future to look forward to.

But beyond revolutionizing entertainment, these simulations raise existential questions, reminding me of Nick Bostrom’s simulation theory.

The entire paper thinking about simulations is interesting to read, but I’ll just quote the conclusion here:

A technologically mature “posthuman” civilization would have enormous computing power. Based on this empirical fact, the simulation argument shows that at least one of the following propositions is true: (1) The fraction of human-level civilizations that reach a posthuman stage is very close to zero; (2) The fraction of posthuman civilizations that are interested in running ancestor-simulations is very close to zero; (3) The fraction of all people with our kind of experiences that are living in a simulation is very close to one.

If (1) is true, then we will almost certainly go extinct before reaching posthumanity. If (2) is true, then there must be a strong convergence among the courses of advanced civilizations so that virtually none contains any relatively wealthy individuals who desire to run ancestor-simulations and are free to do so. If (3) is true, then we almost certainly live in a simulation. In the dark forest of our current ignorance, it seems sensible to apportion one’s credence roughly evenly between (1), (2), and (3).

Unless we are now living in a simulation, our descendants will almost certainly never run an ancestor-simulation.

Thinking about each of these propositions is important.

• If (1) is true, we’re facing a grim future, never making it past our current limitations. My personal optimism makes it hard for me to accept this as our fate.

• (2) is a little counterintuitive because it implies that civilizations will just not be interested in performing these simulations, when we can see in our real world, there is huge amounts of interest.

• But if (1) and (2) are not true, (3) implies that there is a huge probability that we’re living in a simulation, run by an advanced civilization.

So what if we do live in a simulation?

Let’s talk about embeddings because they’re cool. The first time I came across embeddings was when I was building AskYC. To build AskYC, I converted the text from the transcriptions of the YC YouTube channel videos into vectors called embeddings using OpenAI’s API.

The magical thing here is that the distance between these vectors is equivalent to semantic difference between the sentences. This is quite amazing. At that point, I didn’t really understand how any of this worked. However, now I think I have some intuition for how they work, thanks to this chapter of Deep Learning for Coders.

An embedding is a mapping of a discrete variable to a point in a vector space. Let’s think about this in the context of collaborative filtering. Collaborative filtering is a technique used to create recommender systems. On a high level, here’s how it works:

1. We have a dataset that contains users and their ratings for particular items (for the sake of this post, we’ll assume items are movies).

2. From this dataset, we try to learn which users are similar to each other and then use that to recommend new items to users.

In this case, we create vectors for each user and each item. We get to choose how many parameters we want in these vectors. The process of creating these vectors (or training) is simple gradient descent.

1. Initialize vectors with random values.

2. Calculate predictions of ratings for each user and item.

3. Use these predictions and the actual values to calculate a loss.

4. Use this loss to improve your vectors.

Each value in these vectors will eventually represent a particular latent factor. For example, the second value in each movie vector might be directly proportional to how much action there is in the movie. These latent factors are picked up by our model during the training process, which is amazing. At this point, we can calculate the distance between two movie vectors and it’s a good estimate of how similar the two movies are.

So on a high level, this is how embeddings work. As a software engineer who’s mostly experienced with concrete problems and solutions, it is a little hard for me to internalise that we know the process with which how we create these embeddings, but the latent factors just emerge automatically.

Share

Leave a comment

Subscribe now

Decision Trees

Decision trees are pretty intuitive.

Suppose we have tabular data (rows and columns) that we’d like to put into categories. We start by taking a binary split of your data on the most important column. For example: if we’re categorising cars into luxury vs non-luxury, maybe we should start with the price?

This is the simplest decision tree. We can keep splitting down the nodes based on different columns to make the tree consider more columns. For example, perhaps there are expensive cars being sold by a manufacturer that’s not considered legacy. We could add a few nodes for that.

The key thing here is choosing the columns and values to split by correctly. In order to choose a good split, we should create a score metric and then just find the split that has the best score. There are a number of metrics that can be used.

scikit-learn has a decision tree class that does all this for us in a few lines of code. Creating a decision tree is as simple as:

from sklearn.tree import DecisionTreeClassifier
m = DecisionTreeClassifier(min_samples_leaf=25).fit(trn_xs, trn_ys)

Random Forests

Random forests are groups of decision trees. In most cases, it’s hard to create good decision trees without overfitting on the training data. However, we can create a large number of decision trees and take their average. This is known as bagging.

Each tree is trained on a random subset of the original training data. The high-level idea is that if we take the average of a large number of predictions from unrelated models, the errors will average out to zero and the prediction will be closer to the true value.

It’s very easy to create random forests using scikit-learn.

from sklearn.ensemble import RandomForestClassifier

rf = RandomForestClassifier(100, min_samples_leaf=5)
rf.fit(trn_xs, trn_y);

Both these models are relatively simple but still lead to good results many times. Another thing that decision trees or random forests can be helpful for is exploring our data. We can understand which features are more important and which less from the decision trees we create.

Share

Subscribe now

I’ve been building adventures.param.codes for the last few weeks. I only realized today that I hadn’t shared it on this newsletter. So this week’s post is a quick introduction to what it is, what I like about it and what things I’m excited to add to it!

Adventures is a website to play GPT-3 directed text based adventure games (ala Zork). Each day, GPT-4 generates a new starting prompt. This prompt is shared by everyone throughout the day. But from there, things can go anywhere. You make choices and then GPT-3 writes the story from there.

All this is accompanied by Dall-E 2 generated images, to make things more easily imaginable.

Currently, I’m working on making it easy to share the stories that get created. Here’s a sneak peek of a video created from one of the stories I played through:

There are lots of things to improve here, but I’m quite excited by the fact that AI has enabled me to create such media. Hope you like it too, let me know what you think!

Share

Subscribe now

(Programming note upfront: I missed last week’s post because life came in the way. I also haven’t been able to put as much effort into this week’s post as I’d like to, but I still think it’s a nice read. Hope you like it!)

Over the past two weeks, I finished reading The Making of Prince of Persia by Jordan Mechner. The book contains Jordan’s journals from the years that he was working on the first Prince of Persia game. There are lots of motivating and interesting things in the journal, so I thought I’d share a few of them with readers here.

Obsession to the point of dreaming about your work.

I found this quite interesting. I’ve had this happen to me a few times, but not for large projects. Perhaps I haven’t found something that’s made me obsessed enough yet.

You should feel good about what you’re building.

Another consistent theme throughout the journal was that Mechner was convinced that the game was going to be good. He wasn’t sure how successful it would be, but he was always sure that it’d be a high quality game. That’s one thing I’m trying to do as well. I should try to build the best thing I can build, because otherwise what even is the point?

Shouldn’t be afraid of confrontation if you believe in something.

There’s a few moments where Jordan has issues because something some other person had to do for the game wasn’t being done well. It’s never nice to have to confront people about these things, but to get a good result in the end, sometimes you have to. Also, it’s much better to signal that you care about the thing, because that makes other people realize that it’s a thing worth caring about.

There were other parts of the book that I really enjoyed too. Mechner’s evolving thoughts on what would make Prince of Persia fun were really nice to read through.

If you liked these small parts of the book, give it a read and tell me what you thought of it on Twitter! (or leave a comment here, I read all of them.)

Leave a comment

Subscribe now

I breezed through the book “Masters of Doom” this week. It’s a book by David Kushner about id Software, the video game company behind Doom and Quake, focussing on John Carmack and John Romero. It is a great read. Reading a well-written, easy to consume story about the industry was very fun, even though I was a bit young for these games.

The main theme of the book was the contrast between both of the co-founders: John Carmack and John Romero, the two Johns. The book goes through their initial collaborations building absolute gold, and then later falling through after they’d achieved a lot of success.

Carmack is a technical genius, obsessed with programming and computer graphics. He was the lead programmer of all of these games (Wolfenstein 3D, Doom, Quake), the biggest games in the video game industry at that point. In comparison, Romero was focussed on the more creative aspects of the games (storyline, design, look and feel etc). This was a great combination in the beginning. They complemented each other, as Romero was able to use Carmack’s innovations to build things that were amazingly cool at the time.

However, once they became successful, the differences in their personalities started showing up. As more and more money rolled in, Carmack was still focussed on the work and the hard tech while Romero became more and more distracted. Carmack was more successful by the end, having shipped better games (specifically Quake II) while Romero floundered with Daikatana. This seems to be another example of my “five beats a day for three summers” thing.

newsletter.param.codes

Five beats a day for three summers

A common theme I’ve noticed among successful people is extremely impressive levels of consistency and obsession with the thing that they’re working on. By working consistently on a small set of skills, they become the best in the world on it and it’s not easy for others to catch up…

Read more

4 years ago · 5 likes · 3 comments · Param Singh

The other interesting thing was the different ways in which Carmack and Romero ran their companies after splitting up. Carmack kept the team small, opting for a solid team that was laser focussed on releasing one game at a time. In contrast, Romero hired a large number of people, working on multiple things at the same time. By the end, Romero’s game, Daikatana, was a few years behind schedule and not a critical success when it was actually released. This was an interesting parallel to tech companies hiring a large number of people in 2021, with little to show for productivity gains.

Carmack has since worked as CTO of Oculus VR, and now has started a company called Keen Technologies with the goal of building Artificial General Intelligence. He is one of the coolest programmers alive. I’m a fanboy now.

This week, I was playing around with the idea of text-based adventure games created by ChatGPT (with little pushes here and there from me). To see if it could be interesting, I’ve been running a small adventure via Twitter, crowd-sourcing the main character’s choices.

The story is nearing its ending, and has been quite fun. Want to follow along? Vote for the next choice here:

Given that a number of people were interested, I’ll consider automating this and making a separate Twitter account for it.

The thing that it really opens my eyes towards is collaboration between human and AI. I’ve never been one for creative writing. It’s hard for me to invent fictional scenarios and characters. But with ChatGPT, a lot of this becomes easier as I can just nudge it in different directions (“make things more action packed!”, “introduce a group of people for the protagonist to talk with”) and then use the best thing that ChatGPT comes up with. Things are easier when you can use a tool to come up with hundreds of ideas and then use the best one.

If you use any of these tools for creative purposes, I’d love to talk to you about your workflow.

Leave a comment

Let’s talk about gradient descent. This is the topic discussed in Lecture 3 of the FastAI Deep Learning course.

In short, gradient descent is a method to fit a function to data. The steps involved are pretty simple. The fastai deep learning book explains it succinctly and clearly in Chapter 4:

1. Initialize the weights. We can take random weights for now.

2. Predict the value of the function based on these weights.

3. Based on these predictions, calculate how good the model is (its loss).

4. Calculate the gradient, which measures for each weight, how changing that weight would change the loss.

5. Step (that is, change) all the weights based on that calculation.

6. Go back to the step 2, and repeat the process.

7. Iterate until you decide to stop the training process (for instance, because the model is good enough or you don't want to wait any longer).

The most complicated part of this process is actually calculating the gradient. However, the good news is that libraries like pytorch handle the math for us. I was very surprised to see that the code for gradient descent is a very simple function.

def apply_step(params):
    preds = f(params)
    loss = mse(preds, actual)
    loss.backward()
    params.data -= lr * params.grad.data
    params.grad = None
    return preds

weights = torch.randn(3).requires_grad_()
for epoch in range(10):
    weights = apply_step(weights)

Here, the loss.backward() call calculates the gradient and puts it in the params variable for us. After that, everything is pretty easy.

The results of just this simple gradient descent aren’t the most impressive, even for stuff like Kaggle’s Titanic problem, but it’s pretty cool to see how simple and intuitive basic machine learning algorithms can be.

I’m always amazed by the intuitiveness of a lot of these algorithms in computer science. The thing that really differentiates a good algorithm from a great algorithm is simplicity. A great algorithm gives you the feeling that you could easily have come up with it, given time. I felt the same about Dijkstra’s algorithm or Binary Search Trees the first time I came across them.

I hope this small note about gradient descent makes some sense. If you’ve done this stuff before, I’m looking for ideas about things to build as the deep learning course goes on. Right now, I’m not too deep in the course, so Kaggle problems are fun, but I’m hoping to put this knowledge to actual use somewhere. Let me know if you want to chat!

Leave a comment

Things of interest

• People are now able to run GPT-3 level LLMs on consumer grade MacBooks. The future is going to be amazing, I can’t wait.

• Aiku has generated ~200 poems since February 4. I just assumed that no-one used it, but that wasn’t the case. Most of the users seem to be coming from Google, so I tweaked the site a bit (adding sitemaps etc) to make it more user-friendly. The main thing that I wanted to share was that it’s easy to assume that your project is dead if you don’t have any metrics. Having metrics is a good thing. :)

First off, here’s a link to AskYC: https://transcribe.param.codes/ask/yc

In short, AskYC is a GPT-based Q&A bot that’s based on every video on the YCombinator YouTube channel (except for a few very new ones). I spent the weekend hacking together its final parts after I’d been working on it for the last 2 weeks with Suyash.

Here are a few example questions you can ask:

1. “How do I get a startup idea?”

2. “How do I validate my startup idea?”

3. “What is product market fit?”

4. “How did Stripe make its first hire?”

The outputs are quite nice, mostly because the YouTube content is really good. For example, here’s a screenshot:

I transcribed these videos using Transcriber and then generated the embeddings for these transcriptions using llama-index. I’ll jump into the technical details in a future post, but in the meanwhile, if you have any feedback or even any ideas for something similar, please let me know! Now that I’ve done it for one channel, theoretically, it should be possible to do for any youtube channel, the only constraint being money.

Thanks for reading!

In the interest of not becoming entirely redundant in the next one year, I’ve decided to pick up fast.ai’s deep learning course. The course is free on YouTube, and so is the book it’s based on.

I’ve only watched Lecture 1 so far. I thought I would write a few small takeaways for myself (and for readers who are interested). I still highly recommend going through the course, because it’s pretty cool.

So here we go:

1. Creating applications using deep learning is easier than I thought. In essence, most state-of-the-art (or near state-of-the-art) stuff is available freely and easy to fine-tune for your application.

2. Hardware is available in ease that was not really possible even just a few years ago. Most of these models require some sort of GPU to actually work in reasonable amounts of time in practical applications. In the past, especially when I was in university, it was hard and expensive to get access to these things. These days, you can just spin up a Kaggle notebook or a Google Colab session to experiment with. For production, you can use tools like Replicate.

3. Most of the actual work is data cleaning and that sort of labor. So far, from what I’ve experienced, 90% of the time, I’m working on getting the data in a format that’s actually useful for the model and then put the answers in a format that’s actually useful for me. The fine-tuning and predictions are much easier.

4. It feels amazing and a little weird to train (or fine-tune) a model. I solved the digit recognizer problem by fine-tuning resnet34 over the training dataset. On a high level, it’s similar to teaching a kid digits. You show them a few digits and tell them to remember it. This is essentially what you’re doing when you’re fine-tuning a model too. I wonder how the creators of things like ChatGPT feel, seeing a nearly general intelligence be born out of their code.

Subscribe now

Other news

• I’m working on a cool project to create a Q&A bot over all of YCombinator’s YouTube channel, if you’re interested and want to chat, hit me up on Twitter:

• Interesting post about Stripe this week:

  Net Interest

  Striped Down

  Since switching on the paid tier of Net Interest eighteen months ago, I’ve processed a lot of payments. On average, I do around 12 a day and have administered around 7,000 in total. They originate from all over the world: 40 states in America, practically every country in Western Europe, and countries as diverse as Colombia, Kenya and Vietnam. Most subscribers pay by credit card, but some use debit and prepaid cards…

  Read more

  3 years ago · 34 likes · Marc Rubinstein

If you’ve been reading my stuff for some time, you’ll know that one thing that I’ve historically struggled with is consistency.

Here’s the thing: for me, it’s easy to start something, but once I get the initial fun bits of the work done, I start losing interest. It’s hard to motivate myself to do the less glamorous parts of the work. For example, creating the first few pieces of Transcribe was fun, using tools like Replicate and GPT-3. Now that it’s done, the tasks are less fun. There are lots of UX improvements I want to make, so that it’s more convenient for me to use. I also want to create a browser extension to make it easier to submit things. The list is long.

A similar thing happens to me when reading. It’s so much easier to start a new technical book. The first few chapters are relatively basic and do not require a lot of mental work. But as soon as things start getting complicated, it becomes harder for me to keep going.

I’ve been thinking about how to improve, and I wanted to share one thing that’s been helpful in the past few weeks. Instead of having just one thing to focus on for a long period of time, I’ve been trying to have multiple things. For example, instead of just spending all my free time on transcribe, I worked on a small command line tool (called huh) that I thought I would find helpful. Similarly, instead of just focussing on the one technical book, I read multiple simultaneously. This way, if one thing becomes too onerous, I just shift focus to the other and come back when I feel better.

I was thinking about this, and I came to a small realisation that this is one of the reasons why universities don’t just make you deep dive into one or two subjects across a semester. Of course, there are other logistical reasons (credit requirements, faculty expertise etc.) as well, but I think it’s easier for students when they’re not just focussed 24/7 on the same subject.

This is just something I’ve been trying in the last 2-3 weeks, so I’m still not completely convinced that I’m right. General convention of flow states and deep work do not recommend having multiple things vying for your focus, so perhaps it’s not helpful in the long run. Still, I’ve found it helpful in the last few weeks, so I’ll keep doing it until it’s not productive.

Subscribe now

Random news

• I re-did my personal website last week, in case you’re interested: param.codes

• If you want to try out huh, feel free to install it! Install intructions here. Demo:

That’s it for this week, thanks for reading!

Share newsletter.param.codes

What is this?

Transcribe (https://transcribe.param.codes) uses OpenAI’s Whisper to create transcriptions of YouTube videos and then uses GPT-3 to improve those transcriptions (remove filler words, transcription errors etc.)

Why is it useful?

I’ve always preferred reading to watching videos, especially for educational content. However, as the internet has grown, there is an increasing amount of valuable information that’s only available in video form. It’s hard for me to consume this information because I don’t have the attention span to watch a 1.5 hour video. So, I thought I would create a tool to encode this information in text easily.

The sad thing that this encoding is, in essence, lossy. People use videos because they get a visual component that is just not reproducible in text. However, a transcription is still useful in many cases, even if you prefer videos to text.

The most obvious case is recall. It’s hard to find a random part of a video where the person talked about X topic, weeks after you originally watched the video. It’s much easier to do Ctrl+F “X” and then read from there. Another case is when you want an LLM like GPT-3 to process the information for you. Currently, this isn’t possible for videos. But it is possible for a transcription.

See it in action!

Link: https://transcribe.param.codes/

Example transcriptions:

• Terraform explained in 15 minutes

• Karpathy’s “The spelled out intro to neural networks and backpropagation”

Here’s a quick 30 second demo:

How it works

It’s very easy to build AI apps these days. I used Replicate to run whisper on audio files. It’s very nice (at least in the beginning) to not have to worry about managing GPUs when trying to use these models. The code for this is very simple, it’s something like:

import replicate
model = replicate.models.get("openai/whisper")
version = model.versions.get("VERSION_NUMBER_HERE")
transcription = version.predict(audio=open("/path/to/file", "wb"))

After that, I send the transcription to GPT-3 which improves the transcription basically like does in this tweet:

That’s the app for now. It’s very simple, uses Flask, Next.js and SQLite (!)

Last minute notes

If you try it out, I would recommend signing in before transcribing something. That way, you can keep track of the things you’ve transcribed. The transcription process does take a reasonable amount of time and if you haven’t signed in, you have to keep track of your video’s transcription link yourself.

What’s next?

Our goal for this week is to group information from different videos together and then see if we can do some LLM magic with this data. We’re still experimenting here, trying to find what’s useful for us and for other people, so if you have ideas about things you’d want a tool like this to do (or even if you just want to chat), please reach out to me, either by replying to this post or via Twitter.

Nearly a year ago, I created a small bot called Aiku. Aiku could generate a haiku-like poem for you, given two words as input. It’s based on the now-wildly popular GPT-3 model.

Since I didn’t want to have to manually generate poems myself to see them, I also created a Twitter bot. It’s a tiny Python script that takes two random words, generates a poem and tweets it. Every day, at 12 AM GMT, it posts one poem and goes back to sleep.

Sadly, this week, Twitter has decided that it’s not going to allow people to use its API AT ALL without getting paid themselves first. This is weird, bad and funnily hypocritical coming from the guy who was talking about open-sourcing the recommendation algorithm a few months ago. The news made me sad. The bot doesn’t have many followers, but seeing it keep working on its weird poems was a nice moment of my day almost every day.

Coming to Twitter’s API, unsurprisingly, it’s a pain in the ass to use. As a developer, I expect an API to be actually usable without talking to hundreds of support people. There are 2 versions of the Twitter API, and there isn’t an easy way to determine which one you’re supposed to use. I assumed that posting a tweet would be basic enough functionality to be in both the versions, so I went with the version that my client library seemed to support. Sadly, I later found that the version I chose did not actually allow you to tweet. Then, when I moved to the other version, Twitter told me that they needed me to talk with them before they’d actually allow me to post tweets.

So, all in all, it wasn’t a fun experience, especially with me coming from a different API company, where developer experience is literally paramount. Now that they expect me to pay to post a tweet, I’ve moved Aiku off Twitter and into Mastodon. I’m hosting a personal mastodon instance where I will (and you can) follow Aiku as it continues posting a poem every day.

Now, the funniest thing I saw this week was this tweet:

I wonder if Elon Musk seriously thinks that people will pay a 100 USD per month for the Twitter API, which is below par on almost any axis you judge APIs on. I wish he’d go back to building his rockets and cars. Never meet your heroes, friends.

Side note: A lot of other bot owners are pissed and removing their bots from Twitter too. Here’s one of the most popular ones: @year_progress

Either ways, I was thinking about hosting a personal mastodon for a while now, so thanks Elon, I guess, for finally making me do it.

I helped ship a lot of things. The easiest to see is the Stripe Dashboard’s login page. There were a number of interesting internal libraries that I helped develop, learning a lot about web security and cryptography on the way.

If I was to compare before-Stripe Param and after-Stripe Param, I think there’s a huge difference in engineering skills and general ability to get things done. Before Stripe, I didn’t realize how much of a software engineer’s job is simply making sure that your project doesn’t get derailed by external factors.

A popular image of a software engineer is the hacker sitting in a basement, creating a miraculous thing overnight. There are moments when you feel like that, but for the majority of the time, it’s less glamorous. There is a lot of other work to do. You have to talk to users throughout the development cycle, making sure that the project does solve the problem it’s supposed to. You have to provide support, write documentation, figure out which metrics you’re tracking to measure the project’s success.

The other thing that I really cherish is the team that I worked with. In 2020, when COVID happened, we were essentially a pack going through it together. I feel grateful that I was able to work with such amazing people for such a long time.

In summary, it was a very fun experience where I grew quite a lot and made a lot of friends.

React doesn’t escape or sanitize “href” props in URLs. If you allow a user to create custom URLs on your website somehow, there’s a high chance your website can be exploited like this. Let’s say you have a field on your website for your user’s blog. The user can enter a link to their blog and you show it on the user’s profile page on your site. The code could look like this:

const UserBlogLink = (link) => {
  return (
    <a href={link}>Blog</a>
  );
}

This component is not safe from XSS. React doesn’t do any sanitization of the “href” prop here. A user could put in “javascript:alert(0)” or something worse as their blog and your website would execute it.

So how do we avoid these bugs? Well, the first thing you want is a good CSP policy. Things like these pop up every once in a while. While you can try to avoid XSS bugs by trying to safely render and validate all data, it’s nice to have defense in depth in case something does get missed (which it will). Then, the only way to fix this specific bug is to validate that the url is an actual HTTPS or HTTP link, not Javascript. Vue and Angular do this for you out of the box.