Selected publications

Selected work on language models, controllable generation, and applications to biology. See Google Scholar for a broader publication list.

Protein generation

I co-developed the machine-learning modeling, generation, and scoring for ProGen. This work demonstrated that next-token language modeling, without explicit structural modeling, could generate novel proteins with experimentally verified function. Generated enzymes retained activity at sequence identities as low as 31.4% to known natural proteins, and selected designs had catalytic efficiencies comparable to natural lysozymes.

Controllable generation

I introduced generative-discriminator-guided contrastive generation: using Bayes’ rule to contrast desired and undesired class-conditional distributions and guide every next-token decision. The underlying principle of contrasting conditional distributions has been used in subsequent work on controllable generation and reinforcement learning. GeDi also demonstrated zero-shot control of topics unseen during training.

Adaptation in transformer language models

I extended dynamic evaluation to Transformer-XL, setting new state-of-the-art results on enwik8, text8, and WikiText-103 by adapting the model to recent text at inference time.

Multiplicative LSTM

I introduced multiplicative LSTM, combining multiplicative interactions with LSTM memory. OpenAI used this architecture for its 2017 sentiment-neuron language model, learning representations from next-character prediction. OpenAI subsequently identified that work as a precursor to GPT-1.

Dynamic Evaluation

Dynamic evaluation is a method for gradient based adaptation to sequence history that can exploit re-occurring sequential patterns. I explored and developed dynamic evaluation methodology to improve the state-of-the-art at several commonly benchmarked character and word-level language modelling tasks. This work was published at ICML 2018.

Conversational AI

During my time working on the Amazon Alexa prize, I developed data driven methods for building open domain conversation agents that combined retrieval and generative approaches, and contributed to the development of a new data collection technique called self-dialogues. Our conversation corpus collected from Amazon Mechanical Turk is publicly available here.

Optimization

My master's thesis and my early PhD work explored Hessian-free optimization in LSTMs

Neuroimaging

I also have past neuroimaging research studying the role of neurotransmitters in brain blood flow and schizophrenia