Many applications require few-shot adaptation for structured prediction tasks, such as semantic parsing for custom APIs, translation involving low-resource languages, and Text-to-SQL for private databases. These can be framed as structured sequence-to-sequence problems. While In-Context Learning (ICL) is the dominant approach for adapting large language models (LLMs) to new tasks, we show that it has fundamental limitations in this setting.  Through a mechanistic analysis with synthetic formal languages, we attribute the failure to the difficulty of in-context learning input-output alignments in structured tasks.   For light-weight adaptation on such tasks, we will discuss alternative strategies, including in-context fine-tuning, paired decomposition of input-output sequences, and template-constrained decoding.