Skeleton-based action understanding

Research question

Zero-shot action recognition uses language to recognize actions without training examples of those classes. My research asks what makes a useful action description: richer language can introduce details that skeleton joints cannot observe, while repeated phrasing can blur class boundaries in CLIP space. Building on Neuron, I investigate how text representations affect both unseen-class recognition and the balance between seen and unseen classes in generalized zero-shot learning.

My contribution

• Designed and documented six experiments, progressing from three versions of generated action descriptions to label-only, single-stream, and dual-stream text encoders.
• Built a semantic generation and encoding workflow for three spatial granularities and three temporal phases. Controlled body-part cues, observable motion, description length, and continuity between phases.
• Developed diagnostics using text-length statistics, lexical overlap, CLIP cosine-similarity matrices, per-class errors, and spatial/temporal branch metrics to connect representation changes with recognition behavior.
• Implemented branch-specific learnable prompt tokens and a dual-stream CLIP encoder that combines template and prompted hidden states through learned gates, with a cosine-based text-preservation loss.

Technical approach

• Semantic design: first add action attributes and discriminative motion cues; then restore coarse-to-fine spatial structure and chained temporal descriptions; finally remove rigid phase-opening words to prioritize action-specific information.
• Representation analysis: compare class separation before and after CLIP encoding. Similar description lengths do not necessarily produce equally discriminative features, motivating changes to the encoder rather than further prompt wording alone.
• Online learning: compare a shared label template with six branch-specific prompt sequences. In the dual-stream variant, template features guide prompted features in the early Transformer layers, and a preservation objective constrains semantic drift.
• Evaluation: track ZSL top-1 accuracy alongside GZSL seen accuracy, unseen accuracy, harmonic mean, and separate spatial/temporal results on NTU60. Examine training curves and class confusions instead of relying on one aggregate score.

Results & outcomes

• The experiments exposed a concrete tradeoff: improving recognition of unseen actions alone can leave generalized recognition poorly balanced. They also showed why fluent, detailed descriptions are not automatically better class prototypes.
• In the NTU60 cross-subject dual-stream experiment (55 seen / 5 unseen classes, seed 1), the epoch-28 checkpoint achieved 77.16% seen accuracy, 65.94% unseen accuracy, and a 71.11% harmonic mean. ZSL top-1 at the same checkpoint was 72.88%.
• The study established an experimental progression from semantic prompt engineering to trainable text representations, supported by branch-level diagnostics and training records. Ongoing research develops explicit temporal relationships through global action semantics and phase-specific prompts.