I've been working on fabric.js — a canvas library to help with exactly that — manipulating objects on canvas, by handling events and user interactions. It's not released yet, but take a look at a simple preview demo.
You can also see it in action in this design editor, which it was originally created for.
I'm surprised no one has mentioned WebGL, and frameworks built on it. I would consider it high on the list for state-of-the-art for 3D GPU-accelerated graphics and complex animation on HTML canvas / javascript.
WebGL is a cross-platform,
royalty-free web standard for a
low-level 3D graphics API based on
OpenGL ES 2.0, exposed through the
HTML5 Canvas element as Document
Object Model interfaces. ...
WebGL
brings plugin-free 3D to the web,
implemented right into the browser.
Major browser vendors Apple (Safari),
Google (Chrome), Mozilla (Firefox),
and Opera (Opera) are members of the
WebGL Working Group.
WebGL is very solid in its support for GPU-accelerated graphics. Check out these GLSL shaderdemos. :-) And see ChemDoodle as an example of user interaction.
I've been working on an app using Google's O3D framework, which manages the scene graph, and uses WebGL for rendering (it used to use its own plug-in). O3D is a work in progress, and its documentation is not completely up to date, but it is under active development, and there are some good demos out there. 3D Pool may be most up your alley. The Google developers are very responsive to questions in the discussion group.
There are a number of other frameworks built on WebGL; see here. Ones that mention game development and scene graphs include Copperlicht, SceneJS, X3DOM.
WebGL runs in recent development builds of several browsers, but not IE. I've been using Firefox ("Minefield") and Chromium with good results. You will need one of these to run the above demos.
However if your requirements are that it must have no dependencies beyond HTML 5 canvas / js, WebGL may not be the right choice. It doesn't look like IE will support it anytime soon.
Raphael seems a pretty good canvas library; it's SVG-based (or VML-based in Internet Explorer), and thus supports a lot of user input events. It's fairly small (60kb gzipped), so isn't too large a dependency.
It seems to have a nice tweener too: http://raphaeljs.com/reference.html#animate (see here and here for examples).
If you want to use Javascript, Dojo is a great way to go. It has a compact, cross-platform (SVG, VML, Canvas, Silverlight) vector graphics API that is very powerful. You can find it in dojo.gfx and dojox.gfx.
We've used this to build an interactive physics tutor that allows students to draw vectors, ellipses, etc (even append images) and perform all sorts of transformations on them. You can see what we've done at http://gideon.eas.asu.edu/web-UI/login.html --just login with any username.
I've taken a look at fabric.js and dojox.drawing does a lot of the same things. If you look at the tests in the toolkit (once you've got it its dojox/drawing/tests/) you find examples of everything from vector graphics to images to programmatically created shadows.
Also, young, but not bad, Javascript framework, and it (complex animation, managing scene graphs, handling events and user interactions) all about it - jCanvaScript. May be, except 'managing scene graphs'.
three.js, by mr. doob, is a fantastic 3d engine for javascript that includes scenegraph (both software and WebGL/hardware accelerated versions), shading, particles, skinned animation (i think), and lighting effects. Check it out, he is a super-talented fellow.
Been working with bHive to produce graphs and moving site headers, seems impressive and powerful, unlike the others seems to be being developed. Adobe Edge is also worth a gander though strictly not Canvas.
I am impressed with Akihabara as a game engine. It has fantastic documentation in the form of tutorials and an api. I've even seen on some message boards talk of an akihabara 2 release. Unfortunately, all that talk is about one year or more old. I really hope this engine is still being developed.
I just released the first iteration of a new drawing & tweening library geared towards folks with an AS3/Flash development background. While my lib doesn't yet support complex drawing paths or graphs per say I'm hoping it'll help folks quickly draw & animate basic primitives in a familiar way.
I have found two libraries to be extremely competitive and much better then fabric.
Kinetic.js and easel.js both have extremely good event handling, grouping and general shape abstraction. You'll find a lot to love in both of these; easel seems to have more image oriented and filtering.
Fabric's event handlins is MUCH worse than either of these - basically it treats the whole canvas as one big event rrapper and tells you when "Something" has been clicked on. It doesn't attach events to individual shapes or groups of shapes.