As graphics designer, I also learn how to program shaders. Few tools already exist to design shaders like Shader FX for 3DSmax. Nevertheless, you have not a full access to different render targets, and you can't directly watch what happens in the vertex and fragment shader. I mainly used XNA librarybecause it's quite simple, using C# language, and above all, I'm free to play with render targets, stencil buffer, and manage my render passes. I often write my shaders in HLSL but I also use Cg language for other projects.
Here is a test that I did with XNA - This render uses several techniques. These are standard passes using a multi render target (MRT) : color / lightning pass, normal pass and a Z-depth pass. I do also a depth render from the light vector to create a shadow map effect. Normal and Z-depth passes are used to create rim light, outline, and fog effect. At each frame, I also make a separate render of a custom particle system. This way I obtain a pencil drawing map. Lastly, I blend this different kind of "doodles" using the first 3D render.
Test scene that I have created to test the pencil shader.
I'm currently working on a paint effect. I have tried different techniques, and right now, I'm working on a medley using at first, a simple render of geometries, then, an analysis of this pass, and at last, a particles painting (2D). The critical point is the second part, to determine a good vector field. I'm currently using edge detection (with a simple 3x3 convolution filter), and then a distance transform algorithm - But I'm not fully satisfy of this method: there is too much flickering movement between frames (I will try soon with a Kuwahara filter). I also displace particles with a motion pass.
Paint effect screenshots.
I'm still working on fluids shader for the game project Puddle. Those tests use also XNA library - there is no real fluid simulation, I work only the fluid render. By rendering different particle's information (velocity, id, pressure), I'm able to change the type of fluid material (specular, color, refraction, transparency).
Puddle fluid effects screenshots.