I recently played around with Shader Booth, created my Max Bittker, which had me wondering if I could incorporate a shader into any of the ml5 models. Although I know virtually nothing about writing shader code or GLSL, I thought this could be a fun way to learn and experiment with them. I was surprised and happy to discover that, the new p5.js reference has (at least new to me), examples, tutorials, and a handful of functions, expressly to encourage using shaders in p5.js!

To start, I followed the ml5.js FaceMesh guide, to understand which methods (if any) might work well with GLSL, the OpenGL Shading Language.

I then realized that instead of trying to apply a shader to the key points example, it would make more sense to start with am ml5 FaceMesh example and one of p5’s basic shader examples.

The functionality I wanted was to

• Get the ml5 FaceMesh data / UV coordinates to map a texture onto those points
• Have the fragment.frag shader (the file which handles the texture), retrieve the FaceMesh pixel data and mix it with a new color

At this point, I had spent an egregious amount of time trying to understand the basic tenets of shaders / GLSL (mostly confusing myself), so I asked Claude.ai how I could adapt one of p5’s shader examples.

I first needed to orient my brain around using WEBGL coordinates in p5, remembering that the origin point was now the center of the screen, not in the top-left corner. Beginning with Dan’s FaceMesh 3D Texture sketch, I added a vertex.vert and fragment.frag file, for the shader’s structure and texture. After much struggle and strife 😅, I was able to kind of apply a “shader”, albeit with a very basic output.

This wasn’t a satisfying result, so I tried to understand why my fragment.frag code, wasn’t being applied to the FaceMesh model’s output. When I asked Claude.ai for help on this, it suggested a bunch of random changes that weren’t related to my question, or the part of the code I was trouble-shooting. Nothing really new from an LLM when it comes to debugging code, but Claude.ai did remind me that WebGL and video textures have opposite Y directions. 😱 I had a feeling that coordinate system would come back to haunt me!

I kept fiddling with the code making adjustments in my sketch.js file, to see if I could further debug the issue.

→ sketch.js

// Tells p5.js to use coordinates from 0 to 1, instead of pixel positions
textureMode(NORMAL);

function draw();
// Changed lines in the vertex loop to flip the y position
vertex(pointA.x, pointA.y, pointA.z, uvA[0], 1.0 - uvA[1]);
vertex(pointB.x, pointB.y, pointB.z, uvB[0], 1.0 - uvB[1]);
vertex(pointC.x, pointC.y, pointC.z, uvC[0], 1.0 - uvC[1]);