Building a Spotify Powered Virtual Plant App to Nurture Joy Oladokun
If you're anything like me, you're hooked to habit forming apps. I'm talking about those apps that help you learn a new language, stay focused while working, or reward your physical activity with badges, rings, or uhh... a thematic representation of your progress.
I was thinking about these applications when Republic Records reached out to me about the celebrated singer-songwriter Joy Oladokun and her upcoming album Proof of Life. There was an urge to bring her incredible rattail cactus filled album art to life somehow within a goal oriented digital activation. In the artwork, there is a clear connection between Joy's dreads and the cactus stems, with their lengths symbolizing strength. Taking this thought and those habit forming apps as seeds of inspiration, we imagined a virtual plant app which represented a fan's support of Joy using Spotify streams as an indicator of nurturement and one which encouraged fans to check in daily to see how their plant was growing. Fans would then be encouraged to share snapshots of their unique plants, forming a virtual garden of support for Joy. Well, after many hours of development and design, that's exactly what we ended up building.
Visit garden.joyoladokun.com to begin nurturing your own plant today. Haven't streamed Joy Oladokun before? No problem. You plant will still begin to grow and by simply streaming Joy on Spotify today (might I recommend her new single "We're All Gonna Die" ft. Noah Kahan,) your plant's growth will increase tomorrow. Hell, if you find yourself diving through Joy's back catalog also, who knows what ways your plant might evolve. 🪴
From the Spotify Developer platform to a Three JS powered generative cactus plant, read on to learn how this app came together.
Spotify Nurturement Score
So, how does one manifest a plant nurturement score from Spotify? Well it all starts with the Top Items endpoint of the Spotify Web API. This endpoint allows us to pull, among other things, a user's top 50 recently streamed tracks. Using this data, we can check for the appearance and position of Joy's music (or music Joy has appeared on) and use that information to come up with a score. This is the same technique I've used on similar fan affinity applications for Girl In Red, Greta Van Fleet, and others.
For my affinity algorithm, I like to award users for the position of the track, not just the fact that it exists. In basic terms, if the #1 track is by Joy, that is worth 50 points. Whereas the #50 track is worth 1 point. This gives us a more dynamic score of affinity. In order to still reward fans who might not have streamed Joy yet, I employ the use of a friendlier minimum than "0." Here's how that code might look once you have a list of tracks.
// Get max tracks
let maxTracks = tracks.length
// Initialize max nurturment
let maxNurturement = 0
// Loop through tracks
tracks.forEach((track, i) => {
// Increment max nurturement
maxNurturement += maxTracks - i
})
// Initialize nurturement
let nurturement = 0
// Loop through tracks
tracks.forEach((track, i) => {
// If one of track's artists match target artist
if (track.artists.map(artist => artist.id).includes(SPOTIFY_ARTIST_ID)) {
// Increment nurturement
nurturement += maxTracks - i
}
})
// Return nurturement
return Math.max(minNurturement, nurturement / maxNurturement)
Now that I'm working exclusively with Nuxt 3, I do most of this data massaging in a user store powered by Pinia. There was a little bit of a learning curve moving from Vuex to Pinia but once I got the hang of it, I really enjoyed it.
Growth Algorithm
Stems
We now have a Spotify powered nurturement score which we can use to grow our plant but first we need to really understand how our plant will grow. The plant featured on Joy's cover which we are most interested in are the rattail cacti. My thinking is that the stems of our cactus plant would emerge from a suspended round pot, flow over the sides, and fall down the screen (like dreads.) After a bit of thinking and sketching, I realized I only needed to generate and store three main variables for each stem.
rotation
- the direction in which the stem will be positioned and growdistance
- the distance of the stem from the center of the potlength
- how long the stem has grown
The growth algorithm decides, through simple chance tests, whether a new stem appears or an existing stem grows (up to a specified length.) Precedence is given to stems which exist and when 10 stems are present, one of those must grow to a max length before an 11th appears. Again, a stem itself is simply an angle, position, and length. Excuse my laziness, but I'm using lodash here to simply generate some small floats rounded to two decimal places which my database thanks me for. I kept these values very simple so I had more control over interpreting them within the visual.
return [
_.round(Math.random() * Math.PI * 2, 2), // angle
_.round(Math.random(), 2), // distance
2 // length
]
In order to grow a stem, we can simply increase the length.
Flowers
In addition to an array of cactuses, the album cover has many flowers. Rattail Cacti, in fact, do bud flowers of their own at random points on their stems. So, I decided I would extend the growth algorithm to offer a small chance that a flower might appear on one of the segments of one of the stems. I directly connected that chance to a user's overall nurturement score. The higher your score, the more likely a flower would appear on your plant.
if (Math.random() < nurturement) {
// grow flower
}
The flowering algorithm actually became pretty complex because I didn't want flowers to bud too early in a stem's segments (and interact with the pot itself.) In order to handle this, I had to come up with many aptly named variables and calculations such as floweringStems
, flowerableLength
, and floweringSegments
. In the end, a flower, similar to a cactus stem is just a small multi-dimensional array. In this case, it consists of a stemIndex
and a segmentIndex
which decided on which stems' segment this flower would exist. I also added a randomized type
to determine what color the flower would be and scale
to determine how large or small it would appear.
return [
stemIndex, // stem index
segmentIndex, // segment index
Math.floor(Math.random() * 5), // type
_.round(Math.random(), 2) // scale
]
Storage
This combination of stems and flowers make up our plant and this data representation is stored in a DynamoDB via a simple Serverless powered API. In addition to creation, the API allows for updates so that users can evolve their plant over time. It also allows us to track if a user has already nurtured their plant today. We don't store any other data about our users because we don't need it.
Rat Tail Cactus
Our nuturement score powered our growth algorithm and now we have an array of stems (and potentially flowers) for each users' plants. We must now visualize this data. I knew from the start that I wanted to use three.js to do this but I did give myself an opportunity to explore other solutions such as p5.js and just plain ol' HTML canvas. However, in the end, I returned to three.js and just prayed I was going to be able to pull it off technically (and visually) to do the original artwork justice.
Skinned Mesh
What I really needed from three.js was some sort of bendable cylinder and after a bit of digging through the docs, I stumbled upon SkinnedMesh. A SkinnedMesh is a mesh with a skeleton and bone system baked into it which you can use to animate (or in our case, bend) the vertices. Check out the full docs on SkinnedMesh to better understand how these meshes are initialized. For now, I'll just discuss the bits that are unique to this app.
First, the main geometry was indeed a Cylinder and we could use our array of unique stem properties to create it. Let's start with initialization. We establish a few helper variables which determine the height of one segment of the cylinder, how many segments should exist, and the total height of all segments. Again, this is powered by the stored length
we grew earlier. One little trick here is making the top radius 0.05
different from the bottom radius 0.1
. This allows the stem (cylinder) to taper at the end like the real plant. (Though, I still added an additional mesh for the cactus crown itself.)
let segmentHeight = 0.25 // height of one segment
let segmentCount = length // total segments
let height = segmentHeight * segmentCount // total height of stem
// Initialize stem geometry
let stemGeometry = new THREE.CylinderGeometry(0.05, 0.1, height, 8, segmentCount * 3, true)
// (a lot of skinned mesh vertex setup, see docs)
// (also handling the stem material, we'll discuss later)
// Initialize stem
let stem = new THREE.SkinnedMesh(stemGeometry, stemMaterial)
We could then position and rotate the stem within our pot using our angle
and distance
.
// Position stem
stem.position.x = Math.cos(angle) * distance
stem.position.z = Math.sin(angle) * distance
// Rotate stem
stem.rotation.y = -angle
This gave us a series of cylinders sticking out of our pot. The next step is to, you guessed it, bend them.
Bending
Again, make sure to check out the SkinnedMesh docs for a full look on how the skeleton and bones system is initialized. With all of this in place, we simply need to target specific bones and rotate them in order to get the bend we're after.
For starters, I want the first 20 or so segments of the cactus stem to grow up and over the side of the pot and then grow down, like the dreads on Joy's head. Since each bone is influenced by the bone that comes before it, I used a ratio influenced bend calculation to get a more natural bend. I'm sure there are smarter ways to calculate this but this worked fine for our purpose.
bones.slice(1, 20).forEach((bone, i) => {
bone.rotation.z -= 0.3 * (1.0 - (i / 20))
})
In addition, in order to make the overall plant look a bit more organic, I gave all of the bones a little random bend on the x
and y
axis.
bones.forEach(bone => {
bone.rotation.x = _.random(-0.05, 0.05, true)
bone.rotation.y = _.random(-0.05, 0.05, true)
})
Shout out to the three.js docs and the inclusion of this SkinnedMesh solution. It really accomplished exactly what we needed in a performant way with just a little bit of effort.
Flowers
The geometry of the flowers is simply a cone shape I modeled in Blender and imported into the three.js scene using GLTFLoader. I went with Blender instead of the provided three.js ConeGeometry because I wanted a bit more control over the texture mapping. In essence, I wanted the texture to simple lay flat on the cone so designing the flower textures would be simpler. Each flower is initialized, positioned, and scaled using the stored data, including which stems' bone it should be added to.
bones[segmentIndex].add(flower)
With all of this work, I now had some cylinders (cactus stems) covered in cones (flowers) bending out of a half sphere (pot.) It was finally time to do some design work.
Design
After all of this technical work, we return to the initial point of inspiration: Mackenzie Moore's painterly Proof of Life album cover, art directed by Sophia Matinazad. How the hell were we going to translate this 2D painting into our 3D scene? We can start by looking very closely at Mackenzie's work.
Stems
The cactus stems themselves are mostly simple green tubes consisting of a few different shades of green upon which a series of unique dots (thorns) sit. To begin with, I wanted to give the underlying cylinder a green hue which reacted to light in the scene but I wanted the green shades to be more abrupt like they exist in the artwork. In order to accomplish this, I turned to the MeshToonMaterial which three.js provides. This allows us to establish a green material which uses a gradient map to determine how its shading is handled. I then fired up Photoshop and carefully painted a seamless repeatable thorn texture that could be mapped alongside this toon shading. I had grander ambitions to make this texture more generative but in the end, this solution worked nicely.
let stemMaterial = new THREE.MeshToonMaterial({
map: stemTexture,
gradientMap: gradientMap
})
The toon shading worked so well, I applied it to other objects in the scene, like the pot, to create some visual consistency.
Flowers
As I mentioned earlier, the flowers are simple cones with a uv map that allows textures to be draped on top. All I needed was some textures... Luckily, my partner Anne Blenker is an accomplished artist with a keen eye for botanicals, not to mention incredible Procreate skills, and quickly painted several new flower textures which met the aesthetics of the original artwork and the technical needs of our 3D scene. By cutting out the petals of the flower using alpha transparency, the texture appears as transparent petals in the scene. We were both so excited to see these finally integrated into the scene. I should also take a moment to mention that Anne inspired many apsects of this build because artist nurturement and patronage are topics we discuss a lot together.
let flowerMaterial = new THREE.MeshBasicMaterial({
map: flowerTexture,
side: THREE.DoubleSide,
transparent: true
})
Background
The final touch involved establishing a backdrop on which our plant could suspend. I was inspired by the back cover of the album artwork which contains a darker texturing enveloped by some of the cacti and flowers we've been building off of. I took a cropping of this dark area and worked it into a seamless texture using Photoshop which was then applied to a CubeTexture. You can then use this cube texture as the background of the overall Scene to make it look like the plant was placed right into it.
Thanks
This was such a special project to work on and I have many collaborators to thank for the opportunity. First, thanks to Tristin Marshall, Xiarra Diamond, Zoe Ozochiawaeze, and Tim Hrycyshyn from Republic Records for bringing me in on this and supporting the entire process. Thanks to Mackenzie Moore and Sophia Matinazad for the inspiring artwork to develop this concept around. Thanks to Anne Blenker for talking through the concept and assisting in some of the texture creation and testing. Finally, thanks to Joy Oladokun for sharing her art with the world. I hope this application plays a small part in nurturing her career and I greatly look forward to her new album Proof of Life.