Spatial Nodes

A visual programming language for creative coding in virtual reality

Abstract

Node-based programming is usually done on a two-dimensional canvas. What if it were done in a three-dimensional space? Spatial Nodes is a visual programming language for creative coding in virtual reality that tries to answer this question. Users can build interactive and animated worlds by connecting and manipulating nodes directly with their hands. The nodes are evaluated as the user modifies them, so the result of the user's action is visible on the scene in real time. The project shows how programming can become a spatial and embodied activity. It identifies some challenges in creating this kind of a programming environment, such as interaction design and debugging, and suggests that example scenes can be an effective way of learning how to use it.

Coding in a three-dimensional space

When people think of programming, I believe they often envision something that requires a lot of careful and analytical thinking. But there are many programming systems that are designed for a more intuitive, trial-and-error like experience. Sometimes, these kind of programming systems come in the form of node editors. You can find them in 3D software such as Blender, game engines such as Unreal Engine or compositing software such as Nuke.

Node editors usually exist on a two-dimensional canvas. As part of my bachelor’s thesis last summer, I explored how a node editor might function in a three-dimensional space. What emerged from this was proof of concept for a visual programming language for creative coding in virtual reality. I've named it Spatial Nodes.

Spatial Nodes enables users to create animated and colorful worlds. The programming language gives users access to the position of their hands, allowing them to create worlds that respond to the users’ movements. The nodes are evaluated every frame, so they react to the users changes in real-time.

Demo

The users starts in a dark void. Only a grid on the floor and a bunch of nodes are present. The nodes can be grabbed and moved. The nodes' world space can be translated, rotated and scaled by grabbing and moving the void by hand. The user interacts with the program using without any controllers, only their hands.

Moving and scaling nodes.

Nodes can be connected to each other by pinching an output or input of a node using the index finger and thumb. Changes made to the nodes result in immediate changes in the scene.

Connecting nodes.

Node inputs and outputs that have the same data type can be connected in whatever way the user sees fit. For example, to draw mutliple cubes on top of each other.

Drawing multiple cubes.

Learning by Example

Other node editors often provide example files or node trees that the user can learn from. Cognitive psychologists call this "worked examples", I believe. I think that's a very effective way of learning, so I've created eight example node trees that can help people understand how they could use the nodes to create various effects.

Spinning Cube: Shows how to use the time node to rotate an object.
Primitives: Shows the geometry primitives available to the user.
Hands: Shows how to use the position of the hands.
Colors: Shows how to color objects and the background.
Grid: Shows how to draw multiple primitives and arrange them in a grid.
Ocean: Shows how to take the Grid example further.
Circle of Cubes: Shows how to make a larger scene that reacts to the user's body.
Nauseatic Madness: Just for fun.

I like how the web-based node editor cables.gl is using examples extensively: After clicking the button "Run cables directly in your browser!", an example "patch" (project) is opened. They also provide many example patches that show various features of the software and have more than 6'000 public user-generated patches that can be inspected and modified. Every node is documented and oftentimes has an example patch that was created just to explain what a node does. In the documentation, they also list user-generated patches that use this node, so you can see how the particular node can be used in a more sophisticated context. I think lots of these things could also work for Spatial Nodes.

First contact with the editor.
Simple example patches.
User-created patches.
Patch templates.
Example patch for the smoothstep node.
User-created patches that use the smoothstep node.

In the future, I'd also like to offer scenes where some nodes are floating in space and the user has to create a specific effect in order to proceed. Video games sometimes do this: In the beginning of Opus Magnum, the player has to solve a bunch of simple tasks. A lot of them are already solved to some extent, which allows the player to understand how the different building elements function. Factorio does this, too, by placing structures on the map that only need small adjustments for them to run again.

In Factorio, sample structures even fit the backstory: an abandoned stone and ironworks.
Opus Magnum: The four components on the left help explain the newly introduced element.

Inspiration

I like writing shaders, and they way shaders work inspired Spatial Nodes. In shader programming, the same code is applied to every pixel of the shader. Similarly, in Spatial Nodes, the same code is applied to every primitive of a node tree.

Minecraft's Redstone and the Create Mod were also an inspiration. I like how they enable players to build (program) systems in a 3D space and wanted to create a similar experience. In my initial vision for this project, the programming blocks were much more situated in space and the user would navigate around them, instead of moving the nodes.

Redstone in Minecraft by Thgabs.
First concept art for Spatial Nodes.

Challenges and Future Possibilites

Debugging Tools

Sometimes a cube just disappears and it's hard to understand why. Spatial Nodes could benefit a lot from debugging tools. I like how cables.gl, for example, displays a preview of an output when you hover on a connection.

Something similar could exist for Spatial Nodes. But there are some open questions: When drawing multiple objects, nodes can have different output values based on which iteration they're executing. Does it just display the first iteration? Can the user choose which iteration to display? What if the iteration count itself is dynamic?

Grouping

Spatial Nodes doesn't currently have concepts like functions or node groups. As a result, node graphs can become messy very quickly. If I were to implement functions, how would they be used? How could a user select multiple nodes? How could a user "enter" and "exit" a function definition? How could a user preview a function? How could a function be saved and reused?

User Experience

Connecting and disconnecting nodes sometimes is quite finicky. When a user moves their fingers close to an node input or output, it renders an outline to show that the user is close enough to draw a connection. When they then pinch their thumb and index finger, they usually slightly move away from the base and suddenly their not close enough to the input or output anymore to draw a connection. This is frustrating for the user, so I added a small time window during which a base still considered close enough to establish a connection.

There's a lot more that could be done to improve the UX. I like the things mentioned in 3.1 Classic: Nodes and wires of the LIVE Primer. I also like how in Nuke, shaking nodes disconnects them.

Shaking nodes in Nuke disconnects them.

Findings

I learned that it's fun to program three-dimensional worlds with a three-dimensional programming language. I also especially like scenes that react to a users body, and want to push further into this direction

I've used DeepL to translate some parts of the text from German to English.

Submission

What did you discover that other researchers should know about?

Programming with your hands in a three-dimensional space is fun. It’s also fun to create scenes that react to your body and I think it makes sense to create such environments in the same environment they will be experienced in.

What previous systems are similar to yours, in their goals or their methods? How does your system differ?

cables.gl is a similar project where you draw scenes using nodes. But cables.gl is trigger based: It has a main loop node with a trigger that runs decending nodes every frame. In Spatial Nodes, this is less explicit. Nodes run every frame by default* and you just connect them to output nodes like “Draw Cube” or “Draw Sphere”

(* In the future I want to change it so branches of the node tree that don’t depend on dynamic values like the current time or hand positions can be cached and don’t have to be recomputed every frame.)

Redstone in Minecraft feels a bit similar, because players are also laying out programs in a three-dimensional space. In Minecraft, it feels more situated compared to my project, probably because in Spatial Nodes you can move, rotate and scale the node tree and they are placed in a dark void instead of next to grass blocks or trees.

Where are the limits of your system? What unsolved problems lie ahead?

Lot’s of open UX questions: - How to do node groups/functions? - Connecting/disconnecting nodes feels finicky to some users - User can reach situations that are tricky to exit, like drawing primitives on their hands and therefore not seeing their fingers anymore. - How to debug output values from single nodes?