Intro
For the past 2 weeks, we have been working on integrating and improving the more advanced painting module into our application. Additionally, the menu and manipulation gestures were also tested and fine tuned.
Progress
Previously, the original painting module was inflexible as it was originally designed for just painting on a whiteboard. Now, with the more complex painting module, we are now able to paint on multiple holograms with multiple people. While most of the original functionality was ported over, we are still working on some remaining parts such as syncing brush cursors, changing the brush colors, and changing the scale and shape of the brush. The workload for these seem to be fairly minimal as it mainly involves changing a few lines or so in the various button scripts.
We were also able to make the manipulation drawing much more smoother. One thing that we did was to test and fine tune the parameters for the manipulation position that was given when a manipulation event is updated. Currently the change in position is multiplied by two in order to allow the user to draw more freely in the limited detection space that the Hololens device supports. Another thing that we did was to add a stepping function to the painting when doing the manipulation gesture in order to make the painting look smoother. Currently, if a user starts painting at point A and ends at point B, we use the Vector2.Lerp() function and move along the UV mapping in order to paint everything in between A and B.
The hologram position and rotation can also be manipulated through hand gestures, an upgrade from the original always facing user state that was originally designed for the whiteboard.
In terms of the menu, we currently plan on having a button on screen that makes the menu pop up in front of the user. However, we are currently having trouble with the custom animations to show up and will continue to work on that. In the event that we are unable to do so, we plan on just having the buttons all laid out on a board and will appear/disappear when the menu button is clicked.
Currently, our work is split up as such.
Radu is working on a better UI buttons, sound and animations for the menu
Panji is working on adding functionality for deleting holograms as well as clearing paint from holograms.
Andy is working on displaying other users’ cursors in the application.
Thomas is working on connecting the brush buttons to the painting module.
Difficulties we encountered
Filling in the paint between two points and painting in general was harder than expected when switching to the new module. This is due to the fact that the holograms are of varying depth and our original application was planned for a whiteboard of constant depth. As a result, a lot of our original code for painting did not work right off the bat since the paint coordinate would sometimes have inaccurate painting coordinates due to varying depth (paint inside the model or paint far away from desired location). To resolve this, we used Raycasting from the camera to the desired paint coordinate at every paint operation in order to get the correct depth. This allowed us to correctly place paint on the hologram. However, this has introduced another issue. Since we are raycasting from the user’s camera, whenever we move in the world, the brush will move along with it if we are currently using the manipulation gesture. A proposed solution to this would be to use the brush cursor position as origin for raycasting instead of the camera. We will test this soon, though we think that it might not be too big of an issue as a person is not likely to move and paint at the same time.
Demo Scene Details
For the demo scene, we have decided to make an urban city environment with an asset pack that we found on the unity asset store. It contains things like skyscrapers, stadiums, nature items, etc. and we will pre load these holograms upon startup. Users will be allowed to draw on any part of the environment and there will also likely be some sort of graffiti wall that users can draw on and persist across all sessions.
Plan for next week
Our plan for next week is to do the following things:
- Create a scene for the demo
- Have a stylish menu which pops up
- Able to clear all drawn items and reflect that across all instances of a hologram
- Have all the existing functionality hooked up to the new painting module