Summary
The purpose of HoloPaint is to give users an interactive whiteboarding experience, allowing them to work together and share ideas with others using the Microsoft Hololens. We want to allow users to be able to have a real time collaborative experience, where multiple users can interact with a single whiteboard in order to sketch out and discuss their ideas without the need of a huge space or having to crowd around a single writing surface. We want this application to be relatively simple for the casual user, so that any individual would be able to pick up a Hololens and begin contributing ideas. This will be implemented as a Hololens Shared Experience and many users will be able to see and work on a single canvas.
With many diverse applications such as drafting ideas in meetings, drawing examples while teaching, and giving users an outlet to play games like Tic Tac Toe, Hangman, and Pictionary, HoloPaint allows Hololens users to enhance their AR experience and also communicate their ideas clearly and effectively.
Deliverables
Our project is a collaborative painting application on the Hololens device. This application will have a shareable workspace(e.g. A canvas hologram) for multiple users to draw and sketch using hand movements. The user can create their drawings in this workspace. The workspace is a hologram of a whiteboard with available drawing tools (brush, eraser, etc). It is moveable in the actual real space, thus it can be put on the wall, on the table without physical limitations. The workspace is shareable for multiple users in the same room and people can use it to collaborate and work together in real time. Once users are done with sketching, they can also save the results of the sketching.
Critical Features
- Drawing on a virtual whiteboard with hand movement and gestures
- Options for drawing (color and width of cursor)
- Options for whiteboard (multiple whiteboards, placement, color, size)
- Simultaneously editable by multiple Hololens users in the same room
- Creation exportable as an image
Performance Metrics
- Usability of the application. We want our application to be usable without any tutorials, similar to Paint.
- Accuracy of synchronization between multiple Hololens devices. Anything one user creates should be identical for the other users and any conflicts must be handled gracefully.
- Precision of painting. What the user creates on our application must be what the users actually did and expect (which is something we also have to manage if the painting is not 1 to 1 to hand movements).
- Performance of the application. We are aiming for the application to be able to run at a refresh rate of 60 Hz or more.
Milestones
Weeks 4 - 5 Basic Elements of the applications- Be able to track user’s hand gestures and have a cursor follow the focus point
- Display one basic whiteboard hologram in space
- Allow user to draw on whiteboard using simple hand gestures
* Start when basic elements are mostly finished
- Establish what data to transfer between multiple instances of application
- Setup server for communication between Hololens devices (decide between lab machine server or AWS server)
- Implement collaborative painting on whiteboard (changes made to whiteboard appear on all other instances of the whiteboard)
- Whiteboard background manipulation (Transparency and Color)
- Change whiteboard position and orientation within space using gaze and hand movement
- Extend/Zoom whiteboard
- Export/Import whiteboard as image
* Start work as core Whiteboard features are finished
- Shape presets that can be dragged onto whiteboard
- Change color and size of painting tool
- Eraser for deleting drawn items
- Undo/Redo for deleting/restoring previously drawn items
- Debug application
- Polish features and user experience
- Start writing final paper
- Planning for presentation and demo
Responsibilities
- Borui Andy Li Application and UI design
- Panji Wisesa Networking and Painting functionalities
- Radu Cracut Whiteboard and Painting functionalities
- Thomas Yi Painting functionalities and UI design
Materials and outside help needed
We need access to a server to facilitate communication between Hololens devices. And we also may need assistance with networking portion of the application, though this should be resolvable through participating in the Hololens App Development Forum.Budget
- UI Assets $250
- UI Sounds $150
- Background Music $50
- AWS/Azure credit for server $300
Risks and how risks will be addressed
- Hand tracking may be too coarse for user purposes (fine detailed drawing). Allow zoomable whiteboards so users can draw fine details on an enlarged portion of the whiteboard then shrink the whiteboard back down to make imperfections unoticeable.
- Latency between devices might negatively affect user experience. Optimize network code, use local machine as server as backup if connection over third party server is too unresponsive for normal use.