Research & Projects

ATLAS Residency: Changing Norms of Physical Activity

Will our norms of Physical Activity that are accepted in public (and in the workplace) change in the next few years? This question becomes more urgent as we see physical-activity-based furniture such as standing desks become popular, we acknowledge how movement regulates our social connections and wellbeing, and as we adapt to technologies such as immersive VR that may change our daily movement routines.

I was awarded a multi-week summer residency at the Atlas Institute advancing answers to these questions through creation of a performance art piece comprising circus aerials, contra dance, and lecturing while upside-down.

Through our video recording, we invite a global audience to ponder the performance’s meaning in the context of their own physical environments. Through our live performance in the Atlas Institute Black Box Theatre, we invited a real-time dialog with our audience from Atlas Institute, CU Boulder Dance department, and Boulder, Colorado community members.

Centre for Digital Media Workshop Performance: Do You Think Differently Upside-Down?

Through an aerial performance piece, we invited the participants in a workshop on future at the Centre for Digital Media, Vancouver, Canada, to contemplate the question: Do you think differently upside-down? The unusual choreographic and performance combination of circus aerials and lecture-format speaking self-referentially invites the audience to assess the potential effects of movement interventions in the workplace on physical, cognitive, and social wellbeing.

Movement Capture and Interaction Research Group

The Movement Capture and Interaction Research Group builds capacity at NYUSH for motion capture using commercial and homemade equipment (video-camera and IMU-based technologies), 3D modeling of avatars, and research in whole-body interaction.

In 2021/2022 we built a pipeline from a commercial motion capture suit to experiment with dancers inhabiting semi-humanoid avatars including a custom Quilin designed with input from authentic Chinese representations as well as surveys of student-community design preferences. In collaboration with the Dance faculty of the Performing Arts Program, students from Introductory Dance participated with research group members to collect dance movement data, make observations of the proprioceptive and movement effects of dancing in a non-human body, give a workshop for animation students, and give a public demonstration/performance.

Future projects include movement tracking and recognition for circus arts and other movement practices, qualitative and quantitative studies of effects of avatar physics on dance performance, and the design and study of experimental movement-informed workplace furniture.

Craft Computing Group

Turtlestitch programming at NYUSH:

This group of undergraduates and recent graduates of the IMA Lo-Res program creates artifacts and coding projects using programmable embroidery, e-textile, and textiles craft techniques.

We engage in an international collaboration with the authors of Turtlestitch (a blocks-based programming language for controlling graphics and embroidery). We contribute design examples, give feedback on the language design, and test curriculum materials. Recently, we have begun a systematic exploration of free-standing lace designs as physical representations of mathematical and computing concepts, as well as artistic artifacts.

We study the technical manufacturing pipeline, and its relationship to DIY craft processes, of sewn, knit, and embroidered products in Southeastern China through on-site observation in factories and businesses.

We engaged university students in the creation of a swatchbook of computer-programmed embroidery designs, in the context of a course. Highlights of learning that were enabled by this tangible approach include students’ familiarization with material properties of textiles (fabrics, stabilizers, and threads) and their stitchability, approaches to debugging such as slowing the stitching processes to observe the control flow of a program, discussion and acknowledgment of the challenges of artisanal manufacturing, and the diversity of aesthetics of “finishing” the pages in order to reveal, or conceal, the craft processes.

We build capacity for craft computing research at NYU Shanghai through creation of a lab space and conducting hands-on workshops for the student and staff community on e-textiles topics.

PoseShare

PoseShare is a web-enabled platform for Zoom inhabitants to become visitors to each others’ “homes”. Each Zoom participant’s body is inserted into a pipeline through BlazePose in p5.js to create avatars whose movements are tracked. The avatars are reinserted into Zoom through OBS. Spatial coherence is maintained using Zoom’s new “follow host’s video order” feature. Choreography is a combination of a notated, cued dance score along with pose recognition elements programmed in p5.js.

Using the PoseShare platform, we created a choreographic work with avatars whose appearance and pose recognition were programmed by students, and cued choreography co-created by faculty and students. The platform can be run on average laptops and enables interactive inter-tile visiting in video conferencing.

Project website →

Sensing Acrobatic Movement

In 2019 we recruited professional circus performers to create a video database of upside-down movements and poses including handstands and cartwheels. We recruited students to research improvements to neural-net based approaches for AI recognition of these movements, with the aim of improving both non-real-time and real-time tracking of acrobatic movement from conventional camera input.

In Spring 2020, we directed a computer science Capstone project which was able to significantly improve tracking of body position over standard neural network performance, using an approach adapted from a rotational post-processing algorithm by Toyoda, Kono, Rekimoto (K. Toyoda, M. Kono, and J. Rekimoto, “Post-data augmentation to improve deep pose estimation of extreme and wild motions,” 2019 IEEE Conference on Virtual Reality and 3D User Interfaces).

Drawings from Tinbergen

Ants Fabric

DinaCrab: Hermit Behavior

Soft Circuits at Amrita University

A workshop on the design of soft circuits in the context of undergraduate engineering education and empowerment of rural, economically disadvantaged women in India was conducted at Amrita University in Kerala, India. This complemented existing undergraduate engineering education with increased expressivity, initiative, improvisational engineering skills, and integration of hardware and software skills.

The workshop participants developed a Livelihood-Education plan that will be piloted with rural women with less than 9th grade formal schooling, who are in vocational training.

StyleCart

Co-founder of small technology startup company that made an innovative web-based visual shopping cart for retail merchants. I designed web site elements, and invented user interaction techniques, as well as doing business development and sales.

This project was developed before interactive graphical web pages, and supporting services for browser/server synchronization, became widely available. It generated two patents.

Margaret Minsky (design and manufacture of novel garments)

I founded a women’s specialty design/manufacture company based on my invention of a novel fabric processing technique: pleated Polarfleece. I co-designed the garments, and manufactured 30,000 units for a major national catalog retailer, managing a supply and production chain across three US states.

Wearable Audio Garment

A personal communications apparatus using a garment-based audio interface includes an upper-body wearable, embodied as a shirt or as a necklace with an audio output device capable of producing hi-fidelity spatialized 3-D sound adjacent to the neck opening of the wearable.

Computational Haptics

At the MIT Media lab, I created the lateral force algorithm for texture simulation, a seminal contribution to the field of force-feedback haptics, which is now used in consumer devices, medical simulations, and theme parks. My work on haptic materials and textures contributed to the design of two generations of Force-Feedback joystick hardware. Concurrently, as a Visiting Scholar at the Computer Science Dept of the University of North Carolina, I developed a software platform for the texture simulation research, supervised undergraduates, and was awarded a University Research Grant from Apple Computer.

Manipulating Simulated Objects with Real-World Gestures

A flexible interface to computing environments was provided by gestural input to a capacitive touch screen mounted on a novel configuration of strain-gauge force sensors. We describe a prototype system that recognizes some types of single-finger gestures and uses these gestures to manipulate displayed objects. An experimental gesture input device yields information about single finger gestures in terms of position, pressure, and shear forces on a screen. The gestures are classified by a “gesture parser” and used to control actions in a fingerpainting program, an interactive computing system designed for young children, and an interactive digital logic simulation.