Remixing and Resampling Three Dimensional Objects Use of Volumetric Representations And Machine Learning in Design

Design Workflows Integrating Machine Learning and Voxel Representations


Next!


ABSTRACT

This thesis aims to explore the opportunities of voxel modeling with Machine Learning.
First, it introduces (1) what voxel modeling is, compared to traditional model technic,
what the characteristics of voxel model such as pixel map and graph representation,
and what the Deep Learning and network are.
The thesis examines (2) prototypical implementations of proposed design systems or workflows
based on the process from rasterization of space and geometry to Machine Learning.




CONTENT







Next!


- SLIDE 01 -

Introduction





Next!


- SLIDE 02 -

AI in our daily life beyond aid



inference
classification or ranking
recommendation
voice recognition
translation


amazon
google
facebook
microsoft
Tesla
...





Next!


- SLIDE 03 -

technology in design



CAAD (Computer-aided architectural design)

creation, modification, analysis, or optimization of a design





Next!


- SLIDE 04 -

beyond aid in design



fabrication
3D printing
material
physically based rendering
BIM (Building Information Modeling)
modeling library
...





Next!


- SLIDE 05 -

PROBLEM STATEMENT AND OBJECTIVE





Next!


- SLIDE 06 -

PROBLEM

limited geometry resources
lack of skill using 3D tools
managing system


QUESTION

exploring various geometry options
finding geometry among data base
sketch to 3D fabrication


OBJECTIVE

geometry’s blending problem
classification, inference, matching problem
design system problem





Next!
- SLIDE 07 -


LITERATURE REVIEW AND CASE STUDY



literature review: Deep Learning


scene parsing , semantic segmentation, colorization


Learning Deep Features for Scene Recognition


DeepStereo: Learning to Predict New Views from the World's Imagery
Data-driven Visual Similarity for Cross-domain Image Matching


Learning a Probabilistic Latent Space of Object Shapes
via 3D Generative-Adversarial Modeling





Next!
- SLIDE 08 -


Deep neural network






Next!
- SLIDE 09 -


LeNet-5






[LeCun et al 2012]
Conv filters were 5x5, applied at stride 1
Subsampling (Pooling) layers were 2x2 applied at stride 2





Next!
- SLIDE 10 -


AlexNet




[krizhevsky et al 2012]
Input: 227 X 227 X 3 images
ReLu, dropout
7 CNN ensemble: 2.8 reduction of error 18.2% -> 15.4%





Next!
- SLIDE 11 -


GoogLeNet






[Szegedy et al., 2014]
Inception module
ILSVRC 2014 winner (6.7% top 5 error)





Next!
- SLIDE 12 -


ResNet




[He et al., 2015]
Xavier/2 initialization from He et al.
No dropout used
ILSVRC 2015 winner (3.6% top 5 error)





Next!
- SLIDE 13 -


Siamese net




Siamese networks for image ranking/retrieval





Next!
- SLIDE 14 -


Case study: 3D modeling






Next!
- SLIDE 15 -


Polygonal model(mesh)






How they construct:
Adding or removing Vertex and its connectivity in a space

Data:
Containing point and its connectivity including normal, UV and so on





Next!
- SLIDE 16 -


PNURBS(Non-uniform rational B-spline)







How they construct:
Define mathematical model for curves or surfaces

Data:
Control points, Knot vector, and Positional, tangential and curvature continuity





Next!
- SLIDE 17 -


Connectivity







Connectivity or data dependency , relational data base
- Like Graph model or BIM : how they are related in a space





Next!
- SLIDE 18 -


connectivity comparison







voxel & pixel

dense representation
continuous information
implicit relations due to proximity [neighbors]


Mesh (graph like structure)

sparse representation
discrete points or information
explicit relations





Next!
- SLIDE 19 -


pixel as a data structure










- Pixel is point like feature, this pixelated grid carries the information
and help to do position.

- the idea pixelated or voxelated space allows describe properties that derived
by point to point in space.





Next!
- SLIDE 20 -





capturing spatial data in pixel


real time curvature analysis on pixel map






Next!
- SLIDE 21 -


forward propagation:
Walker-ability for urban energy modeling


Properties : distance
slope
retail
tree(park)
outdoor Thermal Comfort
view ...




Next!
- SLIDE 22 -

Voxel (Volumetric Element)





- Voxel could be considered as a set of image(pixel)
Voxel is a 3 dimensional grid containing pixels can have rich data set including R, G, B, A values.
It is frequently utilized for visualization of scientific or medical data which is needed for volumetric representation.

Voxel for geometry in space is a discretized space of geometry
where it has a beam or node are connecting but as a continuous map in space.
This is basically an idea that an object become a 3 dimensional map.




Next!
- SLIDE 23 -

voxel from image





Mesh rasterization
Construction from sequence images
capture data
...




Next!
- SLIDE 24 -

voxel with blending



Blending modes
Threshold
Brushing
Filter: Smoothing and so on
...


Next!
- SLIDE 25 -






METHODOLOGY








Next!
- SLIDE 26 -

OVERVIEW




design workflow
Machine Learning with Voxel




Next!
- SLIDE 27 -

voxel with blending




design workflow
Machine Learning with Voxel




Next!
- SLIDE 28 -

rasterization (vector to bitmap)



rendering


slice mesh with Monolith





Next!
- SLIDE 29 -

train




5 classes and 1,000 images for each class
capture all possible angle along X, Y and Z axis




Next!
- SLIDE 30 -

train




train : 3,200 images (random)
validation : 800 images (random)

optimization
Adam Optimizer
Gradient Descent Optimizer
Momentum Optimizer
Proximal Gradient Descent Optimizer
Ftrl Optimizer
RMSProp Optimizer




Next!
- SLIDE 31 -

optimizer comparison







Next!
- SLIDE 32 -





Next!
- SLIDE 33 -

optimizer comparison








Next!
- SLIDE 34 -

train data exploration






Next!
- SLIDE 35 -

outline and background experimentation





clear outline when resize (224X224)




Next!
- SLIDE 36 -

optimization





data_mean = np.asarray([0.5,0.5,0.5]) image = image - self.data_mean




Next!
- SLIDE 37 -

rasterization of degree and resolution








Next!
- SLIDE 38 -

rasterization of degree and resolution





180 degree of object’s facade
or
380 degree along X, Y and Z axis




Next!
- SLIDE 39 -

rasterization of chair for ranking




design workflow
Machine Learning with Voxel




Next!
- SLIDE 40 -

class







Next!
- SLIDE 41 -

train images for each class







Next!
- SLIDE 42 -

network and properties





TRAIN PROPERTIES
class : 13
train images : 13,000 images
one iteration : 100 images
epoch : 1400 iterations
total iteration : 140,000
network : originated from alexNet
framework: Tensorflow
optimizer: tf.train.AdamOptimizer
learning rate = 0.01




Next!
- SLIDE 43 -

validation








Next!
- SLIDE 44 -

result










Next!
- SLIDE 45 -

ranking and geometry blending








Next!
- SLIDE 46 -

train image for detecting rotation degree










Next!
- SLIDE 47 -

geometry blending








Next!
- SLIDE 48 -

geometry blending




result A


result B




Next!
- SLIDE 49 -

blend modes


Multiply > Screen > Overlay > Hard Light > Soft Light




Blend
f(a,b, blend ) = a (1.0 - blend ) + b * blend

Multiply
f(a,b) = ab

Screen
f(a,b) = 1 - ( 1 - a ) ( 1 - b )

Overlay
f(a,b) = 2ab , if a < 0.5
f(a,b) = 1 - 2 ( 1 - a ) ( 1 - b ) , if a > 0.5

Soft Light
f(a,b) = 2ab + a^2 (1 - 2b) , if a < 0.5
f(a,b) = 2a (1-b) + sqrt(a) ( 2b - 1) , if a > 0.5




Next!
- SLIDE 50 -



Multiply > Screen > Overlay > Hard Light > Soft Light







Next!
- SLIDE 51 -

blend by transform






position
rotation
scale




Next!
- SLIDE 52 -

WORKFLOW B WITH VOXEL MODEL






design workflow
Machine Learning with Voxel




Next!
- SLIDE 53 -

voxel to mesh



         

voxelated by slicing
mesh reconstruction by voxel




Next!
- SLIDE 54 -

voxel to mesh



    

voxelated by slicing
mesh reconstruction by voxel




Next!
- SLIDE 55 -

slicer





voxel slicer, stand-alone




Next!
- SLIDE 57 -

rasterization of geometry






Voxelated geometry




Next!
- SLIDE 58 -

result with different line widths









Next!
- SLIDE 59 -

epoch









Next!
- SLIDE 60 -

overfitting









Next!
- SLIDE 61 -

resolution of voxel









Next!
- SLIDE 62 -

result









Next!
- SLIDE 63 -

train with primitive geometry









Next!
- SLIDE 64 -

result









Next!
- SLIDE 65 -

result









Next!
- SLIDE 66 -

possibly rebuilt geometry










Next!
- SLIDE 67 -







train with depth









Next!
- SLIDE 68 -

train with depth image










Next!
- SLIDE 69 -

validation with sketch










Next!
- SLIDE 70 -

train with depth image










Next!
- SLIDE 71 -

validation with depth image










Next!
- SLIDE 72 -

result with sketch













Next!
- SLIDE 73 -

reconstruction mesh from depth image by training







point cloud




Next!
- SLIDE 74 -

Projection Transformation and Inverse Projection










Next!
- SLIDE 75 -

reconstruction mesh from depth image by result










Next!









VOXEL BLENDING ( IN POST TRAIN)









Next!
- SLIDE 76 -

different blending model and transformation










Next!
- SLIDE 77 -

different blending model










Next!
- SLIDE 78 -

different blending values










Next!
- SLIDE 79 -

different blending values with rotation










Next!
- SLIDE 80 -

different blending with depth at different angles










Next!
- SLIDE 81 -

POSSIBLE IMPLEMENTATION







Next!






CONCLUSION











Next!
- SLIDE 82 -

summary of problem statement










Next!
- SLIDE 83 -

summary of work flow












Next!


contribution




training network for 3D geometry

volumetric representation with Machine Learning

mesh blending for design space, emerging meshes

use of repository of geometry




Next!


future works




3D convolutional neural networks for voxel

to contrive interface that use of geometry database to recreate mesh

optimization with structural and material analysis

interface how people use this workflow




Next!

REFERENCE

NETWORK
LeNet-5 [LeCun et al 2012]
AlexNet [krizhevsky et al 2012]
pix2pix


STAND-ALONE
Slicer


3ds max
JSON export
environment setting(rendering and post-effecting)
...


Rhinio3D
IO for JSON
...


LIBRARY
Monolith, Millipede, openGL, python library for Machine Learning


FRAMEWORKS
tensorflow
network: Alexnet, pix2pix, and others


CAD SOFTWARE
3ds max, Rhino3d

...


CODE
c# python, maxscript, html, CSS, JS


Next!


adviser




Panagiotis Michalatos
Harvard Graduate School of Design


Takehiko Nagakura
Massachusetts Institute of Technology




Next!










NJ Namju Lee

Master in Design Studies(MDes), 2017

nj.namju@gmail.com




www.njstudio.co.kr